How To Write Unit Tests For OpenSSL - Revision history

Mbland at 11:03, 10 August 2017

2017-08-10T11:03:07Z

Mbland: /* Use the Test Template Generator */ Revamped as "Create a new test file using the test template"

2014-08-21T14:15:22Z

Use the Test Template Generator: Revamped as "Create a new test file using the test template"

Ckomlo: /* Forking the Repo */

2014-08-05T21:39:42Z

Forking the Repo

Mbland: /* Building and Running the Test */

2014-08-02T18:28:24Z

Building and Running the Test

Mbland: /* Style */ Combine last two sections in light of ADD_TEST() and run_tests()

2014-07-20T00:00:12Z

Style: Combine last two sections in light of ADD_TEST() and run_tests()

Mbland: /* Add each test case function to the test runner function */ Update with advice to use ADD_TEST() and run_tests()

2014-07-19T23:56:37Z

Add each test case function to the test runner function: Update with advice to use ADD_TEST() and run_tests()

Mbland: /* Add Makefile Targets */ Add testutil.o to test executable build rule

2014-07-19T23:53:01Z

Add Makefile Targets: Add testutil.o to test executable build rule

Mbland: /* Style */ Add testutil.h section

2014-06-18T20:40:19Z

Style: Add testutil.h section

Mbland: /* Style */ Added section on #including the header for the code under test first

2014-06-18T20:28:23Z

Style: Added section on #including the header for the code under test first

Mbland: /* Run make depend */ Updated to clarify that 'make links' must be run first.

2014-06-18T20:13:49Z

Run make depend: Updated to clarify that 'make links' must be run first.

@@ Line 2: / Line 2: @@
 ==Mechanics ==
 === Check out the Tools and Tips page ===
 [[Testing and Development Tools and Tips]] has information on tools that may make navigating and building the code a bit easier.
-=== Create a new test file using the test template ===
+Create a new test file in the same directory as the code under test using this template:
 <pre>
@@ Line 220: / Line 193: @@
 $ ./config enable-unit-test && make && make test
-# To execute just one specific test
+# To execute just one specific test, where <test> is the basename of the test file:
-$ make TESTS=test_heartbeat test
+$ make TESTS=<test> test
 </pre>
@@ Line 231: / Line 204: @@
 </pre>
-The above currently doesn't work on Mac OS X or FreeBSD > 9.1. The [https://github.com/mbland/openssl/commit/a3984ff8a2e84132f221557bc26415314daf3259 {,darwin64-}debug-test-64-clang Configure targets] commit, which should go in soon as part of [https://github.com/openssl/openssl/pull/145 pull request #145], should solve the issues. Other commits from #145 contain other OS X-specific fixes.
+The above currently doesn't work on Mac OS X or FreeBSD > 9.1. The [https://github.com/openssl/openssl/commit/a3984ff8a2e84132f221557bc26415314daf3259 {,darwin64-}debug-test-64-clang Configure targets] commit, which should go in soon as part of [https://github.com/openssl/openssl/pull/145 pull request #145], should solve the issues. Other commits from #145 contain other OS X-specific fixes.
 === Keep your repo up-to-date ===
@@ Line 242: / Line 215: @@
 === Send a pull request ===
-When your test is ready, send a [https://help.github.com/articles/using-pull-requests GitHub pull request]. Note that the pull request should be based on the <code>tests</code> branch of Mike's repository, not <code>master</code>. (This should be the default; let Mike know if it doesn't appear to be!) We'll review the code, and when it's ready, it'll get merged into Mike's repository. From there, it will eventually get pulled into the master OpenSSL repository.
+When your test is ready, send a [https://help.github.com/articles/using-pull-requests GitHub pull request]. We'll review the code, and when it's ready, it'll get merged into the repository.
 == Style ==
-[http://mike-bland.com/2014/06/05/pseudo-xunit-pattern.html The Pseudo-xUnit Pattern] is that established by [http://git.openssl.org/gitweb/?p=openssl.git;a=blob;f=ssl/heartbeat_test.c ssl/heartbeat_test.c]. This pattern organizes code in a fashion reminiscent of the [http://en.wikipedia.org/wiki/XUnit xUnit] family of unit testing frameworks, without actually using a testing framework. This should lower the barrier to entry for people wanting to write unit tests, but enable a relatively easy migration to an xUnit-based framework if we decide to do so one day.
+[http://mike-bland.com/2014/06/05/pseudo-xunit-pattern.html The Pseudo-xUnit Pattern] pattern organizes code in a fashion reminiscent of the [http://en.wikipedia.org/wiki/XUnit xUnit] family of unit testing frameworks, without actually using a testing framework. This should lower the barrier to entry for people wanting to write unit tests, but enable a relatively easy migration to an xUnit-based framework if we decide to do so one day.
 Some of the basic principles to follow are:
@@ Line 264: / Line 237: @@
 === Use <code>SETUP_TEST_FIXTURE()</code> and <code>EXECUTE_TEST()</code> from <code>test/testutil.h</code> ===
-Each test case function should call <code>set_up()</code> as its first statement, and should call <code>tear_down()</code> just before returning. This is handled in a uniform fashion when using the <code>SETUP_TEST_FIXTURE()</code> and <code>EXECUTE_TEST()</code> helper macros from [http://git.openssl.org/gitweb/?p=openssl.git;a=blob;f=test/testutil.h test/testutil.h]. See the comments in <code>test/testutil.h</code> for usage, and <code>ssl/heartbeat_test.c</code> for an example of how to wrap the macros for a specific unit test.
+Each test case function should call <code>set_up()</code> as its first statement, and should call <code>tear_down()</code> just before returning. This is handled in a uniform fashion when using the <code>SETUP_TEST_FIXTURE()</code> and <code>EXECUTE_TEST()</code> helper macros from [http://git.openssl.org/gitweb/?p=openssl.git;a=blob;f=test/testutil.h test/testutil.h]. See the comments in <code>test/testutil.h</code> for usage.
 === Use test case functions, not a table of fixtures ===
@@ Line 273: / Line 246: @@
 === Group test cases into "suites" by naming convention ===
-Give logically-related tests the same prefix, e.g. <code>test_dtls1_</code> and <code>test_tls1</code> from <code>ssl/heartbeat_test.c</code>. If need be, you can define suite-specific <code>set_up()</code> functions that call the common <code>set_up()</code> and elaborate on it. (This generally shouldn't be necessary for <code>tear_down()</code>.)
+Give logically-related test functions the same prefix. If need be, you can define suite-specific <code>set_up()</code> functions that call the common <code>set_up()</code> and elaborate on it. (This generally shouldn't be necessary for <code>tear_down()</code>.)
 === Keep individual test case functions focused on one thing ===
@@ Line 279: / Line 252: @@
 === Write very descriptive error messages ===
-Include the test case function name in each error message, and explain in detail the context for the assertion that failed. Include the expected result (contained in the fixture structure) and the actual result returned from the code under test. Write helper methods for complex values as needed (e.g. <code>print_payload()</code> in <code>ssl/heartbeat_test.c</code>).
+Include the test case function name in each error message, and explain in detail the context for the assertion that failed. Include the expected result (contained in the fixture structure) and the actual result returned from the code under test. Write helper methods for complex values as needed.
 === Return zero on success and one on failure ===
@@ Line 288: / Line 261: @@
 <code>run_tests()</code> will add up the total number of failed test cases and report that number as the last error message of the test. The return value of <code>run_tests()</code> should be the value returned from <code>main()</code>, which will be <code>EXIT_FAILURE</code> if any test cases failed, <code>EXIT_SUCCESS</code> otherwise.
 === Disable for Windows (for now) ===
@@ Line 305: / Line 276: @@
 #endif /* OPENSSL_NO_WINDOWS  */
 </pre>

@@ Line 30: / Line 30: @@
 [[Testing and Development Tools and Tips]] has information on tools that may make navigating and building the code a bit easier.
-=== Use the Test Template Generator ===
+=== Create a new test file using the test template ===
-TODO(mbland): Get template generator checked-in. Maybe have a template generator for each library, e.g. <code>ssl/new-test.sh</code> that has additional setup boilerplate specific to the <code>ssl</code> library.
+TODO(mbland): Get <code>test/new-test.sh</code> test template generator checked-in ([https://github.com/mbland/openssl/compare/test-util pending in the test-util branch]). Maybe have a template generator for each library, e.g. <code>ssl/new-test.sh</code> that has additional setup boilerplate specific to the <code>ssl</code> library.
-Use the <code>test/new-test.sh</code> script to generate a skeleton test file. ([https://github.com/mbland/openssl/compare/test-util pending in the test-util branch])
+Create a new test file in the same directory as the code under test using this template; see [http://git.openssl.org/gitweb/?p=openssl.git;a=blob;f=ssl/heartbeat_test.c ssl/heartbeat_test.c] for a full example:
 === Add Makefile Targets ===

@@ Line 3: / Line 3: @@
 ==Mechanics ==
 === Forking the Repo ===
-Rationale per Matt Caswell: We should think about process here. If you are going off to recruit an army of people writing tests then I wonder if it is worth setting up a separate github repository whilst you are building up the tests. We can then merge in from that on a periodic basis. I wouldn't want availability of openssl team committers to be a bottle neck.
+For now, people contributing new unit tests for uncovered code (as opposed to submitting tests with new code changes) should fork [https://github.com/mbland/openssl Mike Bland's GitHub repository] and issue changes via [https://help.github.com/articles/using-pull-requests GitHub pull requests] to this repo.
 <pre>
 $ cd my-openssl-repo
@@ Line 21: / Line 23: @@
 upstream        https://github.com/openssl/openssl.git (push)
 </pre>
 === Check out the Tools and Tips page ===

@@ Line 70: / Line 70: @@
 If you're initially developing on Mac OS X or (for now) FreeBSD 10, just use the stock method of building and testing:
 <pre>
-$ ./config && make && make test
+$ ./config enable-unit-test && make && make test
 # To execute just one specific test
@@ Line 78: / Line 78: @@
 Ultimately the test will have to compile and pass with developer flags enabled:
 <pre>
-$ ./GitConfigure debug-ben-debug-64-clang
+$ ./GitConfigure debug-ben-debug-64-clang enable-unit-test
 $ ./GitMake -j 2
 $ ./GitMake test_heartbeat
 </pre>
-The above currently doesn't work on Mac OS X or FreeBSD 10; for now, you can install the 9.1 release of [http://www.freebsd.org/ FreeBSD], which uses Clang 3.1, via a virtualization platform such as [https://www.virtualbox.org/ VirtualBox]. (Mac OS X breaks, in part, due to the fact that it doesn't use the GNU assembler; <code>-fsanitize</code> appears to be ignored on FreeBSD 9.1/Clang 3.1, but later versions break because the Clang compiler will emit <code>-fsanitize</code> symbols but the <code>libasan</code> library [http://lists.freebsd.org/pipermail/freebsd-hackers/2013-December/043995.html has yet to be ported to FreeBSD].)
+The above currently doesn't work on Mac OS X or FreeBSD > 9.1. The [https://github.com/mbland/openssl/commit/a3984ff8a2e84132f221557bc26415314daf3259 {,darwin64-}debug-test-64-clang Configure targets] commit, which should go in soon as part of [https://github.com/openssl/openssl/pull/145 pull request #145], should solve the issues. Other commits from #145 contain other OS X-specific fixes.
 === Keep your repo up-to-date ===

@@ Line 136: / Line 136: @@
 The return value will be used to tally the number of test cases that failed. Even if multiple assertions fail for a single test case, the result should be exactly one.
-=== Add each test case function to the test runner function ===
+=== Register each test case using ADD_TEST() and execute using run_tests() ===
 Whatever function is used as the test runner, be that <code>main()</code> or a separate function called by <code>main()</code>, add your test case functions to that function using <code>ADD_TEST()</code> and execute them using <code>run_tests()</code>.
 TODO(mbland): create some kind of automated script or editor macros?
 === Disable for Windows (for now) ===

@@ Line 60: / Line 60: @@
    ../util/shlib_wrap.sh ./$(HEARTBEATTEST)
-$(HEARTBEATTEST)$(EXE_EXT): $(HEARTBEATTEST).o $(DLIBCRYPTO)
+$(HEARTBEATTEST)$(EXE_EXT): $(HEARTBEATTEST).o $(DLIBCRYPTO) testutil.o
-   @target=$(HEARTBEATTEST); $(BUILD_CMD)
+   @target=$(HEARTBEATTEST) testutil=testutil.o; $(BUILD_CMD)
 </pre>

@@ Line 103: / Line 103: @@
 === <code>#include</code> the header for the code under test first ===
 Having the header file for the code under test appear as the first <code>#include</code> directive ensures that that file is self-contained, i.e. it includes every header file it depends on, rather than relying on client code to include its dependencies.
 === Define a fixture structure ===

← Older revision		Revision as of 20:28, 18 June 2014
Line 100:		Line 100:

	Some of the basic principles to follow are:		Some of the basic principles to follow are:
		+
		+	=== <code>#include</code> the header for the code under test first ===
		+	Having the header file for the code under test appear as the first <code>#include</code> directive ensures that that file is self-contained, i.e. it includes every header file it depends on, rather than relying on client code to include its dependencies.

	=== Define a fixture structure ===		=== Define a fixture structure ===

@@ Line 64: / Line 64: @@
 </pre>
-=== Run <code>make depend</code> ===
+=== Run <code>make links && make depend</code> ===
-Finally, run <code>make depend</code> to automatically generate the header file dependencies.
+Finally, run <code>make links && make depend</code> to link the new test into the <code>test/</code> directory and automatically generate the header file dependencies.
 === Building and Running the Test ===