Difference between revisions of "How To Write Unit Tests For OpenSSL"

From OpenSSLWiki
Jump to navigationJump to search
 
(10 intermediate revisions by 2 users not shown)
Line 2: Line 2:
  
 
==Mechanics ==
 
==Mechanics ==
=== Forking the Repo ===
 
For now, people contributing new unit tests for uncovered code (as opposed to submitting tests with new code changes) should fork [https://github.com/mbland/openssl Mike Bland's GitHub repository] and issue [https://help.github.com/articles/using-pull-requests GitHub pull requests]. Remember to do all development on a topic branch (not <code>master</code>). Tests committed to this repo will occasionally get merged into the master OpenSSL repo.
 
  
Rationale per Matt Caswell: We should think about process here. If you are going off to recruit an army of people writing tests then I wonder if it is worth setting up a separate github repository whilst you are building up the tests. We can then merge in from that on a periodic basis. I wouldn't want availability of openssl team committers to be a bottle neck.
+
=== Check out the Tools and Tips page ===
 +
[[Testing and Development Tools and Tips]] has information on tools that may make navigating and building the code a bit easier.
 +
 
 +
Create a new test file in the same directory as the code under test using this template:
  
Set up the [https://github.com/openssl/openssl OpenSSL master repository] as your upstream remote as per [https://help.github.com/articles/fork-a-repo#step-3-configure-remotes GitHub's instructions on configuring remotes]:
 
 
<pre>
 
<pre>
$ cd my-openssl-repo
+
/*
$ git remote add upstream https://github.com/openssl/openssl.git
+
* TODO: Add Description
</pre>
+
*
 +
* Author:  $(git config --get user.name) ($(git config --get user.email))
 +
* Date:    $(date +%Y-%m-%d)
 +
*
 +
* ====================================================================
 +
* Copyright (c) $(date +%Y) The OpenSSL Project.  All rights reserved.
 +
*
 +
* Redistribution and use in source and binary forms, with or without
 +
* modification, are permitted provided that the following conditions
 +
* are met:
 +
*
 +
* 1. Redistributions of source code must retain the above copyright
 +
*    notice, this list of conditions and the following disclaimer.
 +
*
 +
* 2. Redistributions in binary form must reproduce the above copyright
 +
*    notice, this list of conditions and the following disclaimer in
 +
*    the documentation and/or other materials provided with the
 +
*    distribution.
 +
*
 +
* 3. All advertising materials mentioning features or use of this
 +
*    software must display the following acknowledgment:
 +
*    "This product includes software developed by the OpenSSL Project
 +
*    for use in the OpenSSL Toolkit. (http://www.OpenSSL.org/)"
 +
*
 +
* 4. The names "OpenSSL Toolkit" and "OpenSSL Project" must not be used to
 +
*    endorse or promote products derived from this software without
 +
*    prior written permission. For written permission, please contact
 +
*    licensing@OpenSSL.org.
 +
*
 +
* 5. Products derived from this software may not be called "OpenSSL"
 +
*    nor may "OpenSSL" appear in their names without prior written
 +
*    permission of the OpenSSL Project.
 +
*
 +
* 6. Redistributions of any form whatsoever must retain the following
 +
*    acknowledgment:
 +
*    "This product includes software developed by the OpenSSL Project
 +
*    for use in the OpenSSL Toolkit (http://www.OpenSSL.org/)"
 +
*
 +
* THIS SOFTWARE IS PROVIDED BY THE OpenSSL PROJECT ``AS IS'' AND ANY
 +
* EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
 +
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
 +
* PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE OpenSSL PROJECT OR
 +
* ITS CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
 +
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
 +
* NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
 +
* LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
 +
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
 +
* STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
 +
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
 +
* OF THE POSSIBILITY OF SUCH DAMAGE.
 +
* ====================================================================
 +
*/
 +
 
 +
#define OPENSSL_UNIT_TEST
 +
 
 +
/* #include header for interface under test */
 +
 
 +
#include "testutil.h"
 +
#include <ctype.h>
 +
#include <stdio.h>
 +
#include <stdlib.h>
 +
#include <string.h>
 +
 
 +
#if !defined(OPENSSL_NO_UNIT_TEST)
 +
 
 +
/* Add test code as per
 +
* http://wiki.openssl.org/index.php/How_To_Write_Unit_Tests_For_OpenSSL#Style
 +
*/
 +
 
 +
typedef struct test_fixture
 +
{
 +
const char* test_case_name;
 +
} TEST_FIXTURE;
 +
 
 +
static TEST_FIXTURE set_up(const char* const test_case_name)
 +
{
 +
TEST_FIXTURE fixture;
 +
int setup_ok = 1;
 +
memset(&fixture, 0, sizeof(fixture));
 +
fixture.test_case_name = test_case_name;
 +
 
 +
/* Allocate memory owned by the fixture, exit on error */
 +
 
 +
if (!setup_ok)
 +
{
 +
ERR_print_errors_fp(stderr);
 +
exit(EXIT_FAILURE);
 +
}
 +
return fixture;
 +
}
 +
 
 +
static void tear_down(TEST_FIXTURE fixture)
 +
{
 +
ERR_print_errors_fp(stderr);
 +
/* Free any memory owned by the fixture, etc. */
 +
}
 +
 
 +
static int execute(TEST_FIXTURE fixture)
 +
{
 +
int result = 0;
 +
/* Execute the code under test, make assertions, format and print errors,
 +
* return zero on success and one on error */
 +
if (result != 0)
 +
{
 +
printf("** %s failed **\n--------\n", fixture.test_case_name);
 +
}
 +
return result;
 +
}
 +
 
 +
static int test_REPLACE_ME_WITH_A_MEANINGFUL_NAME()
 +
{
 +
SETUP_TEST_FIXTURE(TEST_FIXTURE, set_up);
 +
/* Do test case-specific set up; set expected return values and
 +
* side effects */
 +
EXECUTE_TEST(execute, tear_down);
 +
}
 +
 
 +
int main(int argc, char *argv[])
 +
{
 +
int result = 0;
 +
 
 +
SSL_library_init();
 +
SSL_load_error_strings();
  
You should see the following output from <code>git remove -v</code> (where <code>$USER</code> is your GitHub username):
+
ADD_TEST(test_REPLACE_ME_WITH_A_MEANINGFUL_NAME);
<pre>
 
$ git remote -v
 
origin  https://github.com/$USER/openssl.git (fetch)
 
origin  https://github.com/$USER/openssl.git (push)
 
upstream        https://github.com/openssl/openssl.git (fetch)
 
upstream        https://github.com/openssl/openssl.git (push)
 
</pre>
 
  
=== Check out the Tools and Tips page ===
+
result = run_tests(argv[0]);
[[Testing and Development Tools and Tips]] has information on tools that may make navigating and building the code a bit easier.
+
ERR_print_errors_fp(stderr);
 +
return result;
 +
}
  
=== Use the Test Template Generator ===
+
#else /* OPENSSL_NO_UNIT_TEST*/
TODO(mbland): Get template generator checked-in. Maybe have a template generator for each library, e.g. <code>ssl/new-test.sh</code> that has additional setup boilerplate specific to the <code>ssl</code> library.
 
  
Use the <code>test/new-test.sh</code> script to generate a skeleton test file. ([https://github.com/mbland/openssl/compare/test-util pending in the test-util branch])
+
int main(int argc, char *argv[])
 +
{
 +
return EXIT_SUCCESS;
 +
}
 +
#endif /* OPENSSL_NO_UNIT_TEST */
 +
</pre>
  
 
=== Add Makefile Targets ===
 
=== Add Makefile Targets ===
Line 60: Line 181:
 
   ../util/shlib_wrap.sh ./$(HEARTBEATTEST)
 
   ../util/shlib_wrap.sh ./$(HEARTBEATTEST)
  
$(HEARTBEATTEST)$(EXE_EXT): $(HEARTBEATTEST).o $(DLIBCRYPTO)
+
$(HEARTBEATTEST)$(EXE_EXT): $(HEARTBEATTEST).o $(DLIBCRYPTO) testutil.o
   @target=$(HEARTBEATTEST); $(BUILD_CMD)
+
   @target=$(HEARTBEATTEST) testutil=testutil.o; $(BUILD_CMD)
 
</pre>
 
</pre>
  
=== Run <code>make depend</code> ===
+
=== Run <code>make links && make depend</code> ===
Finally, run <code>make depend</code> to automatically generate the header file dependencies.
+
Finally, run <code>make links && make depend</code> to link the new test into the <code>test/</code> directory and automatically generate the header file dependencies.
  
 
=== Building and Running the Test ===
 
=== Building and Running the Test ===
 
If you're initially developing on Mac OS X or (for now) FreeBSD 10, just use the stock method of building and testing:
 
If you're initially developing on Mac OS X or (for now) FreeBSD 10, just use the stock method of building and testing:
 
<pre>
 
<pre>
$ ./config && make && make test
+
$ ./config enable-unit-test && make && make test
  
# To execute just one specific test
+
# To execute just one specific test, where <test> is the basename of the test file:
$ make TESTS=test_heartbeat test  
+
$ make TESTS=<test> test  
 
</pre>
 
</pre>
  
 
Ultimately the test will have to compile and pass with developer flags enabled:
 
Ultimately the test will have to compile and pass with developer flags enabled:
 
<pre>
 
<pre>
$ ./GitConfigure debug-ben-debug-64-clang
+
$ ./GitConfigure debug-ben-debug-64-clang enable-unit-test
 
$ ./GitMake -j 2
 
$ ./GitMake -j 2
 
$ ./GitMake test_heartbeat
 
$ ./GitMake test_heartbeat
 
</pre>
 
</pre>
  
The above currently doesn't work on Mac OS X or FreeBSD 10; for now, you can install the 9.1 release of [http://www.freebsd.org/ FreeBSD], which uses Clang 3.1, via a virtualization platform such as [https://www.virtualbox.org/ VirtualBox]. (Mac OS X breaks, in part, due to the fact that it doesn't use the GNU assembler; <code>-fsanitize</code> appears to be ignored on FreeBSD 9.1/Clang 3.1, but later versions break because the Clang compiler will emit <code>-fsanitize</code> symbols but the <code>libasan</code> library [http://lists.freebsd.org/pipermail/freebsd-hackers/2013-December/043995.html has yet to be ported to FreeBSD].)
+
The above currently doesn't work on Mac OS X or FreeBSD > 9.1. The [https://github.com/openssl/openssl/commit/a3984ff8a2e84132f221557bc26415314daf3259 {,darwin64-}debug-test-64-clang Configure targets] commit, which should go in soon as part of [https://github.com/openssl/openssl/pull/145 pull request #145], should solve the issues. Other commits from #145 contain other OS X-specific fixes.
  
 
=== Keep your repo up-to-date ===
 
=== Keep your repo up-to-date ===
Line 94: Line 215:
  
 
=== Send a pull request ===
 
=== Send a pull request ===
When your test is ready, send a [https://help.github.com/articles/using-pull-requests GitHub pull request]. We'll review the code, and when it's ready, it'll get merged into Mike's repository. From there, it will eventually get pulled into the master OpenSSL repository.
+
When your test is ready, send a [https://help.github.com/articles/using-pull-requests GitHub pull request]. We'll review the code, and when it's ready, it'll get merged into the repository.
  
 
== Style ==
 
== Style ==
[http://mike-bland.com/2014/06/05/pseudo-xunit-pattern.html The Pseudo-xUnit Pattern] is that established by [http://git.openssl.org/gitweb/?p=openssl.git;a=blob;f=ssl/heartbeat_test.c ssl/heartbeat_test.c]. This pattern organizes code in a fashion reminiscent of the [http://en.wikipedia.org/wiki/XUnit xUnit] family of unit testing frameworks, without actually using a testing framework. This should lower the barrier to entry for people wanting to write unit tests, but enable a relatively easy migration to an xUnit-based framework if we decide to do so one day.
+
[http://mike-bland.com/2014/06/05/pseudo-xunit-pattern.html The Pseudo-xUnit Pattern] pattern organizes code in a fashion reminiscent of the [http://en.wikipedia.org/wiki/XUnit xUnit] family of unit testing frameworks, without actually using a testing framework. This should lower the barrier to entry for people wanting to write unit tests, but enable a relatively easy migration to an xUnit-based framework if we decide to do so one day.
  
 
Some of the basic principles to follow are:
 
Some of the basic principles to follow are:
 +
 +
=== <code>#include</code> the header for the code under test first ===
 +
Having the header file for the code under test appear as the first <code>#include</code> directive ensures that that file is self-contained, i.e. it includes every header file it depends on, rather than relying on client code to include its dependencies.
 +
 +
=== <code>#include "testutil.h"</code> should come second ===
 +
<code>test/testutil.h</code> contains the helper macros used in writing OpenSSL tests. Since the tests will be linked into <code>test/</code> by the [[#Run_make_links_.26.26_make_depend|make links]] step, and built in the <code>test/</code> directory, the "testutil.h" file will appear to be in the same directory as the test file.
  
 
=== Define a fixture structure ===
 
=== Define a fixture structure ===
Line 110: Line 237:
  
 
=== Use <code>SETUP_TEST_FIXTURE()</code> and <code>EXECUTE_TEST()</code> from <code>test/testutil.h</code> ===
 
=== Use <code>SETUP_TEST_FIXTURE()</code> and <code>EXECUTE_TEST()</code> from <code>test/testutil.h</code> ===
Each test case function should call <code>set_up()</code> as its first statement, and should call <code>tear_down()</code> just before returning. This is handled in a uniform fashion when using the <code>SETUP_TEST_FIXTURE()</code> and <code>EXECUTE_TEST()</code> helper macros from [http://git.openssl.org/gitweb/?p=openssl.git;a=blob;f=test/testutil.h test/testutil.h]. See the comments in <code>test/testutil.h</code> for usage, and <code>ssl/heartbeat_test.c</code> for an example of how to wrap the macros for a specific unit test.
+
Each test case function should call <code>set_up()</code> as its first statement, and should call <code>tear_down()</code> just before returning. This is handled in a uniform fashion when using the <code>SETUP_TEST_FIXTURE()</code> and <code>EXECUTE_TEST()</code> helper macros from [http://git.openssl.org/gitweb/?p=openssl.git;a=blob;f=test/testutil.h test/testutil.h]. See the comments in <code>test/testutil.h</code> for usage.
  
 
=== Use test case functions, not a table of fixtures ===
 
=== Use test case functions, not a table of fixtures ===
Line 119: Line 246:
  
 
=== Group test cases into "suites" by naming convention ===
 
=== Group test cases into "suites" by naming convention ===
Give logically-related tests the same prefix, e.g. <code>test_dtls1_</code> and <code>test_tls1</code> from <code>ssl/heartbeat_test.c</code>. If need be, you can define suite-specific <code>set_up()</code> functions that call the common <code>set_up()</code> and elaborate on it. (This generally shouldn't be necessary for <code>tear_down()</code>.)
+
Give logically-related test functions the same prefix. If need be, you can define suite-specific <code>set_up()</code> functions that call the common <code>set_up()</code> and elaborate on it. (This generally shouldn't be necessary for <code>tear_down()</code>.)
  
 
=== Keep individual test case functions focused on one thing ===
 
=== Keep individual test case functions focused on one thing ===
Line 125: Line 252:
  
 
=== Write very descriptive error messages ===
 
=== Write very descriptive error messages ===
Include the test case function name in each error message, and explain in detail the context for the assertion that failed. Include the expected result (contained in the fixture structure) and the actual result returned from the code under test. Write helper methods for complex values as needed (e.g. <code>print_payload()</code> in <code>ssl/heartbeat_test.c</code>).
+
Include the test case function name in each error message, and explain in detail the context for the assertion that failed. Include the expected result (contained in the fixture structure) and the actual result returned from the code under test. Write helper methods for complex values as needed.
  
 
=== Return zero on success and one on failure ===
 
=== Return zero on success and one on failure ===
 
The return value will be used to tally the number of test cases that failed. Even if multiple assertions fail for a single test case, the result should be exactly one.
 
The return value will be used to tally the number of test cases that failed. Even if multiple assertions fail for a single test case, the result should be exactly one.
  
=== Add each test case function to the test runner function ===
+
=== Register each test case using ADD_TEST() and execute using run_tests() ===
Whatever function is used to execute the batch of test case functions, be that <code>main()</code> or a separate function called by <code>main()</code>, don't forget to add your test case functions to that function.
+
Whatever function is used as the test runner, be that <code>main()</code> or a separate function called by <code>main()</code>, add your test case functions to that function using <code>ADD_TEST()</code> and execute them using <code>run_tests()</code>.
  
TODO(mbland): create some kind of automated script or editor macros?
+
<code>run_tests()</code> will add up the total number of failed test cases and report that number as the last error message of the test. The return value of <code>run_tests()</code> should be the value returned from <code>main()</code>, which will be <code>EXIT_FAILURE</code> if any test cases failed, <code>EXIT_SUCCESS</code> otherwise.
 
 
=== Report the number of test cases that failed ===
 
Add up the total number of failed test cases in <code>main()</code> and report that number as the last error message of the test. <code>main()</code> should return <code>EXIT_FAILURE</code> if any test cases failed, <code>EXIT_SUCCESS</code> otherwise.
 
  
 
=== Disable for Windows (for now) ===
 
=== Disable for Windows (for now) ===
Line 152: Line 276:
 
#endif /* OPENSSL_NO_WINDOWS  */
 
#endif /* OPENSSL_NO_WINDOWS  */
 
</pre>
 
</pre>
 
== Samples ==
 
Existing tests that can be used as models for new tests.
 
* [http://git.openssl.org/gitweb/?p=openssl.git;a=blob;f=ssl/heartbeat_test.c ssl/heartbeat_test.c]: Follows [http://mike-bland.com/2014/06/05/pseudo-xunit-pattern.html the Pseudo-xUnit pattern]
 

Latest revision as of 11:03, 10 August 2017

This is an outline of the basic process and principles to follow when writing unit tests. This document will evolve quite a bit as the community gains experience.

Mechanics[edit]

Check out the Tools and Tips page[edit]

Testing and Development Tools and Tips has information on tools that may make navigating and building the code a bit easier.

Create a new test file in the same directory as the code under test using this template:

/*
 * TODO: Add Description
 *
 * Author:  $(git config --get user.name) ($(git config --get user.email))
 * Date:    $(date +%Y-%m-%d)
 *
 * ====================================================================
 * Copyright (c) $(date +%Y) The OpenSSL Project.  All rights reserved.
 * 
 * Redistribution and use in source and binary forms, with or without
 * modification, are permitted provided that the following conditions
 * are met:
 * 
 * 1. Redistributions of source code must retain the above copyright
 *    notice, this list of conditions and the following disclaimer.
 * 
 * 2. Redistributions in binary form must reproduce the above copyright
 *    notice, this list of conditions and the following disclaimer in
 *    the documentation and/or other materials provided with the
 *    distribution.
 * 
 * 3. All advertising materials mentioning features or use of this
 *    software must display the following acknowledgment:
 *    "This product includes software developed by the OpenSSL Project
 *    for use in the OpenSSL Toolkit. (http://www.OpenSSL.org/)"
 * 
 * 4. The names "OpenSSL Toolkit" and "OpenSSL Project" must not be used to
 *    endorse or promote products derived from this software without
 *    prior written permission. For written permission, please contact
 *    licensing@OpenSSL.org.
 * 
 * 5. Products derived from this software may not be called "OpenSSL"
 *    nor may "OpenSSL" appear in their names without prior written
 *    permission of the OpenSSL Project.
 * 
 * 6. Redistributions of any form whatsoever must retain the following
 *    acknowledgment:
 *    "This product includes software developed by the OpenSSL Project
 *    for use in the OpenSSL Toolkit (http://www.OpenSSL.org/)"
 * 
 * THIS SOFTWARE IS PROVIDED BY THE OpenSSL PROJECT ``AS IS'' AND ANY
 * EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
 * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
 * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE OpenSSL PROJECT OR
 * ITS CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
 * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
 * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
 * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
 * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
 * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
 * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
 * OF THE POSSIBILITY OF SUCH DAMAGE.
 * ====================================================================
 */

#define OPENSSL_UNIT_TEST

/* #include header for interface under test */

#include "testutil.h"
#include <ctype.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#if !defined(OPENSSL_NO_UNIT_TEST)

/* Add test code as per
 * http://wiki.openssl.org/index.php/How_To_Write_Unit_Tests_For_OpenSSL#Style
 */

typedef struct test_fixture
	{
	const char* test_case_name;
	} TEST_FIXTURE;

static TEST_FIXTURE set_up(const char* const test_case_name)
	{
	TEST_FIXTURE fixture;
	int setup_ok = 1;
	memset(&fixture, 0, sizeof(fixture));
	fixture.test_case_name = test_case_name;

	/* Allocate memory owned by the fixture, exit on error */

	if (!setup_ok)
		{
		ERR_print_errors_fp(stderr);
		exit(EXIT_FAILURE);
		}
	return fixture;
	}

static void tear_down(TEST_FIXTURE fixture)
	{
	ERR_print_errors_fp(stderr);
	/* Free any memory owned by the fixture, etc. */
	}

static int execute(TEST_FIXTURE fixture)
	{
	int result = 0;
	/* Execute the code under test, make assertions, format and print errors,
 	 * return zero on success and one on error */
	if (result != 0)
		{
		printf("** %s failed **\n--------\n", fixture.test_case_name);
		}
	return result;
	}

static int test_REPLACE_ME_WITH_A_MEANINGFUL_NAME()
	{
	SETUP_TEST_FIXTURE(TEST_FIXTURE, set_up);
	/* Do test case-specific set up; set expected return values and
 	 * side effects */
	EXECUTE_TEST(execute, tear_down);
	}

int main(int argc, char *argv[])
	{
	int result = 0;

	SSL_library_init();
	SSL_load_error_strings();

	ADD_TEST(test_REPLACE_ME_WITH_A_MEANINGFUL_NAME);

	result = run_tests(argv[0]);
	ERR_print_errors_fp(stderr);
	return result;
	}

#else /* OPENSSL_NO_UNIT_TEST*/

int main(int argc, char *argv[])
	{
	return EXIT_SUCCESS;
	}
#endif /* OPENSSL_NO_UNIT_TEST */

Add Makefile Targets[edit]

The following instructions use the Makefile targets for ssl/heartbeat_test.c as an example.

In the Makefile for the library containing the test, add the test source file to the TEST variable:

# ssl/Makefile
TEST=ssltest.c heartbeat_test.c

In test/Makefile:

  • add a variable for the test target near the top of the file, right after the existing test variables
  • use the variable to add an executable target to the EXE variable
  • use the variable to add an object file target to the OBJ variable
  • use the variable to add a source file target to the SRC variable
  • add the test target to the alltests target
  • add the target to execute the test
  • add the target to build the test executable
# test/Makefile
HEARTBEATTEST=  heartbeat_test
EXE=  ... $(HEARTBEATTEST)$(EXE_EXT)
OBJ= ... $(HEARTBEATTEST).o
SRC= ... $(HEARTBEATTEST).c
alltests: \
        ... test_heartbeat

test_heartbeat: $(HEARTBEATTEST)$(EXE_EXT)
  ../util/shlib_wrap.sh ./$(HEARTBEATTEST)

$(HEARTBEATTEST)$(EXE_EXT): $(HEARTBEATTEST).o $(DLIBCRYPTO) testutil.o
  @target=$(HEARTBEATTEST) testutil=testutil.o; $(BUILD_CMD)

Run make links && make depend[edit]

Finally, run make links && make depend to link the new test into the test/ directory and automatically generate the header file dependencies.

Building and Running the Test[edit]

If you're initially developing on Mac OS X or (for now) FreeBSD 10, just use the stock method of building and testing:

$ ./config enable-unit-test && make && make test

# To execute just one specific test, where <test> is the basename of the test file:
$ make TESTS=<test> test 

Ultimately the test will have to compile and pass with developer flags enabled:

$ ./GitConfigure debug-ben-debug-64-clang enable-unit-test
$ ./GitMake -j 2
$ ./GitMake test_heartbeat

The above currently doesn't work on Mac OS X or FreeBSD > 9.1. The {,darwin64-}debug-test-64-clang Configure targets commit, which should go in soon as part of pull request #145, should solve the issues. Other commits from #145 contain other OS X-specific fixes.

Keep your repo up-to-date[edit]

Periodically run the following to keep your branch up-to-date:

$ git fetch upstream master
$ git rebase upstream/master

This will pull all the updates from the master OpenSSL repository into your repository, then update your branch to apply your changes on top of the latest updates.

Send a pull request[edit]

When your test is ready, send a GitHub pull request. We'll review the code, and when it's ready, it'll get merged into the repository.

Style[edit]

The Pseudo-xUnit Pattern pattern organizes code in a fashion reminiscent of the xUnit family of unit testing frameworks, without actually using a testing framework. This should lower the barrier to entry for people wanting to write unit tests, but enable a relatively easy migration to an xUnit-based framework if we decide to do so one day.

Some of the basic principles to follow are:

#include the header for the code under test first[edit]

Having the header file for the code under test appear as the first #include directive ensures that that file is self-contained, i.e. it includes every header file it depends on, rather than relying on client code to include its dependencies.

#include "testutil.h" should come second[edit]

test/testutil.h contains the helper macros used in writing OpenSSL tests. Since the tests will be linked into test/ by the make links step, and built in the test/ directory, the "testutil.h" file will appear to be in the same directory as the test file.

Define a fixture structure[edit]

The fixture structure should contain all of the inputs to the code under test and all of the expected result values. It should also contain a const char* for the name of the test case function that created it, to aid in error message formatting. Even though the fixture may contain dynamically-allocated members, the fixture itself should be copied by value to reduce the necessary degree of memory management in a small unit test program.

Define set_up() and tear_down() functions for the fixture[edit]

set_up() should return a newly-initialized test fixture structure. It should take the name of the test case as an argument (i.e. __func__) and assign it to the fixture. All of the fixture members should be initialized, which each test case function can then override as needed.

tear_down() should take the fixture as an argument and release any resources allocated by set_up(). It can also call any library-wide error printing routines (e.g. ERR_print_errors_fp(stderr)).

Use SETUP_TEST_FIXTURE() and EXECUTE_TEST() from test/testutil.h[edit]

Each test case function should call set_up() as its first statement, and should call tear_down() just before returning. This is handled in a uniform fashion when using the SETUP_TEST_FIXTURE() and EXECUTE_TEST() helper macros from test/testutil.h. See the comments in test/testutil.h for usage.

Use test case functions, not a table of fixtures[edit]

Individual test case functions that call a common execution function are much more readable and maintainable than a loop over a table of fixture structures. Explicit fixture variable assignments aid comprehension when reading a specific test case, which saves time and energy when trying to understand a test or diagnose a failure. When a new member is added to an existing fixture, set_up() can set a default for all test cases, and only the test cases that rely on that new member need to be updated.

Use very descriptive test case names[edit]

Give tests long, descriptive names that provide ample context for the details of the test case. Good test names are also help produce good error messages.

Group test cases into "suites" by naming convention[edit]

Give logically-related test functions the same prefix. If need be, you can define suite-specific set_up() functions that call the common set_up() and elaborate on it. (This generally shouldn't be necessary for tear_down().)

Keep individual test case functions focused on one thing[edit]

If the test name contains the word "and", consider breaking it into two or more separate test case functions.

Write very descriptive error messages[edit]

Include the test case function name in each error message, and explain in detail the context for the assertion that failed. Include the expected result (contained in the fixture structure) and the actual result returned from the code under test. Write helper methods for complex values as needed.

Return zero on success and one on failure[edit]

The return value will be used to tally the number of test cases that failed. Even if multiple assertions fail for a single test case, the result should be exactly one.

Register each test case using ADD_TEST() and execute using run_tests()[edit]

Whatever function is used as the test runner, be that main() or a separate function called by main(), add your test case functions to that function using ADD_TEST() and execute them using run_tests().

run_tests() will add up the total number of failed test cases and report that number as the last error message of the test. The return value of run_tests() should be the value returned from main(), which will be EXIT_FAILURE if any test cases failed, EXIT_SUCCESS otherwise.

Disable for Windows (for now)[edit]

Until we solve the private-symbol problem on Windows, we will need to wrap our unit test code in the following #ifdef block:

#if !defined(OPENSSL_SYS_WINDOWS)

/* All the test code, including main() */

int main(int argc, char *argv[])                                                
    {
        return EXIT_SUCCESS;                                                    
    }

#endif /* OPENSSL_NO_WINDOWS  */