How To Write Unit Tests For OpenSSL

This is an outline of the basic process and principles to follow when writing unit tests. This document will evolve quite a bit as the community gains experience.

Mechanics

Check out the Tools and Tips page

Testing and Development Tools and Tips has information on tools that may make navigating and building the code a bit easier.

Create a new test file in the same directory as the code under test using this template:

/*
 * TODO: Add Description
 *
 * Author:  $(git config --get user.name) ($(git config --get user.email))
 * Date:    $(date +%Y-%m-%d)
 *
 * ====================================================================
 * Copyright (c) $(date +%Y) The OpenSSL Project.  All rights reserved.
 * 
 * Redistribution and use in source and binary forms, with or without
 * modification, are permitted provided that the following conditions
 * are met:
 * 
 * 1. Redistributions of source code must retain the above copyright
 *    notice, this list of conditions and the following disclaimer.
 * 
 * 2. Redistributions in binary form must reproduce the above copyright
 *    notice, this list of conditions and the following disclaimer in
 *    the documentation and/or other materials provided with the
 *    distribution.
 * 
 * 3. All advertising materials mentioning features or use of this
 *    software must display the following acknowledgment:
 *    "This product includes software developed by the OpenSSL Project
 *    for use in the OpenSSL Toolkit. (http://www.OpenSSL.org/)"
 * 
 * 4. The names "OpenSSL Toolkit" and "OpenSSL Project" must not be used to
 *    endorse or promote products derived from this software without
 *    prior written permission. For written permission, please contact
 *    licensing@OpenSSL.org.
 * 
 * 5. Products derived from this software may not be called "OpenSSL"
 *    nor may "OpenSSL" appear in their names without prior written
 *    permission of the OpenSSL Project.
 * 
 * 6. Redistributions of any form whatsoever must retain the following
 *    acknowledgment:
 *    "This product includes software developed by the OpenSSL Project
 *    for use in the OpenSSL Toolkit (http://www.OpenSSL.org/)"
 * 
 * THIS SOFTWARE IS PROVIDED BY THE OpenSSL PROJECT ``AS IS'' AND ANY
 * EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
 * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
 * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE OpenSSL PROJECT OR
 * ITS CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
 * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
 * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
 * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
 * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
 * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
 * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
 * OF THE POSSIBILITY OF SUCH DAMAGE.
 * ====================================================================
 */

#define OPENSSL_UNIT_TEST

/* #include header for interface under test */

#include "testutil.h"
#include <ctype.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#if !defined(OPENSSL_NO_UNIT_TEST)

/* Add test code as per
 * http://wiki.openssl.org/index.php/How_To_Write_Unit_Tests_For_OpenSSL#Style
 */

typedef struct test_fixture
	{
	const char* test_case_name;
	} TEST_FIXTURE;

static TEST_FIXTURE set_up(const char* const test_case_name)
	{
	TEST_FIXTURE fixture;
	int setup_ok = 1;
	memset(&fixture, 0, sizeof(fixture));
	fixture.test_case_name = test_case_name;

	/* Allocate memory owned by the fixture, exit on error */

	if (!setup_ok)
		{
		ERR_print_errors_fp(stderr);
		exit(EXIT_FAILURE);
		}
	return fixture;
	}

static void tear_down(TEST_FIXTURE fixture)
	{
	ERR_print_errors_fp(stderr);
	/* Free any memory owned by the fixture, etc. */
	}

static int execute(TEST_FIXTURE fixture)
	{
	int result = 0;
	/* Execute the code under test, make assertions, format and print errors,
 	 * return zero on success and one on error */
	if (result != 0)
		{
		printf("** %s failed **\n--------\n", fixture.test_case_name);
		}
	return result;
	}

static int test_REPLACE_ME_WITH_A_MEANINGFUL_NAME()
	{
	SETUP_TEST_FIXTURE(TEST_FIXTURE, set_up);
	/* Do test case-specific set up; set expected return values and
 	 * side effects */
	EXECUTE_TEST(execute, tear_down);
	}

int main(int argc, char *argv[])
	{
	int result = 0;

	SSL_library_init();
	SSL_load_error_strings();

	ADD_TEST(test_REPLACE_ME_WITH_A_MEANINGFUL_NAME);

	result = run_tests(argv[0]);
	ERR_print_errors_fp(stderr);
	return result;
	}

#else /* OPENSSL_NO_UNIT_TEST*/

int main(int argc, char *argv[])
	{
	return EXIT_SUCCESS;
	}
#endif /* OPENSSL_NO_UNIT_TEST */

Add Makefile Targets

The following instructions use the Makefile targets for ssl/heartbeat_test.c as an example.

In the Makefile for the library containing the test, add the test source file to the TEST variable:

# ssl/Makefile
TEST=ssltest.c heartbeat_test.c

In test/Makefile:

• add a variable for the test target near the top of the file, right after the existing test variables
• use the variable to add an executable target to the EXE variable
• use the variable to add an object file target to the OBJ variable
• use the variable to add a source file target to the SRC variable
• add the test target to the alltests target
• add the target to execute the test
• add the target to build the test executable

# test/Makefile
HEARTBEATTEST=  heartbeat_test
EXE=  ... $(HEARTBEATTEST)$(EXE_EXT)
OBJ= ... $(HEARTBEATTEST).o
SRC= ... $(HEARTBEATTEST).c
alltests: \
        ... test_heartbeat

test_heartbeat: $(HEARTBEATTEST)$(EXE_EXT)
  ../util/shlib_wrap.sh ./$(HEARTBEATTEST)

$(HEARTBEATTEST)$(EXE_EXT): $(HEARTBEATTEST).o $(DLIBCRYPTO) testutil.o
  @target=$(HEARTBEATTEST) testutil=testutil.o; $(BUILD_CMD)

Run make links && make depend

Finally, run make links && make depend to link the new test into the test/ directory and automatically generate the header file dependencies.

Building and Running the Test

If you're initially developing on Mac OS X or (for now) FreeBSD 10, just use the stock method of building and testing:

$ ./config enable-unit-test && make && make test

# To execute just one specific test, where <test> is the basename of the test file:
$ make TESTS=<test> test 

Ultimately the test will have to compile and pass with developer flags enabled:

$ ./GitConfigure debug-ben-debug-64-clang enable-unit-test
$ ./GitMake -j 2
$ ./GitMake test_heartbeat

The above currently doesn't work on Mac OS X or FreeBSD > 9.1. The {,darwin64-}debug-test-64-clang Configure targets commit, which should go in soon as part of pull request #145, should solve the issues. Other commits from #145 contain other OS X-specific fixes.

Keep your repo up-to-date

Periodically run the following to keep your branch up-to-date:

$ git fetch upstream master
$ git rebase upstream/master

This will pull all the updates from the master OpenSSL repository into your repository, then update your branch to apply your changes on top of the latest updates.

Send a pull request

When your test is ready, send a GitHub pull request. We'll review the code, and when it's ready, it'll get merged into the repository.

Style

The Pseudo-xUnit Pattern pattern organizes code in a fashion reminiscent of the xUnit family of unit testing frameworks, without actually using a testing framework. This should lower the barrier to entry for people wanting to write unit tests, but enable a relatively easy migration to an xUnit-based framework if we decide to do so one day.

Some of the basic principles to follow are:

#include the header for the code under test first

Having the header file for the code under test appear as the first #include directive ensures that that file is self-contained, i.e. it includes every header file it depends on, rather than relying on client code to include its dependencies.

#include "testutil.h" should come second

test/testutil.h contains the helper macros used in writing OpenSSL tests. Since the tests will be linked into test/ by the make links step, and built in the test/ directory, the "testutil.h" file will appear to be in the same directory as the test file.

Define a fixture structure

The fixture structure should contain all of the inputs to the code under test and all of the expected result values. It should also contain a const char* for the name of the test case function that created it, to aid in error message formatting. Even though the fixture may contain dynamically-allocated members, the fixture itself should be copied by value to reduce the necessary degree of memory management in a small unit test program.

Define set_up() and tear_down() functions for the fixture

set_up() should return a newly-initialized test fixture structure. It should take the name of the test case as an argument (i.e. __func__) and assign it to the fixture. All of the fixture members should be initialized, which each test case function can then override as needed.

tear_down() should take the fixture as an argument and release any resources allocated by set_up(). It can also call any library-wide error printing routines (e.g. ERR_print_errors_fp(stderr)).

Use SETUP_TEST_FIXTURE() and EXECUTE_TEST() from test/testutil.h

Each test case function should call set_up() as its first statement, and should call tear_down() just before returning. This is handled in a uniform fashion when using the SETUP_TEST_FIXTURE() and EXECUTE_TEST() helper macros from test/testutil.h. See the comments in test/testutil.h for usage.

Use test case functions, not a table of fixtures

Individual test case functions that call a common execution function are much more readable and maintainable than a loop over a table of fixture structures. Explicit fixture variable assignments aid comprehension when reading a specific test case, which saves time and energy when trying to understand a test or diagnose a failure. When a new member is added to an existing fixture, set_up() can set a default for all test cases, and only the test cases that rely on that new member need to be updated.

Use very descriptive test case names

Give tests long, descriptive names that provide ample context for the details of the test case. Good test names are also help produce good error messages.

Group test cases into "suites" by naming convention

Give logically-related test functions the same prefix. If need be, you can define suite-specific set_up() functions that call the common set_up() and elaborate on it. (This generally shouldn't be necessary for tear_down().)

Keep individual test case functions focused on one thing

If the test name contains the word "and", consider breaking it into two or more separate test case functions.

Write very descriptive error messages

Include the test case function name in each error message, and explain in detail the context for the assertion that failed. Include the expected result (contained in the fixture structure) and the actual result returned from the code under test. Write helper methods for complex values as needed.

Return zero on success and one on failure

The return value will be used to tally the number of test cases that failed. Even if multiple assertions fail for a single test case, the result should be exactly one.

Register each test case using ADD_TEST() and execute using run_tests()

Whatever function is used as the test runner, be that main() or a separate function called by main(), add your test case functions to that function using ADD_TEST() and execute them using run_tests().

run_tests() will add up the total number of failed test cases and report that number as the last error message of the test. The return value of run_tests() should be the value returned from main(), which will be EXIT_FAILURE if any test cases failed, EXIT_SUCCESS otherwise.

Disable for Windows (for now)

Until we solve the private-symbol problem on Windows, we will need to wrap our unit test code in the following #ifdef block:

#if !defined(OPENSSL_SYS_WINDOWS)

/* All the test code, including main() */

int main(int argc, char *argv[])                                                
    {
        return EXIT_SUCCESS;                                                    
    }

#endif /* OPENSSL_NO_WINDOWS  */