OpenSSLWiki - User contributions [en]

How To Write Unit Tests For OpenSSL

2017-08-10T11:03:07Z

Mbland:

This is an outline of the basic process and principles to follow when writing unit tests. This document will evolve quite a bit as the community gains experience.

==Mechanics ==

=== Check out the Tools and Tips page ===
[[Testing and Development Tools and Tips]] has information on tools that may make navigating and building the code a bit easier.

Create a new test file in the same directory as the code under test using this template:

<pre>
/*
* TODO: Add Description
*
* Author: $(git config --get user.name) ($(git config --get user.email))
* Date: $(date +%Y-%m-%d)
*
* ====================================================================
* Copyright (c) $(date +%Y) The OpenSSL Project. All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
*
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in
* the documentation and/or other materials provided with the
* distribution.
*
* 3. All advertising materials mentioning features or use of this
* software must display the following acknowledgment:
* "This product includes software developed by the OpenSSL Project
* for use in the OpenSSL Toolkit. (http://www.OpenSSL.org/)"
*
* 4. The names "OpenSSL Toolkit" and "OpenSSL Project" must not be used to
* endorse or promote products derived from this software without
* prior written permission. For written permission, please contact
* licensing@OpenSSL.org.
*
* 5. Products derived from this software may not be called "OpenSSL"
* nor may "OpenSSL" appear in their names without prior written
* permission of the OpenSSL Project.
*
* 6. Redistributions of any form whatsoever must retain the following
* acknowledgment:
* "This product includes software developed by the OpenSSL Project
* for use in the OpenSSL Toolkit (http://www.OpenSSL.org/)"
*
* THIS SOFTWARE IS PROVIDED BY THE OpenSSL PROJECT ``AS IS'' AND ANY
* EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
* PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE OpenSSL PROJECT OR
* ITS CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
* NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
* LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
* STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
* OF THE POSSIBILITY OF SUCH DAMAGE.
* ====================================================================
*/

#define OPENSSL_UNIT_TEST

/* #include header for interface under test */

#include "testutil.h"
#include <ctype.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#if !defined(OPENSSL_NO_UNIT_TEST)

/* Add test code as per
* http://wiki.openssl.org/index.php/How_To_Write_Unit_Tests_For_OpenSSL#Style
*/

typedef struct test_fixture
{
const char* test_case_name;
} TEST_FIXTURE;

static TEST_FIXTURE set_up(const char* const test_case_name)
{
TEST_FIXTURE fixture;
int setup_ok = 1;
memset(&fixture, 0, sizeof(fixture));
fixture.test_case_name = test_case_name;

/* Allocate memory owned by the fixture, exit on error */

if (!setup_ok)
{
ERR_print_errors_fp(stderr);
exit(EXIT_FAILURE);
}
return fixture;
}

static void tear_down(TEST_FIXTURE fixture)
{
ERR_print_errors_fp(stderr);
/* Free any memory owned by the fixture, etc. */
}

static int execute(TEST_FIXTURE fixture)
{
int result = 0;
/* Execute the code under test, make assertions, format and print errors,
* return zero on success and one on error */
if (result != 0)
{
printf("** %s failed **\n--------\n", fixture.test_case_name);
}
return result;
}

static int test_REPLACE_ME_WITH_A_MEANINGFUL_NAME()
{
SETUP_TEST_FIXTURE(TEST_FIXTURE, set_up);
/* Do test case-specific set up; set expected return values and
* side effects */
EXECUTE_TEST(execute, tear_down);
}

int main(int argc, char *argv[])
{
int result = 0;

SSL_library_init();
SSL_load_error_strings();

ADD_TEST(test_REPLACE_ME_WITH_A_MEANINGFUL_NAME);

result = run_tests(argv[0]);
ERR_print_errors_fp(stderr);
return result;
}

#else /* OPENSSL_NO_UNIT_TEST*/

int main(int argc, char *argv[])
{
return EXIT_SUCCESS;
}
#endif /* OPENSSL_NO_UNIT_TEST */
</pre>

=== Add Makefile Targets ===
The following instructions use the <code>Makefile</code> targets for <code>ssl/heartbeat_test.c</code> as an example.

In the <code>Makefile</code> for the library containing the test, add the test source file to the <code>TEST</code> variable:

<pre>
# ssl/Makefile
TEST=ssltest.c heartbeat_test.c
</pre>

In <code>test/Makefile</code>:
* add a variable for the test target near the top of the file, right after the existing test variables
* use the variable to add an executable target to the <code>EXE</code> variable
* use the variable to add an object file target to the <code>OBJ</code> variable
* use the variable to add a source file target to the <code>SRC</code> variable
* add the test target to the <code>alltests</code> target
* add the target to execute the test
* add the target to build the test executable
<pre>
# test/Makefile
HEARTBEATTEST= heartbeat_test
EXE= ... $(HEARTBEATTEST)$(EXE_EXT)
OBJ= ... $(HEARTBEATTEST).o
SRC= ... $(HEARTBEATTEST).c
alltests: \
... test_heartbeat

test_heartbeat: $(HEARTBEATTEST)$(EXE_EXT)
../util/shlib_wrap.sh ./$(HEARTBEATTEST)

$(HEARTBEATTEST)$(EXE_EXT): $(HEARTBEATTEST).o $(DLIBCRYPTO) testutil.o
@target=$(HEARTBEATTEST) testutil=testutil.o; $(BUILD_CMD)
</pre>

=== Run <code>make links && make depend</code> ===
Finally, run <code>make links && make depend</code> to link the new test into the <code>test/</code> directory and automatically generate the header file dependencies.

=== Building and Running the Test ===
If you're initially developing on Mac OS X or (for now) FreeBSD 10, just use the stock method of building and testing:
<pre>
$ ./config enable-unit-test && make && make test

# To execute just one specific test, where <test> is the basename of the test file:
$ make TESTS=<test> test
</pre>

Ultimately the test will have to compile and pass with developer flags enabled:
<pre>
$ ./GitConfigure debug-ben-debug-64-clang enable-unit-test
$ ./GitMake -j 2
$ ./GitMake test_heartbeat
</pre>

The above currently doesn't work on Mac OS X or FreeBSD > 9.1. The [https://github.com/openssl/openssl/commit/a3984ff8a2e84132f221557bc26415314daf3259 {,darwin64-}debug-test-64-clang Configure targets] commit, which should go in soon as part of [https://github.com/openssl/openssl/pull/145 pull request #145], should solve the issues. Other commits from #145 contain other OS X-specific fixes.

=== Keep your repo up-to-date ===
Periodically run the following to keep your branch up-to-date:
<pre>
$ git fetch upstream master
$ git rebase upstream/master
</pre>
This will pull all the updates from the master OpenSSL repository into your repository, then update your branch to apply your changes on top of the latest updates.

=== Send a pull request ===
When your test is ready, send a [https://help.github.com/articles/using-pull-requests GitHub pull request]. We'll review the code, and when it's ready, it'll get merged into the repository.

== Style ==
[http://mike-bland.com/2014/06/05/pseudo-xunit-pattern.html The Pseudo-xUnit Pattern] pattern organizes code in a fashion reminiscent of the [http://en.wikipedia.org/wiki/XUnit xUnit] family of unit testing frameworks, without actually using a testing framework. This should lower the barrier to entry for people wanting to write unit tests, but enable a relatively easy migration to an xUnit-based framework if we decide to do so one day.

Some of the basic principles to follow are:

=== <code>#include</code> the header for the code under test first ===
Having the header file for the code under test appear as the first <code>#include</code> directive ensures that that file is self-contained, i.e. it includes every header file it depends on, rather than relying on client code to include its dependencies.

=== <code>#include "testutil.h"</code> should come second ===
<code>test/testutil.h</code> contains the helper macros used in writing OpenSSL tests. Since the tests will be linked into <code>test/</code> by the [[#Run_make_links_.26.26_make_depend|make links]] step, and built in the <code>test/</code> directory, the "testutil.h" file will appear to be in the same directory as the test file.

=== Define a fixture structure ===
The fixture structure should contain all of the inputs to the code under test and all of the expected result values. It should also contain a <code>const char*</code> for the name of the test case function that created it, to aid in error message formatting. Even though the fixture may contain dynamically-allocated members, the fixture itself should be copied by value to reduce the necessary degree of memory management in a small unit test program.

=== Define set_up() and tear_down() functions for the fixture ===
<code>set_up()</code> should return a newly-initialized test fixture structure. It should take the name of the test case as an argument (i.e. <code>__func__</code>) and assign it to the fixture. All of the fixture members should be initialized, which each test case function can then override as needed.

<code>tear_down()</code> should take the fixture as an argument and release any resources allocated by <code>set_up()</code>. It can also call any library-wide error printing routines (e.g. <code>ERR_print_errors_fp(stderr)</code>).

=== Use <code>SETUP_TEST_FIXTURE()</code> and <code>EXECUTE_TEST()</code> from <code>test/testutil.h</code> ===
Each test case function should call <code>set_up()</code> as its first statement, and should call <code>tear_down()</code> just before returning. This is handled in a uniform fashion when using the <code>SETUP_TEST_FIXTURE()</code> and <code>EXECUTE_TEST()</code> helper macros from [http://git.openssl.org/gitweb/?p=openssl.git;a=blob;f=test/testutil.h test/testutil.h]. See the comments in <code>test/testutil.h</code> for usage.

=== Use test case functions, not a table of fixtures ===
Individual test case functions that call a common execution function are much more readable and maintainable than a loop over a table of fixture structures. Explicit fixture variable assignments aid comprehension when reading a specific test case, which saves time and energy when trying to understand a test or diagnose a failure. When a new member is added to an existing fixture, <code>set_up()</code> can set a default for all test cases, and only the test cases that rely on that new member need to be updated.

=== Use very descriptive test case names ===
Give tests long, descriptive names that provide ample context for the details of the test case. Good test names are also help produce good error messages.

=== Group test cases into "suites" by naming convention ===
Give logically-related test functions the same prefix. If need be, you can define suite-specific <code>set_up()</code> functions that call the common <code>set_up()</code> and elaborate on it. (This generally shouldn't be necessary for <code>tear_down()</code>.)

=== Keep individual test case functions focused on one thing ===
If the test name contains the word "and", consider breaking it into two or more separate test case functions.

=== Write very descriptive error messages ===
Include the test case function name in each error message, and explain in detail the context for the assertion that failed. Include the expected result (contained in the fixture structure) and the actual result returned from the code under test. Write helper methods for complex values as needed.

=== Return zero on success and one on failure ===
The return value will be used to tally the number of test cases that failed. Even if multiple assertions fail for a single test case, the result should be exactly one.

=== Register each test case using ADD_TEST() and execute using run_tests() ===
Whatever function is used as the test runner, be that <code>main()</code> or a separate function called by <code>main()</code>, add your test case functions to that function using <code>ADD_TEST()</code> and execute them using <code>run_tests()</code>.

<code>run_tests()</code> will add up the total number of failed test cases and report that number as the last error message of the test. The return value of <code>run_tests()</code> should be the value returned from <code>main()</code>, which will be <code>EXIT_FAILURE</code> if any test cases failed, <code>EXIT_SUCCESS</code> otherwise.

=== Disable for Windows (for now) ===
Until we solve the private-symbol problem on Windows, we will need to wrap our unit test code in the following <code>#ifdef</code> block:
<pre>
#if !defined(OPENSSL_SYS_WINDOWS)

/* All the test code, including main() */

int main(int argc, char *argv[])
{
return EXIT_SUCCESS;
}

#endif /* OPENSSL_NO_WINDOWS */
</pre>

How To Write Unit Tests For OpenSSL

2014-08-21T14:15:22Z

Mbland: /* Use the Test Template Generator */ Revamped as "Create a new test file using the test template"

This is an outline of the basic process and principles to follow when writing unit tests. This document will evolve quite a bit as the community gains experience.

==Mechanics ==
=== Forking the Repo ===

For now, people contributing new unit tests for uncovered code (as opposed to submitting tests with new code changes) should fork [https://github.com/mbland/openssl Mike Bland's GitHub repository] and issue changes via [https://help.github.com/articles/using-pull-requests GitHub pull requests] to this repo.

Remember to do all development on a topic branch (not <code>master</code>). Tests committed to this repo will occasionally get merged into the master OpenSSL repo.

Be sure to set up the [https://github.com/openssl/openssl OpenSSL master repository] as your upstream remote as per [https://help.github.com/articles/fork-a-repo#step-3-configure-remotes GitHub's instructions on configuring remotes], and pull in upstream changes frequently.

<pre>
$ cd my-openssl-repo
$ git remote add upstream https://github.com/openssl/openssl.git
</pre>

You should see the following output from <code>git remove -v</code> (where <code>$USER</code> is your GitHub username):
<pre>
$ git remote -v
origin https://github.com/$USER/openssl.git (fetch)
origin https://github.com/$USER/openssl.git (push)
upstream https://github.com/openssl/openssl.git (fetch)
upstream https://github.com/openssl/openssl.git (push)
</pre>

Rationale per Matt Caswell: We should think about process here. If you are going off to recruit an army of people writing tests then I wonder if it is worth setting up a separate github repository whilst you are building up the tests. We can then merge in from that on a periodic basis. I wouldn't want availability of openssl team committers to be a bottle neck.

=== Check out the Tools and Tips page ===
[[Testing and Development Tools and Tips]] has information on tools that may make navigating and building the code a bit easier.

=== Create a new test file using the test template ===
TODO(mbland): Get <code>test/new-test.sh</code> test template generator checked-in ([https://github.com/mbland/openssl/compare/test-util pending in the test-util branch]). Maybe have a template generator for each library, e.g. <code>ssl/new-test.sh</code> that has additional setup boilerplate specific to the <code>ssl</code> library.

Create a new test file in the same directory as the code under test using this template; see [http://git.openssl.org/gitweb/?p=openssl.git;a=blob;f=ssl/heartbeat_test.c ssl/heartbeat_test.c] for a full example:

<pre>
/*
* TODO: Add Description
*
* Author: $(git config --get user.name) ($(git config --get user.email))
* Date: $(date +%Y-%m-%d)
*
* ====================================================================
* Copyright (c) $(date +%Y) The OpenSSL Project. All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
*
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in
* the documentation and/or other materials provided with the
* distribution.
*
* 3. All advertising materials mentioning features or use of this
* software must display the following acknowledgment:
* "This product includes software developed by the OpenSSL Project
* for use in the OpenSSL Toolkit. (http://www.OpenSSL.org/)"
*
* 4. The names "OpenSSL Toolkit" and "OpenSSL Project" must not be used to
* endorse or promote products derived from this software without
* prior written permission. For written permission, please contact
* licensing@OpenSSL.org.
*
* 5. Products derived from this software may not be called "OpenSSL"
* nor may "OpenSSL" appear in their names without prior written
* permission of the OpenSSL Project.
*
* 6. Redistributions of any form whatsoever must retain the following
* acknowledgment:
* "This product includes software developed by the OpenSSL Project
* for use in the OpenSSL Toolkit (http://www.OpenSSL.org/)"
*
* THIS SOFTWARE IS PROVIDED BY THE OpenSSL PROJECT ``AS IS'' AND ANY
* EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
* PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE OpenSSL PROJECT OR
* ITS CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
* NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
* LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
* STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
* OF THE POSSIBILITY OF SUCH DAMAGE.
* ====================================================================
*/

#define OPENSSL_UNIT_TEST

/* #include header for interface under test */

#include "testutil.h"
#include <ctype.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#if !defined(OPENSSL_NO_UNIT_TEST)

/* Add test code as per
* http://wiki.openssl.org/index.php/How_To_Write_Unit_Tests_For_OpenSSL#Style
*/

typedef struct test_fixture
{
const char* test_case_name;
} TEST_FIXTURE;

static TEST_FIXTURE set_up(const char* const test_case_name)
{
TEST_FIXTURE fixture;
int setup_ok = 1;
memset(&fixture, 0, sizeof(fixture));
fixture.test_case_name = test_case_name;

/* Allocate memory owned by the fixture, exit on error */

if (!setup_ok)
{
ERR_print_errors_fp(stderr);
exit(EXIT_FAILURE);
}
return fixture;
}

static void tear_down(TEST_FIXTURE fixture)
{
ERR_print_errors_fp(stderr);
/* Free any memory owned by the fixture, etc. */
}

static int execute(TEST_FIXTURE fixture)
{
int result = 0;
/* Execute the code under test, make assertions, format and print errors,
* return zero on success and one on error */
if (result != 0)
{
printf("** %s failed **\n--------\n", fixture.test_case_name);
}
return result;
}

static int test_REPLACE_ME_WITH_A_MEANINGFUL_NAME()
{
SETUP_TEST_FIXTURE(TEST_FIXTURE, set_up);
/* Do test case-specific set up; set expected return values and
* side effects */
EXECUTE_TEST(execute, tear_down);
}

int main(int argc, char *argv[])
{
int result = 0;

SSL_library_init();
SSL_load_error_strings();

ADD_TEST(test_REPLACE_ME_WITH_A_MEANINGFUL_NAME);

result = run_tests(argv[0]);
ERR_print_errors_fp(stderr);
return result;
}

#else /* OPENSSL_NO_UNIT_TEST*/

int main(int argc, char *argv[])
{
return EXIT_SUCCESS;
}
#endif /* OPENSSL_NO_UNIT_TEST */
</pre>

=== Add Makefile Targets ===
The following instructions use the <code>Makefile</code> targets for <code>ssl/heartbeat_test.c</code> as an example.

In the <code>Makefile</code> for the library containing the test, add the test source file to the <code>TEST</code> variable:

<pre>
# ssl/Makefile
TEST=ssltest.c heartbeat_test.c
</pre>

In <code>test/Makefile</code>:
* add a variable for the test target near the top of the file, right after the existing test variables
* use the variable to add an executable target to the <code>EXE</code> variable
* use the variable to add an object file target to the <code>OBJ</code> variable
* use the variable to add a source file target to the <code>SRC</code> variable
* add the test target to the <code>alltests</code> target
* add the target to execute the test
* add the target to build the test executable
<pre>
# test/Makefile
HEARTBEATTEST= heartbeat_test
EXE= ... $(HEARTBEATTEST)$(EXE_EXT)
OBJ= ... $(HEARTBEATTEST).o
SRC= ... $(HEARTBEATTEST).c
alltests: \
... test_heartbeat

test_heartbeat: $(HEARTBEATTEST)$(EXE_EXT)
../util/shlib_wrap.sh ./$(HEARTBEATTEST)

$(HEARTBEATTEST)$(EXE_EXT): $(HEARTBEATTEST).o $(DLIBCRYPTO) testutil.o
@target=$(HEARTBEATTEST) testutil=testutil.o; $(BUILD_CMD)
</pre>

=== Run <code>make links && make depend</code> ===
Finally, run <code>make links && make depend</code> to link the new test into the <code>test/</code> directory and automatically generate the header file dependencies.

=== Building and Running the Test ===
If you're initially developing on Mac OS X or (for now) FreeBSD 10, just use the stock method of building and testing:
<pre>
$ ./config enable-unit-test && make && make test

# To execute just one specific test
$ make TESTS=test_heartbeat test
</pre>

Ultimately the test will have to compile and pass with developer flags enabled:
<pre>
$ ./GitConfigure debug-ben-debug-64-clang enable-unit-test
$ ./GitMake -j 2
$ ./GitMake test_heartbeat
</pre>

The above currently doesn't work on Mac OS X or FreeBSD > 9.1. The [https://github.com/mbland/openssl/commit/a3984ff8a2e84132f221557bc26415314daf3259 {,darwin64-}debug-test-64-clang Configure targets] commit, which should go in soon as part of [https://github.com/openssl/openssl/pull/145 pull request #145], should solve the issues. Other commits from #145 contain other OS X-specific fixes.

=== Keep your repo up-to-date ===
Periodically run the following to keep your branch up-to-date:
<pre>
$ git fetch upstream master
$ git rebase upstream/master
</pre>
This will pull all the updates from the master OpenSSL repository into your repository, then update your branch to apply your changes on top of the latest updates.

=== Send a pull request ===
When your test is ready, send a [https://help.github.com/articles/using-pull-requests GitHub pull request]. Note that the pull request should be based on the <code>tests</code> branch of Mike's repository, not <code>master</code>. (This should be the default; let Mike know if it doesn't appear to be!) We'll review the code, and when it's ready, it'll get merged into Mike's repository. From there, it will eventually get pulled into the master OpenSSL repository.

== Style ==
[http://mike-bland.com/2014/06/05/pseudo-xunit-pattern.html The Pseudo-xUnit Pattern] is that established by [http://git.openssl.org/gitweb/?p=openssl.git;a=blob;f=ssl/heartbeat_test.c ssl/heartbeat_test.c]. This pattern organizes code in a fashion reminiscent of the [http://en.wikipedia.org/wiki/XUnit xUnit] family of unit testing frameworks, without actually using a testing framework. This should lower the barrier to entry for people wanting to write unit tests, but enable a relatively easy migration to an xUnit-based framework if we decide to do so one day.

Some of the basic principles to follow are:

=== <code>#include</code> the header for the code under test first ===
Having the header file for the code under test appear as the first <code>#include</code> directive ensures that that file is self-contained, i.e. it includes every header file it depends on, rather than relying on client code to include its dependencies.

=== <code>#include "testutil.h"</code> should come second ===
<code>test/testutil.h</code> contains the helper macros used in writing OpenSSL tests. Since the tests will be linked into <code>test/</code> by the [[#Run_make_links_.26.26_make_depend|make links]] step, and built in the <code>test/</code> directory, the "testutil.h" file will appear to be in the same directory as the test file.

=== Define a fixture structure ===
The fixture structure should contain all of the inputs to the code under test and all of the expected result values. It should also contain a <code>const char*</code> for the name of the test case function that created it, to aid in error message formatting. Even though the fixture may contain dynamically-allocated members, the fixture itself should be copied by value to reduce the necessary degree of memory management in a small unit test program.

=== Define set_up() and tear_down() functions for the fixture ===
<code>set_up()</code> should return a newly-initialized test fixture structure. It should take the name of the test case as an argument (i.e. <code>__func__</code>) and assign it to the fixture. All of the fixture members should be initialized, which each test case function can then override as needed.

<code>tear_down()</code> should take the fixture as an argument and release any resources allocated by <code>set_up()</code>. It can also call any library-wide error printing routines (e.g. <code>ERR_print_errors_fp(stderr)</code>).

=== Use <code>SETUP_TEST_FIXTURE()</code> and <code>EXECUTE_TEST()</code> from <code>test/testutil.h</code> ===
Each test case function should call <code>set_up()</code> as its first statement, and should call <code>tear_down()</code> just before returning. This is handled in a uniform fashion when using the <code>SETUP_TEST_FIXTURE()</code> and <code>EXECUTE_TEST()</code> helper macros from [http://git.openssl.org/gitweb/?p=openssl.git;a=blob;f=test/testutil.h test/testutil.h]. See the comments in <code>test/testutil.h</code> for usage, and <code>ssl/heartbeat_test.c</code> for an example of how to wrap the macros for a specific unit test.

=== Use test case functions, not a table of fixtures ===
Individual test case functions that call a common execution function are much more readable and maintainable than a loop over a table of fixture structures. Explicit fixture variable assignments aid comprehension when reading a specific test case, which saves time and energy when trying to understand a test or diagnose a failure. When a new member is added to an existing fixture, <code>set_up()</code> can set a default for all test cases, and only the test cases that rely on that new member need to be updated.

=== Use very descriptive test case names ===
Give tests long, descriptive names that provide ample context for the details of the test case. Good test names are also help produce good error messages.

=== Group test cases into "suites" by naming convention ===
Give logically-related tests the same prefix, e.g. <code>test_dtls1_</code> and <code>test_tls1</code> from <code>ssl/heartbeat_test.c</code>. If need be, you can define suite-specific <code>set_up()</code> functions that call the common <code>set_up()</code> and elaborate on it. (This generally shouldn't be necessary for <code>tear_down()</code>.)

=== Keep individual test case functions focused on one thing ===
If the test name contains the word "and", consider breaking it into two or more separate test case functions.

=== Write very descriptive error messages ===
Include the test case function name in each error message, and explain in detail the context for the assertion that failed. Include the expected result (contained in the fixture structure) and the actual result returned from the code under test. Write helper methods for complex values as needed (e.g. <code>print_payload()</code> in <code>ssl/heartbeat_test.c</code>).

=== Return zero on success and one on failure ===
The return value will be used to tally the number of test cases that failed. Even if multiple assertions fail for a single test case, the result should be exactly one.

=== Register each test case using ADD_TEST() and execute using run_tests() ===
Whatever function is used as the test runner, be that <code>main()</code> or a separate function called by <code>main()</code>, add your test case functions to that function using <code>ADD_TEST()</code> and execute them using <code>run_tests()</code>.

<code>run_tests()</code> will add up the total number of failed test cases and report that number as the last error message of the test. The return value of <code>run_tests()</code> should be the value returned from <code>main()</code>, which will be <code>EXIT_FAILURE</code> if any test cases failed, <code>EXIT_SUCCESS</code> otherwise.

TODO(mbland): create some kind of automated script or editor macros?

=== Disable for Windows (for now) ===
Until we solve the private-symbol problem on Windows, we will need to wrap our unit test code in the following <code>#ifdef</code> block:
<pre>
#if !defined(OPENSSL_SYS_WINDOWS)

/* All the test code, including main() */

int main(int argc, char *argv[])
{
return EXIT_SUCCESS;
}

#endif /* OPENSSL_NO_WINDOWS */
</pre>

== Samples ==
Existing tests that can be used as models for new tests.
* [http://git.openssl.org/gitweb/?p=openssl.git;a=blob;f=ssl/heartbeat_test.c ssl/heartbeat_test.c]: Follows [http://mike-bland.com/2014/06/05/pseudo-xunit-pattern.html the Pseudo-xUnit pattern]

How To Write Unit Tests For OpenSSL

2014-08-02T18:28:24Z

Mbland: /* Building and Running the Test */

This is an outline of the basic process and principles to follow when writing unit tests. This document will evolve quite a bit as the community gains experience.

==Mechanics ==
=== Forking the Repo ===
For now, people contributing new unit tests for uncovered code (as opposed to submitting tests with new code changes) should fork [https://github.com/mbland/openssl Mike Bland's GitHub repository] and issue [https://help.github.com/articles/using-pull-requests GitHub pull requests]. Remember to do all development on a topic branch (not <code>master</code>). Tests committed to this repo will occasionally get merged into the master OpenSSL repo.

Rationale per Matt Caswell: We should think about process here. If you are going off to recruit an army of people writing tests then I wonder if it is worth setting up a separate github repository whilst you are building up the tests. We can then merge in from that on a periodic basis. I wouldn't want availability of openssl team committers to be a bottle neck.

Set up the [https://github.com/openssl/openssl OpenSSL master repository] as your upstream remote as per [https://help.github.com/articles/fork-a-repo#step-3-configure-remotes GitHub's instructions on configuring remotes]:
<pre>
$ cd my-openssl-repo
$ git remote add upstream https://github.com/openssl/openssl.git
</pre>

You should see the following output from <code>git remove -v</code> (where <code>$USER</code> is your GitHub username):
<pre>
$ git remote -v
origin https://github.com/$USER/openssl.git (fetch)
origin https://github.com/$USER/openssl.git (push)
upstream https://github.com/openssl/openssl.git (fetch)
upstream https://github.com/openssl/openssl.git (push)
</pre>

=== Check out the Tools and Tips page ===
[[Testing and Development Tools and Tips]] has information on tools that may make navigating and building the code a bit easier.

=== Use the Test Template Generator ===
TODO(mbland): Get template generator checked-in. Maybe have a template generator for each library, e.g. <code>ssl/new-test.sh</code> that has additional setup boilerplate specific to the <code>ssl</code> library.

Use the <code>test/new-test.sh</code> script to generate a skeleton test file. ([https://github.com/mbland/openssl/compare/test-util pending in the test-util branch])

=== Add Makefile Targets ===
The following instructions use the <code>Makefile</code> targets for <code>ssl/heartbeat_test.c</code> as an example.

In the <code>Makefile</code> for the library containing the test, add the test source file to the <code>TEST</code> variable:

<pre>
# ssl/Makefile
TEST=ssltest.c heartbeat_test.c
</pre>

In <code>test/Makefile</code>:
* add a variable for the test target near the top of the file, right after the existing test variables
* use the variable to add an executable target to the <code>EXE</code> variable
* use the variable to add an object file target to the <code>OBJ</code> variable
* use the variable to add a source file target to the <code>SRC</code> variable
* add the test target to the <code>alltests</code> target
* add the target to execute the test
* add the target to build the test executable
<pre>
# test/Makefile
HEARTBEATTEST= heartbeat_test
EXE= ... $(HEARTBEATTEST)$(EXE_EXT)
OBJ= ... $(HEARTBEATTEST).o
SRC= ... $(HEARTBEATTEST).c
alltests: \
... test_heartbeat

test_heartbeat: $(HEARTBEATTEST)$(EXE_EXT)
../util/shlib_wrap.sh ./$(HEARTBEATTEST)

$(HEARTBEATTEST)$(EXE_EXT): $(HEARTBEATTEST).o $(DLIBCRYPTO) testutil.o
@target=$(HEARTBEATTEST) testutil=testutil.o; $(BUILD_CMD)
</pre>

=== Run <code>make links && make depend</code> ===
Finally, run <code>make links && make depend</code> to link the new test into the <code>test/</code> directory and automatically generate the header file dependencies.

=== Building and Running the Test ===
If you're initially developing on Mac OS X or (for now) FreeBSD 10, just use the stock method of building and testing:
<pre>
$ ./config enable-unit-test && make && make test

# To execute just one specific test
$ make TESTS=test_heartbeat test
</pre>

Ultimately the test will have to compile and pass with developer flags enabled:
<pre>
$ ./GitConfigure debug-ben-debug-64-clang enable-unit-test
$ ./GitMake -j 2
$ ./GitMake test_heartbeat
</pre>

The above currently doesn't work on Mac OS X or FreeBSD > 9.1. The [https://github.com/mbland/openssl/commit/a3984ff8a2e84132f221557bc26415314daf3259 {,darwin64-}debug-test-64-clang Configure targets] commit, which should go in soon as part of [https://github.com/openssl/openssl/pull/145 pull request #145], should solve the issues. Other commits from #145 contain other OS X-specific fixes.

=== Keep your repo up-to-date ===
Periodically run the following to keep your branch up-to-date:
<pre>
$ git fetch upstream master
$ git rebase upstream/master
</pre>
This will pull all the updates from the master OpenSSL repository into your repository, then update your branch to apply your changes on top of the latest updates.

=== Send a pull request ===
When your test is ready, send a [https://help.github.com/articles/using-pull-requests GitHub pull request]. Note that the pull request should be based on the <code>tests</code> branch of Mike's repository, not <code>master</code>. (This should be the default; let Mike know if it doesn't appear to be!) We'll review the code, and when it's ready, it'll get merged into Mike's repository. From there, it will eventually get pulled into the master OpenSSL repository.

== Style ==
[http://mike-bland.com/2014/06/05/pseudo-xunit-pattern.html The Pseudo-xUnit Pattern] is that established by [http://git.openssl.org/gitweb/?p=openssl.git;a=blob;f=ssl/heartbeat_test.c ssl/heartbeat_test.c]. This pattern organizes code in a fashion reminiscent of the [http://en.wikipedia.org/wiki/XUnit xUnit] family of unit testing frameworks, without actually using a testing framework. This should lower the barrier to entry for people wanting to write unit tests, but enable a relatively easy migration to an xUnit-based framework if we decide to do so one day.

Some of the basic principles to follow are:

=== <code>#include</code> the header for the code under test first ===
Having the header file for the code under test appear as the first <code>#include</code> directive ensures that that file is self-contained, i.e. it includes every header file it depends on, rather than relying on client code to include its dependencies.

=== <code>#include "testutil.h"</code> should come second ===
<code>test/testutil.h</code> contains the helper macros used in writing OpenSSL tests. Since the tests will be linked into <code>test/</code> by the [[#Run_make_links_.26.26_make_depend|make links]] step, and built in the <code>test/</code> directory, the "testutil.h" file will appear to be in the same directory as the test file.

=== Define a fixture structure ===
The fixture structure should contain all of the inputs to the code under test and all of the expected result values. It should also contain a <code>const char*</code> for the name of the test case function that created it, to aid in error message formatting. Even though the fixture may contain dynamically-allocated members, the fixture itself should be copied by value to reduce the necessary degree of memory management in a small unit test program.

=== Define set_up() and tear_down() functions for the fixture ===
<code>set_up()</code> should return a newly-initialized test fixture structure. It should take the name of the test case as an argument (i.e. <code>__func__</code>) and assign it to the fixture. All of the fixture members should be initialized, which each test case function can then override as needed.

<code>tear_down()</code> should take the fixture as an argument and release any resources allocated by <code>set_up()</code>. It can also call any library-wide error printing routines (e.g. <code>ERR_print_errors_fp(stderr)</code>).

=== Use <code>SETUP_TEST_FIXTURE()</code> and <code>EXECUTE_TEST()</code> from <code>test/testutil.h</code> ===
Each test case function should call <code>set_up()</code> as its first statement, and should call <code>tear_down()</code> just before returning. This is handled in a uniform fashion when using the <code>SETUP_TEST_FIXTURE()</code> and <code>EXECUTE_TEST()</code> helper macros from [http://git.openssl.org/gitweb/?p=openssl.git;a=blob;f=test/testutil.h test/testutil.h]. See the comments in <code>test/testutil.h</code> for usage, and <code>ssl/heartbeat_test.c</code> for an example of how to wrap the macros for a specific unit test.

=== Use test case functions, not a table of fixtures ===
Individual test case functions that call a common execution function are much more readable and maintainable than a loop over a table of fixture structures. Explicit fixture variable assignments aid comprehension when reading a specific test case, which saves time and energy when trying to understand a test or diagnose a failure. When a new member is added to an existing fixture, <code>set_up()</code> can set a default for all test cases, and only the test cases that rely on that new member need to be updated.

=== Use very descriptive test case names ===
Give tests long, descriptive names that provide ample context for the details of the test case. Good test names are also help produce good error messages.

=== Group test cases into "suites" by naming convention ===
Give logically-related tests the same prefix, e.g. <code>test_dtls1_</code> and <code>test_tls1</code> from <code>ssl/heartbeat_test.c</code>. If need be, you can define suite-specific <code>set_up()</code> functions that call the common <code>set_up()</code> and elaborate on it. (This generally shouldn't be necessary for <code>tear_down()</code>.)

=== Keep individual test case functions focused on one thing ===
If the test name contains the word "and", consider breaking it into two or more separate test case functions.

=== Write very descriptive error messages ===
Include the test case function name in each error message, and explain in detail the context for the assertion that failed. Include the expected result (contained in the fixture structure) and the actual result returned from the code under test. Write helper methods for complex values as needed (e.g. <code>print_payload()</code> in <code>ssl/heartbeat_test.c</code>).

=== Return zero on success and one on failure ===
The return value will be used to tally the number of test cases that failed. Even if multiple assertions fail for a single test case, the result should be exactly one.

=== Register each test case using ADD_TEST() and execute using run_tests() ===
Whatever function is used as the test runner, be that <code>main()</code> or a separate function called by <code>main()</code>, add your test case functions to that function using <code>ADD_TEST()</code> and execute them using <code>run_tests()</code>.

<code>run_tests()</code> will add up the total number of failed test cases and report that number as the last error message of the test. The return value of <code>run_tests()</code> should be the value returned from <code>main()</code>, which will be <code>EXIT_FAILURE</code> if any test cases failed, <code>EXIT_SUCCESS</code> otherwise.

TODO(mbland): create some kind of automated script or editor macros?

=== Disable for Windows (for now) ===
Until we solve the private-symbol problem on Windows, we will need to wrap our unit test code in the following <code>#ifdef</code> block:
<pre>
#if !defined(OPENSSL_SYS_WINDOWS)

/* All the test code, including main() */

int main(int argc, char *argv[])
{
return EXIT_SUCCESS;
}

#endif /* OPENSSL_NO_WINDOWS */
</pre>

== Samples ==
Existing tests that can be used as models for new tests.
* [http://git.openssl.org/gitweb/?p=openssl.git;a=blob;f=ssl/heartbeat_test.c ssl/heartbeat_test.c]: Follows [http://mike-bland.com/2014/06/05/pseudo-xunit-pattern.html the Pseudo-xUnit pattern]

How To Write Unit Tests For OpenSSL

2014-07-20T00:00:12Z

Mbland: /* Style */ Combine last two sections in light of ADD_TEST() and run_tests()

This is an outline of the basic process and principles to follow when writing unit tests. This document will evolve quite a bit as the community gains experience.

==Mechanics ==
=== Forking the Repo ===
For now, people contributing new unit tests for uncovered code (as opposed to submitting tests with new code changes) should fork [https://github.com/mbland/openssl Mike Bland's GitHub repository] and issue [https://help.github.com/articles/using-pull-requests GitHub pull requests]. Remember to do all development on a topic branch (not <code>master</code>). Tests committed to this repo will occasionally get merged into the master OpenSSL repo.

Rationale per Matt Caswell: We should think about process here. If you are going off to recruit an army of people writing tests then I wonder if it is worth setting up a separate github repository whilst you are building up the tests. We can then merge in from that on a periodic basis. I wouldn't want availability of openssl team committers to be a bottle neck.

Set up the [https://github.com/openssl/openssl OpenSSL master repository] as your upstream remote as per [https://help.github.com/articles/fork-a-repo#step-3-configure-remotes GitHub's instructions on configuring remotes]:
<pre>
$ cd my-openssl-repo
$ git remote add upstream https://github.com/openssl/openssl.git
</pre>

You should see the following output from <code>git remove -v</code> (where <code>$USER</code> is your GitHub username):
<pre>
$ git remote -v
origin https://github.com/$USER/openssl.git (fetch)
origin https://github.com/$USER/openssl.git (push)
upstream https://github.com/openssl/openssl.git (fetch)
upstream https://github.com/openssl/openssl.git (push)
</pre>

=== Check out the Tools and Tips page ===
[[Testing and Development Tools and Tips]] has information on tools that may make navigating and building the code a bit easier.

=== Use the Test Template Generator ===
TODO(mbland): Get template generator checked-in. Maybe have a template generator for each library, e.g. <code>ssl/new-test.sh</code> that has additional setup boilerplate specific to the <code>ssl</code> library.

Use the <code>test/new-test.sh</code> script to generate a skeleton test file. ([https://github.com/mbland/openssl/compare/test-util pending in the test-util branch])

=== Add Makefile Targets ===
The following instructions use the <code>Makefile</code> targets for <code>ssl/heartbeat_test.c</code> as an example.

In the <code>Makefile</code> for the library containing the test, add the test source file to the <code>TEST</code> variable:

<pre>
# ssl/Makefile
TEST=ssltest.c heartbeat_test.c
</pre>

In <code>test/Makefile</code>:
* add a variable for the test target near the top of the file, right after the existing test variables
* use the variable to add an executable target to the <code>EXE</code> variable
* use the variable to add an object file target to the <code>OBJ</code> variable
* use the variable to add a source file target to the <code>SRC</code> variable
* add the test target to the <code>alltests</code> target
* add the target to execute the test
* add the target to build the test executable
<pre>
# test/Makefile
HEARTBEATTEST= heartbeat_test
EXE= ... $(HEARTBEATTEST)$(EXE_EXT)
OBJ= ... $(HEARTBEATTEST).o
SRC= ... $(HEARTBEATTEST).c
alltests: \
... test_heartbeat

test_heartbeat: $(HEARTBEATTEST)$(EXE_EXT)
../util/shlib_wrap.sh ./$(HEARTBEATTEST)

$(HEARTBEATTEST)$(EXE_EXT): $(HEARTBEATTEST).o $(DLIBCRYPTO) testutil.o
@target=$(HEARTBEATTEST) testutil=testutil.o; $(BUILD_CMD)
</pre>

=== Run <code>make links && make depend</code> ===
Finally, run <code>make links && make depend</code> to link the new test into the <code>test/</code> directory and automatically generate the header file dependencies.

=== Building and Running the Test ===
If you're initially developing on Mac OS X or (for now) FreeBSD 10, just use the stock method of building and testing:
<pre>
$ ./config && make && make test

# To execute just one specific test
$ make TESTS=test_heartbeat test
</pre>

Ultimately the test will have to compile and pass with developer flags enabled:
<pre>
$ ./GitConfigure debug-ben-debug-64-clang
$ ./GitMake -j 2
$ ./GitMake test_heartbeat
</pre>

The above currently doesn't work on Mac OS X or FreeBSD 10; for now, you can install the 9.1 release of [http://www.freebsd.org/ FreeBSD], which uses Clang 3.1, via a virtualization platform such as [https://www.virtualbox.org/ VirtualBox]. (Mac OS X breaks, in part, due to the fact that it doesn't use the GNU assembler; <code>-fsanitize</code> appears to be ignored on FreeBSD 9.1/Clang 3.1, but later versions break because the Clang compiler will emit <code>-fsanitize</code> symbols but the <code>libasan</code> library [http://lists.freebsd.org/pipermail/freebsd-hackers/2013-December/043995.html has yet to be ported to FreeBSD].)

=== Keep your repo up-to-date ===
Periodically run the following to keep your branch up-to-date:
<pre>
$ git fetch upstream master
$ git rebase upstream/master
</pre>
This will pull all the updates from the master OpenSSL repository into your repository, then update your branch to apply your changes on top of the latest updates.

=== Send a pull request ===
When your test is ready, send a [https://help.github.com/articles/using-pull-requests GitHub pull request]. Note that the pull request should be based on the <code>tests</code> branch of Mike's repository, not <code>master</code>. (This should be the default; let Mike know if it doesn't appear to be!) We'll review the code, and when it's ready, it'll get merged into Mike's repository. From there, it will eventually get pulled into the master OpenSSL repository.

== Style ==
[http://mike-bland.com/2014/06/05/pseudo-xunit-pattern.html The Pseudo-xUnit Pattern] is that established by [http://git.openssl.org/gitweb/?p=openssl.git;a=blob;f=ssl/heartbeat_test.c ssl/heartbeat_test.c]. This pattern organizes code in a fashion reminiscent of the [http://en.wikipedia.org/wiki/XUnit xUnit] family of unit testing frameworks, without actually using a testing framework. This should lower the barrier to entry for people wanting to write unit tests, but enable a relatively easy migration to an xUnit-based framework if we decide to do so one day.

Some of the basic principles to follow are:

=== <code>#include</code> the header for the code under test first ===
Having the header file for the code under test appear as the first <code>#include</code> directive ensures that that file is self-contained, i.e. it includes every header file it depends on, rather than relying on client code to include its dependencies.

=== <code>#include "testutil.h"</code> should come second ===
<code>test/testutil.h</code> contains the helper macros used in writing OpenSSL tests. Since the tests will be linked into <code>test/</code> by the [[#Run_make_links_.26.26_make_depend|make links]] step, and built in the <code>test/</code> directory, the "testutil.h" file will appear to be in the same directory as the test file.

=== Define a fixture structure ===
The fixture structure should contain all of the inputs to the code under test and all of the expected result values. It should also contain a <code>const char*</code> for the name of the test case function that created it, to aid in error message formatting. Even though the fixture may contain dynamically-allocated members, the fixture itself should be copied by value to reduce the necessary degree of memory management in a small unit test program.

=== Define set_up() and tear_down() functions for the fixture ===
<code>set_up()</code> should return a newly-initialized test fixture structure. It should take the name of the test case as an argument (i.e. <code>__func__</code>) and assign it to the fixture. All of the fixture members should be initialized, which each test case function can then override as needed.

<code>tear_down()</code> should take the fixture as an argument and release any resources allocated by <code>set_up()</code>. It can also call any library-wide error printing routines (e.g. <code>ERR_print_errors_fp(stderr)</code>).

=== Use <code>SETUP_TEST_FIXTURE()</code> and <code>EXECUTE_TEST()</code> from <code>test/testutil.h</code> ===
Each test case function should call <code>set_up()</code> as its first statement, and should call <code>tear_down()</code> just before returning. This is handled in a uniform fashion when using the <code>SETUP_TEST_FIXTURE()</code> and <code>EXECUTE_TEST()</code> helper macros from [http://git.openssl.org/gitweb/?p=openssl.git;a=blob;f=test/testutil.h test/testutil.h]. See the comments in <code>test/testutil.h</code> for usage, and <code>ssl/heartbeat_test.c</code> for an example of how to wrap the macros for a specific unit test.

=== Use test case functions, not a table of fixtures ===
Individual test case functions that call a common execution function are much more readable and maintainable than a loop over a table of fixture structures. Explicit fixture variable assignments aid comprehension when reading a specific test case, which saves time and energy when trying to understand a test or diagnose a failure. When a new member is added to an existing fixture, <code>set_up()</code> can set a default for all test cases, and only the test cases that rely on that new member need to be updated.

=== Use very descriptive test case names ===
Give tests long, descriptive names that provide ample context for the details of the test case. Good test names are also help produce good error messages.

=== Group test cases into "suites" by naming convention ===
Give logically-related tests the same prefix, e.g. <code>test_dtls1_</code> and <code>test_tls1</code> from <code>ssl/heartbeat_test.c</code>. If need be, you can define suite-specific <code>set_up()</code> functions that call the common <code>set_up()</code> and elaborate on it. (This generally shouldn't be necessary for <code>tear_down()</code>.)

=== Keep individual test case functions focused on one thing ===
If the test name contains the word "and", consider breaking it into two or more separate test case functions.

=== Write very descriptive error messages ===
Include the test case function name in each error message, and explain in detail the context for the assertion that failed. Include the expected result (contained in the fixture structure) and the actual result returned from the code under test. Write helper methods for complex values as needed (e.g. <code>print_payload()</code> in <code>ssl/heartbeat_test.c</code>).

=== Return zero on success and one on failure ===
The return value will be used to tally the number of test cases that failed. Even if multiple assertions fail for a single test case, the result should be exactly one.

=== Register each test case using ADD_TEST() and execute using run_tests() ===
Whatever function is used as the test runner, be that <code>main()</code> or a separate function called by <code>main()</code>, add your test case functions to that function using <code>ADD_TEST()</code> and execute them using <code>run_tests()</code>.

<code>run_tests()</code> will add up the total number of failed test cases and report that number as the last error message of the test. The return value of <code>run_tests()</code> should be the value returned from <code>main()</code>, which will be <code>EXIT_FAILURE</code> if any test cases failed, <code>EXIT_SUCCESS</code> otherwise.

TODO(mbland): create some kind of automated script or editor macros?

=== Disable for Windows (for now) ===
Until we solve the private-symbol problem on Windows, we will need to wrap our unit test code in the following <code>#ifdef</code> block:
<pre>
#if !defined(OPENSSL_SYS_WINDOWS)

/* All the test code, including main() */

int main(int argc, char *argv[])
{
return EXIT_SUCCESS;
}

#endif /* OPENSSL_NO_WINDOWS */
</pre>

== Samples ==
Existing tests that can be used as models for new tests.
* [http://git.openssl.org/gitweb/?p=openssl.git;a=blob;f=ssl/heartbeat_test.c ssl/heartbeat_test.c]: Follows [http://mike-bland.com/2014/06/05/pseudo-xunit-pattern.html the Pseudo-xUnit pattern]

How To Write Unit Tests For OpenSSL

2014-07-19T23:56:37Z

Mbland: /* Add each test case function to the test runner function */ Update with advice to use ADD_TEST() and run_tests()

This is an outline of the basic process and principles to follow when writing unit tests. This document will evolve quite a bit as the community gains experience.

==Mechanics ==
=== Forking the Repo ===
For now, people contributing new unit tests for uncovered code (as opposed to submitting tests with new code changes) should fork [https://github.com/mbland/openssl Mike Bland's GitHub repository] and issue [https://help.github.com/articles/using-pull-requests GitHub pull requests]. Remember to do all development on a topic branch (not <code>master</code>). Tests committed to this repo will occasionally get merged into the master OpenSSL repo.

Rationale per Matt Caswell: We should think about process here. If you are going off to recruit an army of people writing tests then I wonder if it is worth setting up a separate github repository whilst you are building up the tests. We can then merge in from that on a periodic basis. I wouldn't want availability of openssl team committers to be a bottle neck.

Set up the [https://github.com/openssl/openssl OpenSSL master repository] as your upstream remote as per [https://help.github.com/articles/fork-a-repo#step-3-configure-remotes GitHub's instructions on configuring remotes]:
<pre>
$ cd my-openssl-repo
$ git remote add upstream https://github.com/openssl/openssl.git
</pre>

You should see the following output from <code>git remove -v</code> (where <code>$USER</code> is your GitHub username):
<pre>
$ git remote -v
origin https://github.com/$USER/openssl.git (fetch)
origin https://github.com/$USER/openssl.git (push)
upstream https://github.com/openssl/openssl.git (fetch)
upstream https://github.com/openssl/openssl.git (push)
</pre>

=== Check out the Tools and Tips page ===
[[Testing and Development Tools and Tips]] has information on tools that may make navigating and building the code a bit easier.

=== Use the Test Template Generator ===
TODO(mbland): Get template generator checked-in. Maybe have a template generator for each library, e.g. <code>ssl/new-test.sh</code> that has additional setup boilerplate specific to the <code>ssl</code> library.

Use the <code>test/new-test.sh</code> script to generate a skeleton test file. ([https://github.com/mbland/openssl/compare/test-util pending in the test-util branch])

=== Add Makefile Targets ===
The following instructions use the <code>Makefile</code> targets for <code>ssl/heartbeat_test.c</code> as an example.

In the <code>Makefile</code> for the library containing the test, add the test source file to the <code>TEST</code> variable:

<pre>
# ssl/Makefile
TEST=ssltest.c heartbeat_test.c
</pre>

In <code>test/Makefile</code>:
* add a variable for the test target near the top of the file, right after the existing test variables
* use the variable to add an executable target to the <code>EXE</code> variable
* use the variable to add an object file target to the <code>OBJ</code> variable
* use the variable to add a source file target to the <code>SRC</code> variable
* add the test target to the <code>alltests</code> target
* add the target to execute the test
* add the target to build the test executable
<pre>
# test/Makefile
HEARTBEATTEST= heartbeat_test
EXE= ... $(HEARTBEATTEST)$(EXE_EXT)
OBJ= ... $(HEARTBEATTEST).o
SRC= ... $(HEARTBEATTEST).c
alltests: \
... test_heartbeat

test_heartbeat: $(HEARTBEATTEST)$(EXE_EXT)
../util/shlib_wrap.sh ./$(HEARTBEATTEST)

$(HEARTBEATTEST)$(EXE_EXT): $(HEARTBEATTEST).o $(DLIBCRYPTO) testutil.o
@target=$(HEARTBEATTEST) testutil=testutil.o; $(BUILD_CMD)
</pre>

=== Run <code>make links && make depend</code> ===
Finally, run <code>make links && make depend</code> to link the new test into the <code>test/</code> directory and automatically generate the header file dependencies.

=== Building and Running the Test ===
If you're initially developing on Mac OS X or (for now) FreeBSD 10, just use the stock method of building and testing:
<pre>
$ ./config && make && make test

# To execute just one specific test
$ make TESTS=test_heartbeat test
</pre>

Ultimately the test will have to compile and pass with developer flags enabled:
<pre>
$ ./GitConfigure debug-ben-debug-64-clang
$ ./GitMake -j 2
$ ./GitMake test_heartbeat
</pre>

The above currently doesn't work on Mac OS X or FreeBSD 10; for now, you can install the 9.1 release of [http://www.freebsd.org/ FreeBSD], which uses Clang 3.1, via a virtualization platform such as [https://www.virtualbox.org/ VirtualBox]. (Mac OS X breaks, in part, due to the fact that it doesn't use the GNU assembler; <code>-fsanitize</code> appears to be ignored on FreeBSD 9.1/Clang 3.1, but later versions break because the Clang compiler will emit <code>-fsanitize</code> symbols but the <code>libasan</code> library [http://lists.freebsd.org/pipermail/freebsd-hackers/2013-December/043995.html has yet to be ported to FreeBSD].)

=== Keep your repo up-to-date ===
Periodically run the following to keep your branch up-to-date:
<pre>
$ git fetch upstream master
$ git rebase upstream/master
</pre>
This will pull all the updates from the master OpenSSL repository into your repository, then update your branch to apply your changes on top of the latest updates.

=== Send a pull request ===
When your test is ready, send a [https://help.github.com/articles/using-pull-requests GitHub pull request]. Note that the pull request should be based on the <code>tests</code> branch of Mike's repository, not <code>master</code>. (This should be the default; let Mike know if it doesn't appear to be!) We'll review the code, and when it's ready, it'll get merged into Mike's repository. From there, it will eventually get pulled into the master OpenSSL repository.

== Style ==
[http://mike-bland.com/2014/06/05/pseudo-xunit-pattern.html The Pseudo-xUnit Pattern] is that established by [http://git.openssl.org/gitweb/?p=openssl.git;a=blob;f=ssl/heartbeat_test.c ssl/heartbeat_test.c]. This pattern organizes code in a fashion reminiscent of the [http://en.wikipedia.org/wiki/XUnit xUnit] family of unit testing frameworks, without actually using a testing framework. This should lower the barrier to entry for people wanting to write unit tests, but enable a relatively easy migration to an xUnit-based framework if we decide to do so one day.

Some of the basic principles to follow are:

=== <code>#include</code> the header for the code under test first ===
Having the header file for the code under test appear as the first <code>#include</code> directive ensures that that file is self-contained, i.e. it includes every header file it depends on, rather than relying on client code to include its dependencies.

=== <code>#include "testutil.h"</code> should come second ===
<code>test/testutil.h</code> contains the helper macros used in writing OpenSSL tests. Since the tests will be linked into <code>test/</code> by the [[#Run_make_links_.26.26_make_depend|make links]] step, and built in the <code>test/</code> directory, the "testutil.h" file will appear to be in the same directory as the test file.

=== Define a fixture structure ===
The fixture structure should contain all of the inputs to the code under test and all of the expected result values. It should also contain a <code>const char*</code> for the name of the test case function that created it, to aid in error message formatting. Even though the fixture may contain dynamically-allocated members, the fixture itself should be copied by value to reduce the necessary degree of memory management in a small unit test program.

=== Define set_up() and tear_down() functions for the fixture ===
<code>set_up()</code> should return a newly-initialized test fixture structure. It should take the name of the test case as an argument (i.e. <code>__func__</code>) and assign it to the fixture. All of the fixture members should be initialized, which each test case function can then override as needed.

<code>tear_down()</code> should take the fixture as an argument and release any resources allocated by <code>set_up()</code>. It can also call any library-wide error printing routines (e.g. <code>ERR_print_errors_fp(stderr)</code>).

=== Use <code>SETUP_TEST_FIXTURE()</code> and <code>EXECUTE_TEST()</code> from <code>test/testutil.h</code> ===
Each test case function should call <code>set_up()</code> as its first statement, and should call <code>tear_down()</code> just before returning. This is handled in a uniform fashion when using the <code>SETUP_TEST_FIXTURE()</code> and <code>EXECUTE_TEST()</code> helper macros from [http://git.openssl.org/gitweb/?p=openssl.git;a=blob;f=test/testutil.h test/testutil.h]. See the comments in <code>test/testutil.h</code> for usage, and <code>ssl/heartbeat_test.c</code> for an example of how to wrap the macros for a specific unit test.

=== Use test case functions, not a table of fixtures ===
Individual test case functions that call a common execution function are much more readable and maintainable than a loop over a table of fixture structures. Explicit fixture variable assignments aid comprehension when reading a specific test case, which saves time and energy when trying to understand a test or diagnose a failure. When a new member is added to an existing fixture, <code>set_up()</code> can set a default for all test cases, and only the test cases that rely on that new member need to be updated.

=== Use very descriptive test case names ===
Give tests long, descriptive names that provide ample context for the details of the test case. Good test names are also help produce good error messages.

=== Group test cases into "suites" by naming convention ===
Give logically-related tests the same prefix, e.g. <code>test_dtls1_</code> and <code>test_tls1</code> from <code>ssl/heartbeat_test.c</code>. If need be, you can define suite-specific <code>set_up()</code> functions that call the common <code>set_up()</code> and elaborate on it. (This generally shouldn't be necessary for <code>tear_down()</code>.)

=== Keep individual test case functions focused on one thing ===
If the test name contains the word "and", consider breaking it into two or more separate test case functions.

=== Write very descriptive error messages ===
Include the test case function name in each error message, and explain in detail the context for the assertion that failed. Include the expected result (contained in the fixture structure) and the actual result returned from the code under test. Write helper methods for complex values as needed (e.g. <code>print_payload()</code> in <code>ssl/heartbeat_test.c</code>).

=== Return zero on success and one on failure ===
The return value will be used to tally the number of test cases that failed. Even if multiple assertions fail for a single test case, the result should be exactly one.

=== Add each test case function to the test runner function ===
Whatever function is used as the test runner, be that <code>main()</code> or a separate function called by <code>main()</code>, add your test case functions to that function using <code>ADD_TEST()</code> and execute them using <code>run_tests()</code>.

TODO(mbland): create some kind of automated script or editor macros?

=== Report the number of test cases that failed ===
Add up the total number of failed test cases in <code>main()</code> and report that number as the last error message of the test. <code>main()</code> should return <code>EXIT_FAILURE</code> if any test cases failed, <code>EXIT_SUCCESS</code> otherwise.

=== Disable for Windows (for now) ===
Until we solve the private-symbol problem on Windows, we will need to wrap our unit test code in the following <code>#ifdef</code> block:
<pre>
#if !defined(OPENSSL_SYS_WINDOWS)

/* All the test code, including main() */

int main(int argc, char *argv[])
{
return EXIT_SUCCESS;
}

#endif /* OPENSSL_NO_WINDOWS */
</pre>

== Samples ==
Existing tests that can be used as models for new tests.
* [http://git.openssl.org/gitweb/?p=openssl.git;a=blob;f=ssl/heartbeat_test.c ssl/heartbeat_test.c]: Follows [http://mike-bland.com/2014/06/05/pseudo-xunit-pattern.html the Pseudo-xUnit pattern]

How To Write Unit Tests For OpenSSL

2014-07-19T23:53:01Z

Mbland: /* Add Makefile Targets */ Add testutil.o to test executable build rule

This is an outline of the basic process and principles to follow when writing unit tests. This document will evolve quite a bit as the community gains experience.

==Mechanics ==
=== Forking the Repo ===
For now, people contributing new unit tests for uncovered code (as opposed to submitting tests with new code changes) should fork [https://github.com/mbland/openssl Mike Bland's GitHub repository] and issue [https://help.github.com/articles/using-pull-requests GitHub pull requests]. Remember to do all development on a topic branch (not <code>master</code>). Tests committed to this repo will occasionally get merged into the master OpenSSL repo.

Rationale per Matt Caswell: We should think about process here. If you are going off to recruit an army of people writing tests then I wonder if it is worth setting up a separate github repository whilst you are building up the tests. We can then merge in from that on a periodic basis. I wouldn't want availability of openssl team committers to be a bottle neck.

Set up the [https://github.com/openssl/openssl OpenSSL master repository] as your upstream remote as per [https://help.github.com/articles/fork-a-repo#step-3-configure-remotes GitHub's instructions on configuring remotes]:
<pre>
$ cd my-openssl-repo
$ git remote add upstream https://github.com/openssl/openssl.git
</pre>

You should see the following output from <code>git remove -v</code> (where <code>$USER</code> is your GitHub username):
<pre>
$ git remote -v
origin https://github.com/$USER/openssl.git (fetch)
origin https://github.com/$USER/openssl.git (push)
upstream https://github.com/openssl/openssl.git (fetch)
upstream https://github.com/openssl/openssl.git (push)
</pre>

=== Check out the Tools and Tips page ===
[[Testing and Development Tools and Tips]] has information on tools that may make navigating and building the code a bit easier.

=== Use the Test Template Generator ===
TODO(mbland): Get template generator checked-in. Maybe have a template generator for each library, e.g. <code>ssl/new-test.sh</code> that has additional setup boilerplate specific to the <code>ssl</code> library.

Use the <code>test/new-test.sh</code> script to generate a skeleton test file. ([https://github.com/mbland/openssl/compare/test-util pending in the test-util branch])

=== Add Makefile Targets ===
The following instructions use the <code>Makefile</code> targets for <code>ssl/heartbeat_test.c</code> as an example.

In the <code>Makefile</code> for the library containing the test, add the test source file to the <code>TEST</code> variable:

<pre>
# ssl/Makefile
TEST=ssltest.c heartbeat_test.c
</pre>

In <code>test/Makefile</code>:
* add a variable for the test target near the top of the file, right after the existing test variables
* use the variable to add an executable target to the <code>EXE</code> variable
* use the variable to add an object file target to the <code>OBJ</code> variable
* use the variable to add a source file target to the <code>SRC</code> variable
* add the test target to the <code>alltests</code> target
* add the target to execute the test
* add the target to build the test executable
<pre>
# test/Makefile
HEARTBEATTEST= heartbeat_test
EXE= ... $(HEARTBEATTEST)$(EXE_EXT)
OBJ= ... $(HEARTBEATTEST).o
SRC= ... $(HEARTBEATTEST).c
alltests: \
... test_heartbeat

test_heartbeat: $(HEARTBEATTEST)$(EXE_EXT)
../util/shlib_wrap.sh ./$(HEARTBEATTEST)

$(HEARTBEATTEST)$(EXE_EXT): $(HEARTBEATTEST).o $(DLIBCRYPTO) testutil.o
@target=$(HEARTBEATTEST) testutil=testutil.o; $(BUILD_CMD)
</pre>

=== Run <code>make links && make depend</code> ===
Finally, run <code>make links && make depend</code> to link the new test into the <code>test/</code> directory and automatically generate the header file dependencies.

=== Building and Running the Test ===
If you're initially developing on Mac OS X or (for now) FreeBSD 10, just use the stock method of building and testing:
<pre>
$ ./config && make && make test

# To execute just one specific test
$ make TESTS=test_heartbeat test
</pre>

Ultimately the test will have to compile and pass with developer flags enabled:
<pre>
$ ./GitConfigure debug-ben-debug-64-clang
$ ./GitMake -j 2
$ ./GitMake test_heartbeat
</pre>

The above currently doesn't work on Mac OS X or FreeBSD 10; for now, you can install the 9.1 release of [http://www.freebsd.org/ FreeBSD], which uses Clang 3.1, via a virtualization platform such as [https://www.virtualbox.org/ VirtualBox]. (Mac OS X breaks, in part, due to the fact that it doesn't use the GNU assembler; <code>-fsanitize</code> appears to be ignored on FreeBSD 9.1/Clang 3.1, but later versions break because the Clang compiler will emit <code>-fsanitize</code> symbols but the <code>libasan</code> library [http://lists.freebsd.org/pipermail/freebsd-hackers/2013-December/043995.html has yet to be ported to FreeBSD].)

=== Keep your repo up-to-date ===
Periodically run the following to keep your branch up-to-date:
<pre>
$ git fetch upstream master
$ git rebase upstream/master
</pre>
This will pull all the updates from the master OpenSSL repository into your repository, then update your branch to apply your changes on top of the latest updates.

=== Send a pull request ===
When your test is ready, send a [https://help.github.com/articles/using-pull-requests GitHub pull request]. Note that the pull request should be based on the <code>tests</code> branch of Mike's repository, not <code>master</code>. (This should be the default; let Mike know if it doesn't appear to be!) We'll review the code, and when it's ready, it'll get merged into Mike's repository. From there, it will eventually get pulled into the master OpenSSL repository.

== Style ==
[http://mike-bland.com/2014/06/05/pseudo-xunit-pattern.html The Pseudo-xUnit Pattern] is that established by [http://git.openssl.org/gitweb/?p=openssl.git;a=blob;f=ssl/heartbeat_test.c ssl/heartbeat_test.c]. This pattern organizes code in a fashion reminiscent of the [http://en.wikipedia.org/wiki/XUnit xUnit] family of unit testing frameworks, without actually using a testing framework. This should lower the barrier to entry for people wanting to write unit tests, but enable a relatively easy migration to an xUnit-based framework if we decide to do so one day.

Some of the basic principles to follow are:

=== <code>#include</code> the header for the code under test first ===
Having the header file for the code under test appear as the first <code>#include</code> directive ensures that that file is self-contained, i.e. it includes every header file it depends on, rather than relying on client code to include its dependencies.

=== <code>#include "testutil.h"</code> should come second ===
<code>test/testutil.h</code> contains the helper macros used in writing OpenSSL tests. Since the tests will be linked into <code>test/</code> by the [[#Run_make_links_.26.26_make_depend|make links]] step, and built in the <code>test/</code> directory, the "testutil.h" file will appear to be in the same directory as the test file.

=== Define a fixture structure ===
The fixture structure should contain all of the inputs to the code under test and all of the expected result values. It should also contain a <code>const char*</code> for the name of the test case function that created it, to aid in error message formatting. Even though the fixture may contain dynamically-allocated members, the fixture itself should be copied by value to reduce the necessary degree of memory management in a small unit test program.

=== Define set_up() and tear_down() functions for the fixture ===
<code>set_up()</code> should return a newly-initialized test fixture structure. It should take the name of the test case as an argument (i.e. <code>__func__</code>) and assign it to the fixture. All of the fixture members should be initialized, which each test case function can then override as needed.

<code>tear_down()</code> should take the fixture as an argument and release any resources allocated by <code>set_up()</code>. It can also call any library-wide error printing routines (e.g. <code>ERR_print_errors_fp(stderr)</code>).

=== Use <code>SETUP_TEST_FIXTURE()</code> and <code>EXECUTE_TEST()</code> from <code>test/testutil.h</code> ===
Each test case function should call <code>set_up()</code> as its first statement, and should call <code>tear_down()</code> just before returning. This is handled in a uniform fashion when using the <code>SETUP_TEST_FIXTURE()</code> and <code>EXECUTE_TEST()</code> helper macros from [http://git.openssl.org/gitweb/?p=openssl.git;a=blob;f=test/testutil.h test/testutil.h]. See the comments in <code>test/testutil.h</code> for usage, and <code>ssl/heartbeat_test.c</code> for an example of how to wrap the macros for a specific unit test.

=== Use test case functions, not a table of fixtures ===
Individual test case functions that call a common execution function are much more readable and maintainable than a loop over a table of fixture structures. Explicit fixture variable assignments aid comprehension when reading a specific test case, which saves time and energy when trying to understand a test or diagnose a failure. When a new member is added to an existing fixture, <code>set_up()</code> can set a default for all test cases, and only the test cases that rely on that new member need to be updated.

=== Use very descriptive test case names ===
Give tests long, descriptive names that provide ample context for the details of the test case. Good test names are also help produce good error messages.

=== Group test cases into "suites" by naming convention ===
Give logically-related tests the same prefix, e.g. <code>test_dtls1_</code> and <code>test_tls1</code> from <code>ssl/heartbeat_test.c</code>. If need be, you can define suite-specific <code>set_up()</code> functions that call the common <code>set_up()</code> and elaborate on it. (This generally shouldn't be necessary for <code>tear_down()</code>.)

=== Keep individual test case functions focused on one thing ===
If the test name contains the word "and", consider breaking it into two or more separate test case functions.

=== Write very descriptive error messages ===
Include the test case function name in each error message, and explain in detail the context for the assertion that failed. Include the expected result (contained in the fixture structure) and the actual result returned from the code under test. Write helper methods for complex values as needed (e.g. <code>print_payload()</code> in <code>ssl/heartbeat_test.c</code>).

=== Return zero on success and one on failure ===
The return value will be used to tally the number of test cases that failed. Even if multiple assertions fail for a single test case, the result should be exactly one.

=== Add each test case function to the test runner function ===
Whatever function is used to execute the batch of test case functions, be that <code>main()</code> or a separate function called by <code>main()</code>, don't forget to add your test case functions to that function.

TODO(mbland): create some kind of automated script or editor macros?

=== Report the number of test cases that failed ===
Add up the total number of failed test cases in <code>main()</code> and report that number as the last error message of the test. <code>main()</code> should return <code>EXIT_FAILURE</code> if any test cases failed, <code>EXIT_SUCCESS</code> otherwise.

=== Disable for Windows (for now) ===
Until we solve the private-symbol problem on Windows, we will need to wrap our unit test code in the following <code>#ifdef</code> block:
<pre>
#if !defined(OPENSSL_SYS_WINDOWS)

/* All the test code, including main() */

int main(int argc, char *argv[])
{
return EXIT_SUCCESS;
}

#endif /* OPENSSL_NO_WINDOWS */
</pre>

== Samples ==
Existing tests that can be used as models for new tests.
* [http://git.openssl.org/gitweb/?p=openssl.git;a=blob;f=ssl/heartbeat_test.c ssl/heartbeat_test.c]: Follows [http://mike-bland.com/2014/06/05/pseudo-xunit-pattern.html the Pseudo-xUnit pattern]

Testing and Development Tools and Tips

2014-07-19T23:49:50Z

Mbland: /* test/testutil.h */ Add ADD_TEST() and run_tests()

This is a collection of helpful tools and tips for navigating the OpenSSL code base and managing a local git repository.

== Testing Environment and Tools ==
In [https://github.com/mbland/openssl/compare/test-util the test-util branch of Mike Bland's fork] are a few helper files that you may wish to copy or pull into your branch. As they are incorporated into the main OpenSSL repository, links to the mainline versions will be incorporated into the descriptions below. However, there may be newer versions in Mike's fork.

=== test/testutil.h ===
[http://git.openssl.org/gitweb/?p=openssl.git;a=blob;f=test/testutil.h test/testutil.h] defines:
* the generic <code>SETUP_TEST_FIXTURE</code> and <code>EXECUTE_TEST</code> macros; the header comments describe how to define test-specific macros based on these
* <code>ADD_TEST()</code> and <code>run_tests()</code> to standardize the registration and execution of test case functions within <code>main()</code>

=== test/new-test.sh ===
[https://github.com/mbland/openssl/blob/test-util/test/new-test.sh test/new-test.sh (mbland's fork)] generates a new automated test stub, following [http://mike-bland.com/2014/06/05/pseudo-xunit-pattern.html the Pseudo-xUnit Pattern] and using the macros from <code>test/testutil.h</code>. The generated stub will compile standalone.

=== test/test_env.bash ===
[https://github.com/mbland/openssl/blob/test-util/test/test_env.bash test/test_env.bash (mbland's fork)]: contains environment variables, functions, and aliases to help with OpenSSL unit testing. The header comments contain documentation on each of the functions and aliases in the file.

== Cscope ==
[http://cscope.sourceforge.net/ Cscope] is a powerful C source code browser. Once you have it installed, you can use the <code>make-cscope</code> and <code>open-cscope</code> functions from <code>test/test_env.bash</code>.

=== Vim Integration ===
Once <code>$CSCOPE_DB</code> (defined in <code>test/test_env.bash</code>) is built using <code>make-cscope</code>, you can take advantage of the Cscope integration in Vim by adding something like the following to your <code>.vimrc</code>:

<pre>
" Detect whether cscope features are present and whether we should add a
" connection to an existing cscope.out database file
if has("cscope")
set nocsverb
if filereadable("./cscope.out")
cs add cscope.out
elseif $CSCOPE_DB != ""
if filereadable($CSCOPE_DB)
cs add $CSCOPE_DB
else
echo "Can't read $CSCOPE_DB; cscope add not run"
endif
endif
set csre
set csverb
endif
</pre>

== Ctags ==
Ctags generates an index of symbols that many editors can use to quickly navigate the source code. Though many Unices ship with a version installed at <code>/usr/bin/ctags</code>, <code>test_env.bash</code> presumes that [http://ctags.sourceforge.net/ Exuberant Ctags] is installed.

=== Vim Integration ===
Add this to your <code>.vimrc</code> to take advantage of Ctags integration in vim (where <code>$TAGS_FILE</code> is defined in <code>test/test_env.bash</code>):

<pre>
" The tags path, as set below, will search the local file first, then the
" project-wide file.
if $TAGS_FILE != ""
if filereadable($TAGS_FILE)
set tags+=$TAGS_FILE
else
echo "Can't read $TAGS_FILE; tags not set"
endif
endif
</pre>

== git-new-workdir ==
Usually installed as <code>/usr/local/share/git-core/contrib/workdir/git-new-workdir</code>, this will create a new working directory for a specified branch that's linked to the original repository. This is nice when working on multiple branches in parallel, as it doesn't require committing or stashing changes before issuing a <code>git checkout</code> to switch between branches. The following <code>bash</code> function wraps <code>git-new-workdir</code> to create a working dir called <code>reponame-branchname</code> in the same directory as the original repository:

<pre>
# Creates a branch-specific git working directory in the same directory as the
# original repository.
git-new-workdir() {
if test $# -ne 2; then
echo "Usage: $FUNCNAME <git repo> <branch name>"
return 1
elif test ! -d $1; then
echo "$1 does not exist"
return 1
fi
new_workdir=$1-$2
if /usr/local/share/git-core/contrib/workdir/git-new-workdir\
$1 $new_workdir $2; then
echo "Created $new_workdir"
else
return 1
fi
}
</pre>

'''Note:''' On Ubuntu 14.04, <code>git-new-workdir</code> is installed as part of the <code>git</code> package as <code>/usr/share/doc/git/contrib/workdir/git-new-workdir</code> and does not have execute permission by default.

== tmux Terminal Multiplexer ==
[http://tmux.sourceforge.net/ The tmux terminal multiplexer] is helpful for using a single terminal window to efficiently manage different editing and build sessions, amongst many other useful functions. [http://pragprog.com/book/bhtmux/tmux tmux: Productive Mouse-Free Development] from The Pragmatic Bookshelf is a great primer.

These are some helpful config options to add to <code>~/.tmux.conf</code>, including getting GNU screen-like CTRL-a behavior:

<pre>
set-option -g prefix C-a
unbind-key C-b
bind-key C-a send-prefix

bind R source-file ~/.tmux.conf \; display "Reloaded ~/.tmux.conf"
set -g default-terminal "screen-256color"
set -g status-style fg=black,bg=colour7
setw -g window-status-style fg=default,bg=default,none
setw -g window-status-current-style fg=black,bg=colour15,bold
set -g pane-active-border-style fg=default,bg=colour7
set -g message-style fg=black,bg=colour11,bright
set -g status-utf8 on
setw -g monitor-activity on
set -g visual-activity on

bind -r H resize-pane -L 5
bind -r J resize-pane -D 5
bind -r K resize-pane -U 5
bind -r L resize-pane -R 5

# Enables nohup to work. From:
# https://github.com/ChrisJohnsen/tmux-MacOSX-pasteboard
set-option -g default-command "reattach-to-user-namespace -l $SHELL"
</pre>

Testing and Development Tools and Tips

2014-07-02T18:46:07Z

Mbland: /* git-new-workdir */ Notice about git-new-workdir location on Ubuntu

This is a collection of helpful tools and tips for navigating the OpenSSL code base and managing a local git repository.

== Testing Environment and Tools ==
In [https://github.com/mbland/openssl/compare/test-util the test-util branch of Mike Bland's fork] are a few helper files that you may wish to copy or pull into your branch. As they are incorporated into the main OpenSSL repository, links to the mainline versions will be incorporated into the descriptions below. However, there may be newer versions in Mike's fork.

=== test/testutil.h ===
[http://git.openssl.org/gitweb/?p=openssl.git;a=blob;f=test/testutil.h test/testutil.h] defines the generic <code>SETUP_TEST_FIXTURE</code> and <code>EXECUTE_TEST</code> macros. The header comments describe how to define test-specific macros based on these.

=== test/new-test.sh ===
[https://github.com/mbland/openssl/blob/test-util/test/new-test.sh test/new-test.sh (mbland's fork)] generates a new automated test stub, following [http://mike-bland.com/2014/06/05/pseudo-xunit-pattern.html the Pseudo-xUnit Pattern] and using the macros from <code>test/testutil.h</code>. The generated stub will compile standalone.

=== test/test_env.bash ===
[https://github.com/mbland/openssl/blob/test-util/test/test_env.bash test/test_env.bash (mbland's fork)]: contains environment variables, functions, and aliases to help with OpenSSL unit testing. The header comments contain documentation on each of the functions and aliases in the file.

== Cscope ==
[http://cscope.sourceforge.net/ Cscope] is a powerful C source code browser. Once you have it installed, you can use the <code>make-cscope</code> and <code>open-cscope</code> functions from <code>test/test_env.bash</code>.

=== Vim Integration ===
Once <code>$CSCOPE_DB</code> (defined in <code>test/test_env.bash</code>) is built using <code>make-cscope</code>, you can take advantage of the Cscope integration in Vim by adding something like the following to your <code>.vimrc</code>:

<pre>
" Detect whether cscope features are present and whether we should add a
" connection to an existing cscope.out database file
if has("cscope")
set nocsverb
if filereadable("./cscope.out")
cs add cscope.out
elseif $CSCOPE_DB != ""
if filereadable($CSCOPE_DB)
cs add $CSCOPE_DB
else
echo "Can't read $CSCOPE_DB; cscope add not run"
endif
endif
set csre
set csverb
endif
</pre>

== Ctags ==
Ctags generates an index of symbols that many editors can use to quickly navigate the source code. Though many Unices ship with a version installed at <code>/usr/bin/ctags</code>, <code>test_env.bash</code> presumes that [http://ctags.sourceforge.net/ Exuberant Ctags] is installed.

=== Vim Integration ===
Add this to your <code>.vimrc</code> to take advantage of Ctags integration in vim (where <code>$TAGS_FILE</code> is defined in <code>test/test_env.bash</code>):

<pre>
" The tags path, as set below, will search the local file first, then the
" project-wide file.
if $TAGS_FILE != ""
if filereadable($TAGS_FILE)
set tags+=$TAGS_FILE
else
echo "Can't read $TAGS_FILE; tags not set"
endif
endif
</pre>

== git-new-workdir ==
Usually installed as <code>/usr/local/share/git-core/contrib/workdir/git-new-workdir</code>, this will create a new working directory for a specified branch that's linked to the original repository. This is nice when working on multiple branches in parallel, as it doesn't require committing or stashing changes before issuing a <code>git checkout</code> to switch between branches. The following <code>bash</code> function wraps <code>git-new-workdir</code> to create a working dir called <code>reponame-branchname</code> in the same directory as the original repository:

<pre>
# Creates a branch-specific git working directory in the same directory as the
# original repository.
git-new-workdir() {
if test $# -ne 2; then
echo "Usage: $FUNCNAME <git repo> <branch name>"
return 1
elif test ! -d $1; then
echo "$1 does not exist"
return 1
fi
new_workdir=$1-$2
if /usr/local/share/git-core/contrib/workdir/git-new-workdir\
$1 $new_workdir $2; then
echo "Created $new_workdir"
else
return 1
fi
}
</pre>

'''Note:''' On Ubuntu 14.04, <code>git-new-workdir</code> is installed as part of the <code>git</code> package as <code>/usr/share/doc/git/contrib/workdir/git-new-workdir</code> and does not have execute permission by default.

== tmux Terminal Multiplexer ==
[http://tmux.sourceforge.net/ The tmux terminal multiplexer] is helpful for using a single terminal window to efficiently manage different editing and build sessions, amongst many other useful functions. [http://pragprog.com/book/bhtmux/tmux tmux: Productive Mouse-Free Development] from The Pragmatic Bookshelf is a great primer.

These are some helpful config options to add to <code>~/.tmux.conf</code>, including getting GNU screen-like CTRL-a behavior:

<pre>
set-option -g prefix C-a
unbind-key C-b
bind-key C-a send-prefix

bind R source-file ~/.tmux.conf \; display "Reloaded ~/.tmux.conf"
set -g default-terminal "screen-256color"
set -g status-style fg=black,bg=colour7
setw -g window-status-style fg=default,bg=default,none
setw -g window-status-current-style fg=black,bg=colour15,bold
set -g pane-active-border-style fg=default,bg=colour7
set -g message-style fg=black,bg=colour11,bright
set -g status-utf8 on
setw -g monitor-activity on
set -g visual-activity on

bind -r H resize-pane -L 5
bind -r J resize-pane -D 5
bind -r K resize-pane -U 5
bind -r L resize-pane -R 5

# Enables nohup to work. From:
# https://github.com/ChrisJohnsen/tmux-MacOSX-pasteboard
set-option -g default-command "reattach-to-user-namespace -l $SHELL"
</pre>

Unit Testing

2014-06-26T18:32:10Z

Mbland: /* Tech Writer */ Add Lisa Carey to the Tech Writer team! :-)

This page will track the effort to improve unit test and other automated test coverage for OpenSSL. Everyone is encouraged to help with this effort, but this page will track which community members have accepted specific responsibilities, and what tasks are currently underway.

== Goals ==

The goals of this project are:

* Establish a written contribution/code review policy that requires:
** tests for every new code change, except for very trivial changes that do not directly impact program logic (e.g. documentation fixes, string constant updates, build script tweaks); this can be limited to an agreed-upon subset of the code to start, but would hopefully be expanded to cover most, if not all, of the code base
** tests for every bugfix
** smaller, well-tested changes building up to a complete feature (as opposed to large and/or untested changes that implement a large, complex feature all at once)
* Establish a schedule of Build Cops and Tutors to begin enforcing rollbacks/fixes and to help shepherd new contributors through the testing process.
* Set up a series of publicly-accessible continuous integration servers using whatever system the community prefers (e.g. Jenkins, Buildbot, etc.). These can be dedicated hardware or possibly Amazon Web Services-based. Establish a written policy that these remain green at all times, and that breakages be rolled-back or fixed within no more than 30 minutes.

We can add more concrete goals to this list after we make progress on these. Getting people to own roles, setting up build infrastructure, and setting and enforcing policy are important first steps. Upon that foundation, we can begin to make serious progress towards improved test coverage, code quality, and defect prevention.

== Discussion List ==
[https://groups.google.com/d/forum/openssl-testing openssl-testing] is the mailing list dedicated to the unit/automated testing effort, to avoid clogging openssl-dev. Membership is open to whoever wishes to join, even if only to lurk. Ideally all testing discussions will eventually move to openssl-dev once we have processes, tools, conventions, etc. in place.

== Team ==

We can define new roles as needed. The success of this effort will be inversely proportional to the number of roles with any one individual's name next to them. Each role can have more than one person filling it, but needs specific people to fill them.

The point is not to tell people what to do and not do: it’s to establish clear responsibilities so everyone can run with them right away and begin applying their creative energy to solutions, rather than waste it figuring out what the hell is going on, what they should do to contribute, and who they need to coordinate with to get things done.

=== Organizer===

Contact: [mailto:mbland@acm.org Mike Bland]

Ensures the overall effort is making progress by facilitating communication, removing obstacles, and making hands-on contributions when necessary.

=== Tech Writer ===

Contacts: [mailto:mbland@acm.org Mike Bland], Sandeep Mankar, Chelsea Komlo, Graham Brooks, Lisa Carey

Maintains documentation of tools, techniques, outstanding issues, etc. May maintain a code review/testing style guide based on the community consensus. Also documents who currently fills each role, as well as who might've filled it in the past.

=== Buildmaster ===

Contacts: [mailto:mbland@acm.org Mike Bland], Isaac Truett, Dileep Bapat, Graham Brooks

Sets up and maintains automated builds and any infrastructure used to monitor them and report breakage.

=== Toolsmith ===

Contact: Tony Aiuto

Maintains any tools and frameworks that are a part of the core build process; proposes, integrates, and/or retires existing tools.

=== Tutor ===

Contacts: [mailto:mbland@acm.org Mike Bland], Darshan Mody, Tom Francis, Brian N. Makin, Edward Knapp, Dileep Bapat, Graham Brooks

Helps contributors, particularly new contributors, with the details of writing unit or other automated tests. May be assigned by the reviewer of a patch to help a contributor get a new change under test. Submission of the patch should require the approval of the tutor in most cases. (Should ideally be more than one person.)

== Tasks ==
The current list is posted in the [http://goo.gl/5u9nbw OpenSSL Testing Tasks Google Spreadsheet].

(Would be nice to enable the [http://www.mediawiki.org/wiki/Extension:Widget MediaWiki Widget extension] and the [http://www.mediawikiwidgets.org/Google_Spreadsheet Google Spreadsheet widget].)

== Howtos and Style Guides ==
Documents that provide background on specific testing issues and idioms should be linked from here.

* [[How To Write Unit Tests For OpenSSL]]
* [http://mike-bland.com/2014/06/05/pseudo-xunit-pattern.html The Pseudo-xUnit Pattern]
* [[Testing and Development Tools and Tips]]

== Discussion Topics ==
These are discussions related to some of the current testing tasks.

=== First Area of Focus ===
Matt Caswell: One criterion might be ease of testing. If you were going down that route you might choose some of the low level packages within libcrypto which are relatively self-contained. I'm not talking about the crypto algorithms themselves (there are at least *some* tests for those), I'm thinking more about bio, pem, x509, x509v3, etc.

Another criterion might be where you get the most value. I suspect we will get the most value from a really good set of unit tests for libssl. I suspect this might be a big ask to start off with (you would probably have to mock out a lot of stuff).

=== Build Systems/Machines ===
Where can we get the machines? Is someone willing to donate them? What system should we use?

=== Get Unit Tests to Build on Windows ===
Steve Henson on building ssl/heartbeat_test.c on Windows: The first problem is that __func__ isn't understood. Changing that to __FUNCTION__ resolves that easily enough.

The second problem is more serious. Windows (and a couple of other platforms IIRC) is strict about only exporting approved symbols from global headers in shared libraries and wont export internal only functions.

That means you get linker errors:

heartbeat_test.obj : error LNK2019: unresolved external symbol _ssl3_setup_buffe
rs referenced in function _set_up
heartbeat_test.obj : error LNK2019: unresolved external symbol _ssl_init_wbio_bu
ffer referenced in function _set_up
heartbeat_test.obj : error LNK2019: unresolved external symbol _tls1_process_hea
rtbeat referenced in function _set_up_tls
heartbeat_test.obj : error LNK2019: unresolved external symbol _dtls1_process_he
artbeat referenced in function _set_up_dtls
out32dll\heartbeat_test.exe : fatal error LNK1120: 4 unresolved externals
NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual Studio 11.0
\VC\BIN\link.EXE"' : return code '0x460'

Potential solutions:
* http://stackoverflow.com/questions/7759777/is-it-possible-to-test-internal-class-from-a-c-dll-using-mstest
* http://stackoverflow.com/questions/19934714/googletest-c-unit-testing-linking-error
* http://stackoverflow.com/questions/14010627/unresolved-externals-when-compiling-unit-tests-for-visual-c-2012

ifdef'd out (for now): https://github.com/openssl/openssl/commit/7ce79a5bfdbcd53ae75f85e94eed665a05b5dea3

=== How to Manage Private Symbols ===
Steve Henson: As this also affects other platforms a general solution might be in order.

One option is to link to static libraries as has been mentioned. If you have a
lot of unit tests you could end up with a lot of large binaries. Would need some
changes to the dreaded Windows build system too.

A different option would be to export the unit test functions through a
structure pointer which a function returns. So you'd have something like..

unit_test = SSL_get_unit_test_funcs();

unit_test.some_internal_funcction(...);

Where the actual structure is opaque to applications (so they aren't tempted to
poke around in internals) but visible to the unit test code. Then the unit test
code doesn't call any internal functions not defined in there and doesn't need
to load private headers (unless it needs them for some definitions).

Mike Bland: This is an interesting idea, though I'd expect the unit tests will
still need access to internal headers.

But I have a question: Why do any of the symbols need to be private?
Is that degree of encapsulation necessary, and does it really
discourage irresponsible clients? The source code is open, so people
can always build their own copy and depend on internals if they really
want to, and the default UNIX/Linux/OS X build (i.e. ./configure &&
make && make test) doesn't lock down internal symbols. Have
dependencies on internal APIs ever been a problem in practice?

I'm not saying I feel strongly either way, but it seems that clients
should be clear that the public API is what's available in
openssl/include, and shouldn't depend on anything else. I just don't
understand what making symbols private buys anyone in the context of
OpenSSL, in light of how it appears to complicate the ability to unit
test.

Not trying to be argumentative, I just want to understand the
rationale. I'll work with whatever environment I have to.

Tom Francis: Making the symbols private allows the link editors on Windows and pretty much every UNIX system that doesn't use GNU ld to:
1) Run much, much faster at link time (not a minor consideration for most of the software I've worked on); and
2) optimize the final binary to load much faster.
Last time I checked, there was a minor improvement for GNU ld for #1, but it's trivial compared to Windows and HP's link editor, e.g. (I can't overstate the difference there). The runtime improvement is also dramatic on Windows, HP-UX, and AIX (we're talking tens for milliseconds for libraries that are much smaller than OpenSSL -- never tried it with OpenSSL, too lazy to mark everything, after having just done so for the library I was really interested in).

If you want to avoid a bunch of small binaries, you could combine all the unit tests into a single test application, and use multiple runs of it for each test, similar to how the openssl binary is created; if you link the object files directly (or a static library), you avoid the private symbols issue. Of course, this may be an issue for some of the embedded systems, where the loading a large binary for running the tests might not be acceptable (but might be preferable to a bunch of binaries?). Mike Bland was concerned about parallelization, and for any kind of automated tests on commits, that's a big point, if you can guarantee that each system only compiles/executes a limited number of tests (it could be useful for OS vendors, too). For the average OpenSSL user, that's not really a consideration; an average user is more likely concerned with how long (wall clock) it takes to compile and run the tests, and depending on the size of a single binary v the size of each smaller binary, it'll vary from system to system (generally, it'll be many times faster to create a single large binary than a few hundred smaller binaries, but the larger the binary, the longer it takes to load -- at some point, the load time might well outweigh the hundreds of smaller binaries (assuming you execute the large binary several times, rather than have a method for it to execute all tests in a single go). That point is probably > 100MB for larger systems, but for an embedded system, it could be much smaller.

For Steve's second suggestion, I think my biggest concerns would be how to logically separate out a unit test method without including the test framework (maybe throw it all in there?). I kinda like the idea of keeping the test inside the libraries, but I'm not sure why -- maybe 'cause I recently spent a few months back in a java world, where it's normal to throw all your unit tests into the same jar for production. I'm not sure I like that, though. The overhead of a single function per "module" added to the exports is also unlikely to drastically decrease link-time or run-time performance, too (compared with exporting everything).

=== Improved Patch/Cherry-pick Process ===
Geoff Thorpe: (a) patches that are not applied/cherry-picked to all the branches they apply to, or (b) applied to more branches than they should be. A complication of (a) is that some patches may need to be reworked in order to apply (or in order to not break binary compatibility of a stable branch). The difficulty of (b) is determining when the nature of the change either breaks binary compatibility or causes some other behavioral deviation that is not in keeping with the "promises" of stable branches.

http://ispras.linuxbase.org/index.php/ABI_compliance_checker
https://github.com/lvc/abi-compliance-checker/

Geoff on 2014-05-30: ...we might move to a scheme whereby we only maintain one most-recent stable branch for production releases, with snapshots or whatever of the development head. Ie. two branches, one for maximum-compatibility/bug-fixes-only, and the other for more active development. The number of "stable" branches would thus be 1 rather than 'n'. The idea is that we may defer to downstream "slow-moving" users (eg. long-term releases of linux distros) the opportunity to step up and do the work of maintaining (cherry-picking, back-porting, ...) older branches. Ie. what they typically already do in-house, but the opportunity to do it under our roof and thus combine their efforts with any other similarly-interested parties. If there's noone prepared to do that, the logical conclusion is that there is probably no need for it (or so the argument goes).

=== Improve/Standardize Algorithm Testing ===
Steve Henson: While the algorithms are partly tested I don't think they're anything like comprehensive enough. The way they are handled is also haphazard and spread across multiple tests covering all sorts of odd APIs, some of which are deprecated.

In fact I'd say they're (with the odd exception) an example of how *not* to do things.

Examples rc2test tests RC2 it has various test vectors hard coded inconsistent indentation and use of low level APIs. The rc4test is a similar story and uses a different format to rc2test. The same tale is repeated all over the place with ideatest, rmdtest etc etc.

Then we come to the code which I think is closest to how things should be which is evp_test. It uses the standard high level EVP API the format is consistent (if undocumented) and adding new tests can be done simply by editing a text file.

It currently however only works for some algorithms and the text file format could be better. It currently doesn't handle public key algorithms or test variants of the same algorithm (e.g. AES-NI versus non AES-NI) either.

=== Develop SSL/TLS Stack and Other Tools ===
Steve Henson: The heartbeat bug is an example where you can perform some testing without a complete SSL/TLS stack. However looking through the code in heartbeat_test.c it uses a lot of functionality which would normally be strictly off-limits for an application.

Some bugs are far more subtle and harder to test. For example some involve doing *almost* everything right and then slipping in some illegal parameter or badly formatted message. If you want to do that in a standalone bit of code you need a near complete SSL/TLS stack.

I can think of a couple of ways to handle that. One is to have some kind of hooks in libssl so testing code can misbehave and send out illegal messages in controlled ways (I had a prototype "bad heartbeat" API which only added a couple of lines to libssl). An alternative would be to write some kind of tiny SSL/TLS code used only by the test programs and carefully tailored to permit such things.

Testing and Development Tools and Tips

2014-06-25T13:56:47Z

Mbland: /* test/new-test.sh */ Fix new-test.sh link

This is a collection of helpful tools and tips for navigating the OpenSSL code base and managing a local git repository.

== Testing Environment and Tools ==
In [https://github.com/mbland/openssl/compare/test-util the test-util branch of Mike Bland's fork] are a few helper files that you may wish to copy or pull into your branch. As they are incorporated into the main OpenSSL repository, links to the mainline versions will be incorporated into the descriptions below. However, there may be newer versions in Mike's fork.

=== test/testutil.h ===
[http://git.openssl.org/gitweb/?p=openssl.git;a=blob;f=test/testutil.h test/testutil.h] defines the generic <code>SETUP_TEST_FIXTURE</code> and <code>EXECUTE_TEST</code> macros. The header comments describe how to define test-specific macros based on these.

=== test/new-test.sh ===
[https://github.com/mbland/openssl/blob/test-util/test/new-test.sh test/new-test.sh (mbland's fork)] generates a new automated test stub, following [http://mike-bland.com/2014/06/05/pseudo-xunit-pattern.html the Pseudo-xUnit Pattern] and using the macros from <code>test/testutil.h</code>. The generated stub will compile standalone.

=== test/test_env.bash ===
[https://github.com/mbland/openssl/blob/test-util/test/test_env.bash test/test_env.bash (mbland's fork)]: contains environment variables, functions, and aliases to help with OpenSSL unit testing. The header comments contain documentation on each of the functions and aliases in the file.

== Cscope ==
[http://cscope.sourceforge.net/ Cscope] is a powerful C source code browser. Once you have it installed, you can use the <code>make-cscope</code> and <code>open-cscope</code> functions from <code>test/test_env.bash</code>.

=== Vim Integration ===
Once <code>$CSCOPE_DB</code> (defined in <code>test/test_env.bash</code>) is built using <code>make-cscope</code>, you can take advantage of the Cscope integration in Vim by adding something like the following to your <code>.vimrc</code>:

<pre>
" Detect whether cscope features are present and whether we should add a
" connection to an existing cscope.out database file
if has("cscope")
set nocsverb
if filereadable("./cscope.out")
cs add cscope.out
elseif $CSCOPE_DB != ""
if filereadable($CSCOPE_DB)
cs add $CSCOPE_DB
else
echo "Can't read $CSCOPE_DB; cscope add not run"
endif
endif
set csre
set csverb
endif
</pre>

== Ctags ==
Ctags generates an index of symbols that many editors can use to quickly navigate the source code. Though many Unices ship with a version installed at <code>/usr/bin/ctags</code>, <code>test_env.bash</code> presumes that [http://ctags.sourceforge.net/ Exuberant Ctags] is installed.

=== Vim Integration ===
Add this to your <code>.vimrc</code> to take advantage of Ctags integration in vim (where <code>$TAGS_FILE</code> is defined in <code>test/test_env.bash</code>):

<pre>
" The tags path, as set below, will search the local file first, then the
" project-wide file.
if $TAGS_FILE != ""
if filereadable($TAGS_FILE)
set tags+=$TAGS_FILE
else
echo "Can't read $TAGS_FILE; tags not set"
endif
endif
</pre>

== git-new-workdir ==
Usually installed as <code>/usr/local/share/git-core/contrib/workdir/git-new-workdir</code>, this will create a new working directory for a specified branch that's linked to the original repository. This is nice when working on multiple branches in parallel, as it doesn't require committing or stashing changes before issuing a <code>git checkout</code> to switch between branches. The following <code>bash</code> function wraps <code>git-new-workdir</code> to create a working dir called <code>reponame-branchname</code> in the same directory as the original repository:

<pre>
# Creates a branch-specific git working directory in the same directory as the
# original repository.
git-new-workdir() {
if test $# -ne 2; then
echo "Usage: $FUNCNAME <git repo> <branch name>"
return 1
elif test ! -d $1; then
echo "$1 does not exist"
return 1
fi
new_workdir=$1-$2
if /usr/local/share/git-core/contrib/workdir/git-new-workdir\
$1 $new_workdir $2; then
echo "Created $new_workdir"
else
return 1
fi
}
</pre>

== tmux Terminal Multiplexer ==
[http://tmux.sourceforge.net/ The tmux terminal multiplexer] is helpful for using a single terminal window to efficiently manage different editing and build sessions, amongst many other useful functions. [http://pragprog.com/book/bhtmux/tmux tmux: Productive Mouse-Free Development] from The Pragmatic Bookshelf is a great primer.

These are some helpful config options to add to <code>~/.tmux.conf</code>, including getting GNU screen-like CTRL-a behavior:

<pre>
set-option -g prefix C-a
unbind-key C-b
bind-key C-a send-prefix

bind R source-file ~/.tmux.conf \; display "Reloaded ~/.tmux.conf"
set -g default-terminal "screen-256color"
set -g status-style fg=black,bg=colour7
setw -g window-status-style fg=default,bg=default,none
setw -g window-status-current-style fg=black,bg=colour15,bold
set -g pane-active-border-style fg=default,bg=colour7
set -g message-style fg=black,bg=colour11,bright
set -g status-utf8 on
setw -g monitor-activity on
set -g visual-activity on

bind -r H resize-pane -L 5
bind -r J resize-pane -D 5
bind -r K resize-pane -U 5
bind -r L resize-pane -R 5

# Enables nohup to work. From:
# https://github.com/ChrisJohnsen/tmux-MacOSX-pasteboard
set-option -g default-command "reattach-to-user-namespace -l $SHELL"
</pre>

Testing and Development Tools and Tips

2014-06-25T13:56:14Z

Mbland: /* Testing Environment and Tools */ Add links to test scripts in mbland's test-util fork on GitHub

This is a collection of helpful tools and tips for navigating the OpenSSL code base and managing a local git repository.

== Testing Environment and Tools ==
In [https://github.com/mbland/openssl/compare/test-util the test-util branch of Mike Bland's fork] are a few helper files that you may wish to copy or pull into your branch. As they are incorporated into the main OpenSSL repository, links to the mainline versions will be incorporated into the descriptions below. However, there may be newer versions in Mike's fork.

=== test/testutil.h ===
[http://git.openssl.org/gitweb/?p=openssl.git;a=blob;f=test/testutil.h test/testutil.h] defines the generic <code>SETUP_TEST_FIXTURE</code> and <code>EXECUTE_TEST</code> macros. The header comments describe how to define test-specific macros based on these.

=== test/new-test.sh ===
[https://github.com/mbland/openssl/blob/test-util/test/test_env.bash test/new-test.sh (mbland's fork)] generates a new automated test stub, following [http://mike-bland.com/2014/06/05/pseudo-xunit-pattern.html the Pseudo-xUnit Pattern] and using the macros from <code>test/testutil.h</code>. The generated stub will compile standalone.

=== test/test_env.bash ===
[https://github.com/mbland/openssl/blob/test-util/test/test_env.bash test/test_env.bash (mbland's fork)]: contains environment variables, functions, and aliases to help with OpenSSL unit testing. The header comments contain documentation on each of the functions and aliases in the file.

== Cscope ==
[http://cscope.sourceforge.net/ Cscope] is a powerful C source code browser. Once you have it installed, you can use the <code>make-cscope</code> and <code>open-cscope</code> functions from <code>test/test_env.bash</code>.

=== Vim Integration ===
Once <code>$CSCOPE_DB</code> (defined in <code>test/test_env.bash</code>) is built using <code>make-cscope</code>, you can take advantage of the Cscope integration in Vim by adding something like the following to your <code>.vimrc</code>:

<pre>
" Detect whether cscope features are present and whether we should add a
" connection to an existing cscope.out database file
if has("cscope")
set nocsverb
if filereadable("./cscope.out")
cs add cscope.out
elseif $CSCOPE_DB != ""
if filereadable($CSCOPE_DB)
cs add $CSCOPE_DB
else
echo "Can't read $CSCOPE_DB; cscope add not run"
endif
endif
set csre
set csverb
endif
</pre>

== Ctags ==
Ctags generates an index of symbols that many editors can use to quickly navigate the source code. Though many Unices ship with a version installed at <code>/usr/bin/ctags</code>, <code>test_env.bash</code> presumes that [http://ctags.sourceforge.net/ Exuberant Ctags] is installed.

=== Vim Integration ===
Add this to your <code>.vimrc</code> to take advantage of Ctags integration in vim (where <code>$TAGS_FILE</code> is defined in <code>test/test_env.bash</code>):

<pre>
" The tags path, as set below, will search the local file first, then the
" project-wide file.
if $TAGS_FILE != ""
if filereadable($TAGS_FILE)
set tags+=$TAGS_FILE
else
echo "Can't read $TAGS_FILE; tags not set"
endif
endif
</pre>

== git-new-workdir ==
Usually installed as <code>/usr/local/share/git-core/contrib/workdir/git-new-workdir</code>, this will create a new working directory for a specified branch that's linked to the original repository. This is nice when working on multiple branches in parallel, as it doesn't require committing or stashing changes before issuing a <code>git checkout</code> to switch between branches. The following <code>bash</code> function wraps <code>git-new-workdir</code> to create a working dir called <code>reponame-branchname</code> in the same directory as the original repository:

<pre>
# Creates a branch-specific git working directory in the same directory as the
# original repository.
git-new-workdir() {
if test $# -ne 2; then
echo "Usage: $FUNCNAME <git repo> <branch name>"
return 1
elif test ! -d $1; then
echo "$1 does not exist"
return 1
fi
new_workdir=$1-$2
if /usr/local/share/git-core/contrib/workdir/git-new-workdir\
$1 $new_workdir $2; then
echo "Created $new_workdir"
else
return 1
fi
}
</pre>

== tmux Terminal Multiplexer ==
[http://tmux.sourceforge.net/ The tmux terminal multiplexer] is helpful for using a single terminal window to efficiently manage different editing and build sessions, amongst many other useful functions. [http://pragprog.com/book/bhtmux/tmux tmux: Productive Mouse-Free Development] from The Pragmatic Bookshelf is a great primer.

These are some helpful config options to add to <code>~/.tmux.conf</code>, including getting GNU screen-like CTRL-a behavior:

<pre>
set-option -g prefix C-a
unbind-key C-b
bind-key C-a send-prefix

bind R source-file ~/.tmux.conf \; display "Reloaded ~/.tmux.conf"
set -g default-terminal "screen-256color"
set -g status-style fg=black,bg=colour7
setw -g window-status-style fg=default,bg=default,none
setw -g window-status-current-style fg=black,bg=colour15,bold
set -g pane-active-border-style fg=default,bg=colour7
set -g message-style fg=black,bg=colour11,bright
set -g status-utf8 on
setw -g monitor-activity on
set -g visual-activity on

bind -r H resize-pane -L 5
bind -r J resize-pane -D 5
bind -r K resize-pane -U 5
bind -r L resize-pane -R 5

# Enables nohup to work. From:
# https://github.com/ChrisJohnsen/tmux-MacOSX-pasteboard
set-option -g default-command "reattach-to-user-namespace -l $SHELL"
</pre>

How To Write Unit Tests For OpenSSL

2014-06-18T20:40:19Z

Mbland: /* Style */ Add testutil.h section

This is an outline of the basic process and principles to follow when writing unit tests. This document will evolve quite a bit as the community gains experience.

==Mechanics ==
=== Forking the Repo ===
For now, people contributing new unit tests for uncovered code (as opposed to submitting tests with new code changes) should fork [https://github.com/mbland/openssl Mike Bland's GitHub repository] and issue [https://help.github.com/articles/using-pull-requests GitHub pull requests]. Remember to do all development on a topic branch (not <code>master</code>). Tests committed to this repo will occasionally get merged into the master OpenSSL repo.

Rationale per Matt Caswell: We should think about process here. If you are going off to recruit an army of people writing tests then I wonder if it is worth setting up a separate github repository whilst you are building up the tests. We can then merge in from that on a periodic basis. I wouldn't want availability of openssl team committers to be a bottle neck.

Set up the [https://github.com/openssl/openssl OpenSSL master repository] as your upstream remote as per [https://help.github.com/articles/fork-a-repo#step-3-configure-remotes GitHub's instructions on configuring remotes]:
<pre>
$ cd my-openssl-repo
$ git remote add upstream https://github.com/openssl/openssl.git
</pre>

You should see the following output from <code>git remove -v</code> (where <code>$USER</code> is your GitHub username):
<pre>
$ git remote -v
origin https://github.com/$USER/openssl.git (fetch)
origin https://github.com/$USER/openssl.git (push)
upstream https://github.com/openssl/openssl.git (fetch)
upstream https://github.com/openssl/openssl.git (push)
</pre>

=== Check out the Tools and Tips page ===
[[Testing and Development Tools and Tips]] has information on tools that may make navigating and building the code a bit easier.

=== Use the Test Template Generator ===
TODO(mbland): Get template generator checked-in. Maybe have a template generator for each library, e.g. <code>ssl/new-test.sh</code> that has additional setup boilerplate specific to the <code>ssl</code> library.

Use the <code>test/new-test.sh</code> script to generate a skeleton test file. ([https://github.com/mbland/openssl/compare/test-util pending in the test-util branch])

=== Add Makefile Targets ===
The following instructions use the <code>Makefile</code> targets for <code>ssl/heartbeat_test.c</code> as an example.

In the <code>Makefile</code> for the library containing the test, add the test source file to the <code>TEST</code> variable:

<pre>
# ssl/Makefile
TEST=ssltest.c heartbeat_test.c
</pre>

In <code>test/Makefile</code>:
* add a variable for the test target near the top of the file, right after the existing test variables
* use the variable to add an executable target to the <code>EXE</code> variable
* use the variable to add an object file target to the <code>OBJ</code> variable
* use the variable to add a source file target to the <code>SRC</code> variable
* add the test target to the <code>alltests</code> target
* add the target to execute the test
* add the target to build the test executable
<pre>
# test/Makefile
HEARTBEATTEST= heartbeat_test
EXE= ... $(HEARTBEATTEST)$(EXE_EXT)
OBJ= ... $(HEARTBEATTEST).o
SRC= ... $(HEARTBEATTEST).c
alltests: \
... test_heartbeat

test_heartbeat: $(HEARTBEATTEST)$(EXE_EXT)
../util/shlib_wrap.sh ./$(HEARTBEATTEST)

$(HEARTBEATTEST)$(EXE_EXT): $(HEARTBEATTEST).o $(DLIBCRYPTO)
@target=$(HEARTBEATTEST); $(BUILD_CMD)
</pre>

=== Run <code>make links && make depend</code> ===
Finally, run <code>make links && make depend</code> to link the new test into the <code>test/</code> directory and automatically generate the header file dependencies.

=== Building and Running the Test ===
If you're initially developing on Mac OS X or (for now) FreeBSD 10, just use the stock method of building and testing:
<pre>
$ ./config && make && make test

# To execute just one specific test
$ make TESTS=test_heartbeat test
</pre>

Ultimately the test will have to compile and pass with developer flags enabled:
<pre>
$ ./GitConfigure debug-ben-debug-64-clang
$ ./GitMake -j 2
$ ./GitMake test_heartbeat
</pre>

The above currently doesn't work on Mac OS X or FreeBSD 10; for now, you can install the 9.1 release of [http://www.freebsd.org/ FreeBSD], which uses Clang 3.1, via a virtualization platform such as [https://www.virtualbox.org/ VirtualBox]. (Mac OS X breaks, in part, due to the fact that it doesn't use the GNU assembler; <code>-fsanitize</code> appears to be ignored on FreeBSD 9.1/Clang 3.1, but later versions break because the Clang compiler will emit <code>-fsanitize</code> symbols but the <code>libasan</code> library [http://lists.freebsd.org/pipermail/freebsd-hackers/2013-December/043995.html has yet to be ported to FreeBSD].)

=== Keep your repo up-to-date ===
Periodically run the following to keep your branch up-to-date:
<pre>
$ git fetch upstream master
$ git rebase upstream/master
</pre>
This will pull all the updates from the master OpenSSL repository into your repository, then update your branch to apply your changes on top of the latest updates.

=== Send a pull request ===
When your test is ready, send a [https://help.github.com/articles/using-pull-requests GitHub pull request]. Note that the pull request should be based on the <code>tests</code> branch of Mike's repository, not <code>master</code>. (This should be the default; let Mike know if it doesn't appear to be!) We'll review the code, and when it's ready, it'll get merged into Mike's repository. From there, it will eventually get pulled into the master OpenSSL repository.

== Style ==
[http://mike-bland.com/2014/06/05/pseudo-xunit-pattern.html The Pseudo-xUnit Pattern] is that established by [http://git.openssl.org/gitweb/?p=openssl.git;a=blob;f=ssl/heartbeat_test.c ssl/heartbeat_test.c]. This pattern organizes code in a fashion reminiscent of the [http://en.wikipedia.org/wiki/XUnit xUnit] family of unit testing frameworks, without actually using a testing framework. This should lower the barrier to entry for people wanting to write unit tests, but enable a relatively easy migration to an xUnit-based framework if we decide to do so one day.

Some of the basic principles to follow are:

=== <code>#include</code> the header for the code under test first ===
Having the header file for the code under test appear as the first <code>#include</code> directive ensures that that file is self-contained, i.e. it includes every header file it depends on, rather than relying on client code to include its dependencies.

=== <code>#include "testutil.h"</code> should come second ===
<code>test/testutil.h</code> contains the helper macros used in writing OpenSSL tests. Since the tests will be linked into <code>test/</code> by the [[#Run_make_links_.26.26_make_depend|make links]] step, and built in the <code>test/</code> directory, the "testutil.h" file will appear to be in the same directory as the test file.

=== Define a fixture structure ===
The fixture structure should contain all of the inputs to the code under test and all of the expected result values. It should also contain a <code>const char*</code> for the name of the test case function that created it, to aid in error message formatting. Even though the fixture may contain dynamically-allocated members, the fixture itself should be copied by value to reduce the necessary degree of memory management in a small unit test program.

=== Define set_up() and tear_down() functions for the fixture ===
<code>set_up()</code> should return a newly-initialized test fixture structure. It should take the name of the test case as an argument (i.e. <code>__func__</code>) and assign it to the fixture. All of the fixture members should be initialized, which each test case function can then override as needed.

<code>tear_down()</code> should take the fixture as an argument and release any resources allocated by <code>set_up()</code>. It can also call any library-wide error printing routines (e.g. <code>ERR_print_errors_fp(stderr)</code>).

=== Use <code>SETUP_TEST_FIXTURE()</code> and <code>EXECUTE_TEST()</code> from <code>test/testutil.h</code> ===
Each test case function should call <code>set_up()</code> as its first statement, and should call <code>tear_down()</code> just before returning. This is handled in a uniform fashion when using the <code>SETUP_TEST_FIXTURE()</code> and <code>EXECUTE_TEST()</code> helper macros from [http://git.openssl.org/gitweb/?p=openssl.git;a=blob;f=test/testutil.h test/testutil.h]. See the comments in <code>test/testutil.h</code> for usage, and <code>ssl/heartbeat_test.c</code> for an example of how to wrap the macros for a specific unit test.

=== Use test case functions, not a table of fixtures ===
Individual test case functions that call a common execution function are much more readable and maintainable than a loop over a table of fixture structures. Explicit fixture variable assignments aid comprehension when reading a specific test case, which saves time and energy when trying to understand a test or diagnose a failure. When a new member is added to an existing fixture, <code>set_up()</code> can set a default for all test cases, and only the test cases that rely on that new member need to be updated.

=== Use very descriptive test case names ===
Give tests long, descriptive names that provide ample context for the details of the test case. Good test names are also help produce good error messages.

=== Group test cases into "suites" by naming convention ===
Give logically-related tests the same prefix, e.g. <code>test_dtls1_</code> and <code>test_tls1</code> from <code>ssl/heartbeat_test.c</code>. If need be, you can define suite-specific <code>set_up()</code> functions that call the common <code>set_up()</code> and elaborate on it. (This generally shouldn't be necessary for <code>tear_down()</code>.)

=== Keep individual test case functions focused on one thing ===
If the test name contains the word "and", consider breaking it into two or more separate test case functions.

=== Write very descriptive error messages ===
Include the test case function name in each error message, and explain in detail the context for the assertion that failed. Include the expected result (contained in the fixture structure) and the actual result returned from the code under test. Write helper methods for complex values as needed (e.g. <code>print_payload()</code> in <code>ssl/heartbeat_test.c</code>).

=== Return zero on success and one on failure ===
The return value will be used to tally the number of test cases that failed. Even if multiple assertions fail for a single test case, the result should be exactly one.

=== Add each test case function to the test runner function ===
Whatever function is used to execute the batch of test case functions, be that <code>main()</code> or a separate function called by <code>main()</code>, don't forget to add your test case functions to that function.

TODO(mbland): create some kind of automated script or editor macros?

=== Report the number of test cases that failed ===
Add up the total number of failed test cases in <code>main()</code> and report that number as the last error message of the test. <code>main()</code> should return <code>EXIT_FAILURE</code> if any test cases failed, <code>EXIT_SUCCESS</code> otherwise.

=== Disable for Windows (for now) ===
Until we solve the private-symbol problem on Windows, we will need to wrap our unit test code in the following <code>#ifdef</code> block:
<pre>
#if !defined(OPENSSL_SYS_WINDOWS)

/* All the test code, including main() */

int main(int argc, char *argv[])
{
return EXIT_SUCCESS;
}

#endif /* OPENSSL_NO_WINDOWS */
</pre>

== Samples ==
Existing tests that can be used as models for new tests.
* [http://git.openssl.org/gitweb/?p=openssl.git;a=blob;f=ssl/heartbeat_test.c ssl/heartbeat_test.c]: Follows [http://mike-bland.com/2014/06/05/pseudo-xunit-pattern.html the Pseudo-xUnit pattern]

How To Write Unit Tests For OpenSSL

2014-06-18T20:28:23Z

Mbland: /* Style */ Added section on #including the header for the code under test first

This is an outline of the basic process and principles to follow when writing unit tests. This document will evolve quite a bit as the community gains experience.

==Mechanics ==
=== Forking the Repo ===
For now, people contributing new unit tests for uncovered code (as opposed to submitting tests with new code changes) should fork [https://github.com/mbland/openssl Mike Bland's GitHub repository] and issue [https://help.github.com/articles/using-pull-requests GitHub pull requests]. Remember to do all development on a topic branch (not <code>master</code>). Tests committed to this repo will occasionally get merged into the master OpenSSL repo.

Rationale per Matt Caswell: We should think about process here. If you are going off to recruit an army of people writing tests then I wonder if it is worth setting up a separate github repository whilst you are building up the tests. We can then merge in from that on a periodic basis. I wouldn't want availability of openssl team committers to be a bottle neck.

Set up the [https://github.com/openssl/openssl OpenSSL master repository] as your upstream remote as per [https://help.github.com/articles/fork-a-repo#step-3-configure-remotes GitHub's instructions on configuring remotes]:
<pre>
$ cd my-openssl-repo
$ git remote add upstream https://github.com/openssl/openssl.git
</pre>

You should see the following output from <code>git remove -v</code> (where <code>$USER</code> is your GitHub username):
<pre>
$ git remote -v
origin https://github.com/$USER/openssl.git (fetch)
origin https://github.com/$USER/openssl.git (push)
upstream https://github.com/openssl/openssl.git (fetch)
upstream https://github.com/openssl/openssl.git (push)
</pre>

=== Check out the Tools and Tips page ===
[[Testing and Development Tools and Tips]] has information on tools that may make navigating and building the code a bit easier.

=== Use the Test Template Generator ===
TODO(mbland): Get template generator checked-in. Maybe have a template generator for each library, e.g. <code>ssl/new-test.sh</code> that has additional setup boilerplate specific to the <code>ssl</code> library.

Use the <code>test/new-test.sh</code> script to generate a skeleton test file. ([https://github.com/mbland/openssl/compare/test-util pending in the test-util branch])

=== Add Makefile Targets ===
The following instructions use the <code>Makefile</code> targets for <code>ssl/heartbeat_test.c</code> as an example.

In the <code>Makefile</code> for the library containing the test, add the test source file to the <code>TEST</code> variable:

<pre>
# ssl/Makefile
TEST=ssltest.c heartbeat_test.c
</pre>

In <code>test/Makefile</code>:
* add a variable for the test target near the top of the file, right after the existing test variables
* use the variable to add an executable target to the <code>EXE</code> variable
* use the variable to add an object file target to the <code>OBJ</code> variable
* use the variable to add a source file target to the <code>SRC</code> variable
* add the test target to the <code>alltests</code> target
* add the target to execute the test
* add the target to build the test executable
<pre>
# test/Makefile
HEARTBEATTEST= heartbeat_test
EXE= ... $(HEARTBEATTEST)$(EXE_EXT)
OBJ= ... $(HEARTBEATTEST).o
SRC= ... $(HEARTBEATTEST).c
alltests: \
... test_heartbeat

test_heartbeat: $(HEARTBEATTEST)$(EXE_EXT)
../util/shlib_wrap.sh ./$(HEARTBEATTEST)

$(HEARTBEATTEST)$(EXE_EXT): $(HEARTBEATTEST).o $(DLIBCRYPTO)
@target=$(HEARTBEATTEST); $(BUILD_CMD)
</pre>

=== Run <code>make links && make depend</code> ===
Finally, run <code>make links && make depend</code> to link the new test into the <code>test/</code> directory and automatically generate the header file dependencies.

=== Building and Running the Test ===
If you're initially developing on Mac OS X or (for now) FreeBSD 10, just use the stock method of building and testing:
<pre>
$ ./config && make && make test

# To execute just one specific test
$ make TESTS=test_heartbeat test
</pre>

Ultimately the test will have to compile and pass with developer flags enabled:
<pre>
$ ./GitConfigure debug-ben-debug-64-clang
$ ./GitMake -j 2
$ ./GitMake test_heartbeat
</pre>

The above currently doesn't work on Mac OS X or FreeBSD 10; for now, you can install the 9.1 release of [http://www.freebsd.org/ FreeBSD], which uses Clang 3.1, via a virtualization platform such as [https://www.virtualbox.org/ VirtualBox]. (Mac OS X breaks, in part, due to the fact that it doesn't use the GNU assembler; <code>-fsanitize</code> appears to be ignored on FreeBSD 9.1/Clang 3.1, but later versions break because the Clang compiler will emit <code>-fsanitize</code> symbols but the <code>libasan</code> library [http://lists.freebsd.org/pipermail/freebsd-hackers/2013-December/043995.html has yet to be ported to FreeBSD].)

=== Keep your repo up-to-date ===
Periodically run the following to keep your branch up-to-date:
<pre>
$ git fetch upstream master
$ git rebase upstream/master
</pre>
This will pull all the updates from the master OpenSSL repository into your repository, then update your branch to apply your changes on top of the latest updates.

=== Send a pull request ===
When your test is ready, send a [https://help.github.com/articles/using-pull-requests GitHub pull request]. Note that the pull request should be based on the <code>tests</code> branch of Mike's repository, not <code>master</code>. (This should be the default; let Mike know if it doesn't appear to be!) We'll review the code, and when it's ready, it'll get merged into Mike's repository. From there, it will eventually get pulled into the master OpenSSL repository.

== Style ==
[http://mike-bland.com/2014/06/05/pseudo-xunit-pattern.html The Pseudo-xUnit Pattern] is that established by [http://git.openssl.org/gitweb/?p=openssl.git;a=blob;f=ssl/heartbeat_test.c ssl/heartbeat_test.c]. This pattern organizes code in a fashion reminiscent of the [http://en.wikipedia.org/wiki/XUnit xUnit] family of unit testing frameworks, without actually using a testing framework. This should lower the barrier to entry for people wanting to write unit tests, but enable a relatively easy migration to an xUnit-based framework if we decide to do so one day.

Some of the basic principles to follow are:

=== <code>#include</code> the header for the code under test first ===
Having the header file for the code under test appear as the first <code>#include</code> directive ensures that that file is self-contained, i.e. it includes every header file it depends on, rather than relying on client code to include its dependencies.

=== Define a fixture structure ===
The fixture structure should contain all of the inputs to the code under test and all of the expected result values. It should also contain a <code>const char*</code> for the name of the test case function that created it, to aid in error message formatting. Even though the fixture may contain dynamically-allocated members, the fixture itself should be copied by value to reduce the necessary degree of memory management in a small unit test program.

=== Define set_up() and tear_down() functions for the fixture ===
<code>set_up()</code> should return a newly-initialized test fixture structure. It should take the name of the test case as an argument (i.e. <code>__func__</code>) and assign it to the fixture. All of the fixture members should be initialized, which each test case function can then override as needed.

<code>tear_down()</code> should take the fixture as an argument and release any resources allocated by <code>set_up()</code>. It can also call any library-wide error printing routines (e.g. <code>ERR_print_errors_fp(stderr)</code>).

=== Use <code>SETUP_TEST_FIXTURE()</code> and <code>EXECUTE_TEST()</code> from <code>test/testutil.h</code> ===
Each test case function should call <code>set_up()</code> as its first statement, and should call <code>tear_down()</code> just before returning. This is handled in a uniform fashion when using the <code>SETUP_TEST_FIXTURE()</code> and <code>EXECUTE_TEST()</code> helper macros from [http://git.openssl.org/gitweb/?p=openssl.git;a=blob;f=test/testutil.h test/testutil.h]. See the comments in <code>test/testutil.h</code> for usage, and <code>ssl/heartbeat_test.c</code> for an example of how to wrap the macros for a specific unit test.

=== Use test case functions, not a table of fixtures ===
Individual test case functions that call a common execution function are much more readable and maintainable than a loop over a table of fixture structures. Explicit fixture variable assignments aid comprehension when reading a specific test case, which saves time and energy when trying to understand a test or diagnose a failure. When a new member is added to an existing fixture, <code>set_up()</code> can set a default for all test cases, and only the test cases that rely on that new member need to be updated.

=== Use very descriptive test case names ===
Give tests long, descriptive names that provide ample context for the details of the test case. Good test names are also help produce good error messages.

=== Group test cases into "suites" by naming convention ===
Give logically-related tests the same prefix, e.g. <code>test_dtls1_</code> and <code>test_tls1</code> from <code>ssl/heartbeat_test.c</code>. If need be, you can define suite-specific <code>set_up()</code> functions that call the common <code>set_up()</code> and elaborate on it. (This generally shouldn't be necessary for <code>tear_down()</code>.)

=== Keep individual test case functions focused on one thing ===
If the test name contains the word "and", consider breaking it into two or more separate test case functions.

=== Write very descriptive error messages ===
Include the test case function name in each error message, and explain in detail the context for the assertion that failed. Include the expected result (contained in the fixture structure) and the actual result returned from the code under test. Write helper methods for complex values as needed (e.g. <code>print_payload()</code> in <code>ssl/heartbeat_test.c</code>).

=== Return zero on success and one on failure ===
The return value will be used to tally the number of test cases that failed. Even if multiple assertions fail for a single test case, the result should be exactly one.

=== Add each test case function to the test runner function ===
Whatever function is used to execute the batch of test case functions, be that <code>main()</code> or a separate function called by <code>main()</code>, don't forget to add your test case functions to that function.

TODO(mbland): create some kind of automated script or editor macros?

=== Report the number of test cases that failed ===
Add up the total number of failed test cases in <code>main()</code> and report that number as the last error message of the test. <code>main()</code> should return <code>EXIT_FAILURE</code> if any test cases failed, <code>EXIT_SUCCESS</code> otherwise.

=== Disable for Windows (for now) ===
Until we solve the private-symbol problem on Windows, we will need to wrap our unit test code in the following <code>#ifdef</code> block:
<pre>
#if !defined(OPENSSL_SYS_WINDOWS)

/* All the test code, including main() */

int main(int argc, char *argv[])
{
return EXIT_SUCCESS;
}

#endif /* OPENSSL_NO_WINDOWS */
</pre>

== Samples ==
Existing tests that can be used as models for new tests.
* [http://git.openssl.org/gitweb/?p=openssl.git;a=blob;f=ssl/heartbeat_test.c ssl/heartbeat_test.c]: Follows [http://mike-bland.com/2014/06/05/pseudo-xunit-pattern.html the Pseudo-xUnit pattern]

How To Write Unit Tests For OpenSSL

2014-06-18T20:13:49Z

Mbland: /* Run make depend */ Updated to clarify that 'make links' must be run first.

How To Write Unit Tests For OpenSSL

2014-06-16T16:15:33Z

Mbland: /* Send a pull request */ Mention that pull requests should be against the tests branch, which should be set by default.

How To Write Unit Tests For OpenSSL

2014-06-16T16:12:56Z

Mbland: /* Keep your repo up-to-date */

How To Write Unit Tests For OpenSSL

2014-06-16T16:06:16Z

Mbland: /* Mechanics */ Better git remote and rebase instructions.

How To Write Unit Tests For OpenSSL

2014-06-16T15:59:04Z

Mbland: /* Keep your repo up-to-date */

How To Write Unit Tests For OpenSSL

2014-06-16T15:56:33Z

Mbland: /* Keep your repo up-to-date */ Better explain how to keep up with changes in the master repository.

How To Write Unit Tests For OpenSSL

2014-06-16T15:42:31Z

Mbland: /* Building and Running the Test */

Testing and Development Tools and Tips

2014-06-10T19:19:35Z

Mbland: Typo fix in "Cscope > Vim Integration"

This is a collection of helpful tools and tips for navigating the OpenSSL code base and managing a local git repository.

== Testing Environment and Tools ==
In [https://github.com/mbland/openssl/compare/test-util the test-util branch of Mike Bland's fork] are a few helper files that you may wish to copy or pull into your branch. As they are incorporated into the main OpenSSL repository, links to the mainline versions will be incorporated into the descriptions below. However, there may be newer versions in Mike's fork.

=== test/testutil.h ===
[http://git.openssl.org/gitweb/?p=openssl.git;a=blob;f=test/testutil.h test/testutil.h] defines the generic <code>SETUP_TEST_FIXTURE</code> and <code>EXECUTE_TEST</code> macros. The header comments describe how to define test-specific macros based on these.

=== test/new-test.sh ===
Generates a new automated test stub, following [http://mike-bland.com/2014/06/05/pseudo-xunit-pattern.html the Pseudo-xUnit Pattern] and using the macros from <code>test/testutil.h</code>. The generated stub will compile standalone.

=== test/test_env.bash ===
Environment variables, functions, and aliases to help with OpenSSL unit testing. The header comments contain documentation on each of the functions and aliases in the file.

== Cscope ==
[http://cscope.sourceforge.net/ Cscope] is a powerful C source code browser. Once you have it installed, you can use the <code>make-cscope</code> and <code>open-cscope</code> functions from <code>test/test_env.bash</code>.

=== Vim Integration ===
Once <code>$CSCOPE_DB</code> (defined in <code>test/test_env.bash</code>) is built using <code>make-cscope</code>, you can take advantage of the Cscope integration in Vim by adding something like the following to your <code>.vimrc</code>:

<pre>
" Detect whether cscope features are present and whether we should add a
" connection to an existing cscope.out database file
if has("cscope")
set nocsverb
if filereadable("./cscope.out")
cs add cscope.out
elseif $CSCOPE_DB != ""
if filereadable($CSCOPE_DB)
cs add $CSCOPE_DB
else
echo "Can't read $CSCOPE_DB; cscope add not run"
endif
endif
set csre
set csverb
endif
</pre>

== Ctags ==
Ctags generates an index of symbols that many editors can use to quickly navigate the source code. Though many Unices ship with a version installed at <code>/usr/bin/ctags</code>, <code>test_env.bash</code> presumes that [http://ctags.sourceforge.net/ Exuberant Ctags] is installed.

=== Vim Integration ===
Add this to your <code>.vimrc</code> to take advantage of Ctags integration in vim (where <code>$TAGS_FILE</code> is defined in <code>test/test_env.bash</code>):

<pre>
" The tags path, as set below, will search the local file first, then the
" project-wide file.
if $TAGS_FILE != ""
if filereadable($TAGS_FILE)
set tags+=$TAGS_FILE
else
echo "Can't read $TAGS_FILE; tags not set"
endif
endif
</pre>

== git-new-workdir ==
Usually installed as <code>/usr/local/share/git-core/contrib/workdir/git-new-workdir</code>, this will create a new working directory for a specified branch that's linked to the original repository. This is nice when working on multiple branches in parallel, as it doesn't require committing or stashing changes before issuing a <code>git checkout</code> to switch between branches. The following <code>bash</code> function wraps <code>git-new-workdir</code> to create a working dir called <code>reponame-branchname</code> in the same directory as the original repository:

<pre>
# Creates a branch-specific git working directory in the same directory as the
# original repository.
git-new-workdir() {
if test $# -ne 2; then
echo "Usage: $FUNCNAME <git repo> <branch name>"
return 1
elif test ! -d $1; then
echo "$1 does not exist"
return 1
fi
new_workdir=$1-$2
if /usr/local/share/git-core/contrib/workdir/git-new-workdir\
$1 $new_workdir $2; then
echo "Created $new_workdir"
else
return 1
fi
}
</pre>

== tmux Terminal Multiplexer ==
[http://tmux.sourceforge.net/ The tmux terminal multiplexer] is helpful for using a single terminal window to efficiently manage different editing and build sessions, amongst many other useful functions. [http://pragprog.com/book/bhtmux/tmux tmux: Productive Mouse-Free Development] from The Pragmatic Bookshelf is a great primer.

These are some helpful config options to add to <code>~/.tmux.conf</code>, including getting GNU screen-like CTRL-a behavior:

<pre>
set-option -g prefix C-a
unbind-key C-b
bind-key C-a send-prefix

bind R source-file ~/.tmux.conf \; display "Reloaded ~/.tmux.conf"
set -g default-terminal "screen-256color"
set -g status-style fg=black,bg=colour7
setw -g window-status-style fg=default,bg=default,none
setw -g window-status-current-style fg=black,bg=colour15,bold
set -g pane-active-border-style fg=default,bg=colour7
set -g message-style fg=black,bg=colour11,bright
set -g status-utf8 on
setw -g monitor-activity on
set -g visual-activity on

bind -r H resize-pane -L 5
bind -r J resize-pane -D 5
bind -r K resize-pane -U 5
bind -r L resize-pane -R 5

# Enables nohup to work. From:
# https://github.com/ChrisJohnsen/tmux-MacOSX-pasteboard
set-option -g default-command "reattach-to-user-namespace -l $SHELL"
</pre>

Testing and Development Tools and Tips

2014-06-10T19:18:44Z

Mbland: Explain that $TAGS_FILE is defined in test/test_env.bash

This is a collection of helpful tools and tips for navigating the OpenSSL code base and managing a local git repository.

== Testing Environment and Tools ==
In [https://github.com/mbland/openssl/compare/test-util the test-util branch of Mike Bland's fork] are a few helper files that you may wish to copy or pull into your branch. As they are incorporated into the main OpenSSL repository, links to the mainline versions will be incorporated into the descriptions below. However, there may be newer versions in Mike's fork.

=== test/testutil.h ===
[http://git.openssl.org/gitweb/?p=openssl.git;a=blob;f=test/testutil.h test/testutil.h] defines the generic <code>SETUP_TEST_FIXTURE</code> and <code>EXECUTE_TEST</code> macros. The header comments describe how to define test-specific macros based on these.

=== test/new-test.sh ===
Generates a new automated test stub, following [http://mike-bland.com/2014/06/05/pseudo-xunit-pattern.html the Pseudo-xUnit Pattern] and using the macros from <code>test/testutil.h</code>. The generated stub will compile standalone.

=== test/test_env.bash ===
Environment variables, functions, and aliases to help with OpenSSL unit testing. The header comments contain documentation on each of the functions and aliases in the file.

== Cscope ==
[http://cscope.sourceforge.net/ Cscope] is a powerful C source code browser. Once you have it installed, you can use the <code>make-cscope</code> and <code>open-cscope</code> functions from <code>test/test_env.bash</code>.

=== Vim Integration ===
Once <code>$CSCOPE_DB</code> (defined in <code>test/test_env.bash</code> is built using <code>make-cscope</code>, you can take advantage of the Cscope integration in Vim by adding something like the following to your <code>.vimrc</code>:

<pre>
" Detect whether cscope features are present and whether we should add a
" connection to an existing cscope.out database file
if has("cscope")
set nocsverb
if filereadable("./cscope.out")
cs add cscope.out
elseif $CSCOPE_DB != ""
if filereadable($CSCOPE_DB)
cs add $CSCOPE_DB
else
echo "Can't read $CSCOPE_DB; cscope add not run"
endif
endif
set csre
set csverb
endif
</pre>

== Ctags ==
Ctags generates an index of symbols that many editors can use to quickly navigate the source code. Though many Unices ship with a version installed at <code>/usr/bin/ctags</code>, <code>test_env.bash</code> presumes that [http://ctags.sourceforge.net/ Exuberant Ctags] is installed.

=== Vim Integration ===
Add this to your <code>.vimrc</code> to take advantage of Ctags integration in vim (where <code>$TAGS_FILE</code> is defined in <code>test/test_env.bash</code>):

<pre>
" The tags path, as set below, will search the local file first, then the
" project-wide file.
if $TAGS_FILE != ""
if filereadable($TAGS_FILE)
set tags+=$TAGS_FILE
else
echo "Can't read $TAGS_FILE; tags not set"
endif
endif
</pre>

== git-new-workdir ==
Usually installed as <code>/usr/local/share/git-core/contrib/workdir/git-new-workdir</code>, this will create a new working directory for a specified branch that's linked to the original repository. This is nice when working on multiple branches in parallel, as it doesn't require committing or stashing changes before issuing a <code>git checkout</code> to switch between branches. The following <code>bash</code> function wraps <code>git-new-workdir</code> to create a working dir called <code>reponame-branchname</code> in the same directory as the original repository:

<pre>
# Creates a branch-specific git working directory in the same directory as the
# original repository.
git-new-workdir() {
if test $# -ne 2; then
echo "Usage: $FUNCNAME <git repo> <branch name>"
return 1
elif test ! -d $1; then
echo "$1 does not exist"
return 1
fi
new_workdir=$1-$2
if /usr/local/share/git-core/contrib/workdir/git-new-workdir\
$1 $new_workdir $2; then
echo "Created $new_workdir"
else
return 1
fi
}
</pre>

== tmux Terminal Multiplexer ==
[http://tmux.sourceforge.net/ The tmux terminal multiplexer] is helpful for using a single terminal window to efficiently manage different editing and build sessions, amongst many other useful functions. [http://pragprog.com/book/bhtmux/tmux tmux: Productive Mouse-Free Development] from The Pragmatic Bookshelf is a great primer.

These are some helpful config options to add to <code>~/.tmux.conf</code>, including getting GNU screen-like CTRL-a behavior:

<pre>
set-option -g prefix C-a
unbind-key C-b
bind-key C-a send-prefix

bind R source-file ~/.tmux.conf \; display "Reloaded ~/.tmux.conf"
set -g default-terminal "screen-256color"
set -g status-style fg=black,bg=colour7
setw -g window-status-style fg=default,bg=default,none
setw -g window-status-current-style fg=black,bg=colour15,bold
set -g pane-active-border-style fg=default,bg=colour7
set -g message-style fg=black,bg=colour11,bright
set -g status-utf8 on
setw -g monitor-activity on
set -g visual-activity on

bind -r H resize-pane -L 5
bind -r J resize-pane -D 5
bind -r K resize-pane -U 5
bind -r L resize-pane -R 5

# Enables nohup to work. From:
# https://github.com/ChrisJohnsen/tmux-MacOSX-pasteboard
set-option -g default-command "reattach-to-user-namespace -l $SHELL"
</pre>

Testing and Development Tools and Tips

2014-06-10T19:15:59Z

Mbland: Add link to PragProg tmux book

This is a collection of helpful tools and tips for navigating the OpenSSL code base and managing a local git repository.

== Testing Environment and Tools ==
In [https://github.com/mbland/openssl/compare/test-util the test-util branch of Mike Bland's fork] are a few helper files that you may wish to copy or pull into your branch. As they are incorporated into the main OpenSSL repository, links to the mainline versions will be incorporated into the descriptions below. However, there may be newer versions in Mike's fork.

=== test/testutil.h ===
[http://git.openssl.org/gitweb/?p=openssl.git;a=blob;f=test/testutil.h test/testutil.h] defines the generic <code>SETUP_TEST_FIXTURE</code> and <code>EXECUTE_TEST</code> macros. The header comments describe how to define test-specific macros based on these.

=== test/new-test.sh ===
Generates a new automated test stub, following [http://mike-bland.com/2014/06/05/pseudo-xunit-pattern.html the Pseudo-xUnit Pattern] and using the macros from <code>test/testutil.h</code>. The generated stub will compile standalone.

=== test/test_env.bash ===
Environment variables, functions, and aliases to help with OpenSSL unit testing. The header comments contain documentation on each of the functions and aliases in the file.

== Cscope ==
[http://cscope.sourceforge.net/ Cscope] is a powerful C source code browser. Once you have it installed, you can use the <code>make-cscope</code> and <code>open-cscope</code> functions from <code>test/test_env.bash</code>.

=== Vim Integration ===
Once <code>$CSCOPE_DB</code> (defined in <code>test/test_env.bash</code> is built using <code>make-cscope</code>, you can take advantage of the Cscope integration in Vim by adding something like the following to your <code>.vimrc</code>:

<pre>
" Detect whether cscope features are present and whether we should add a
" connection to an existing cscope.out database file
if has("cscope")
set nocsverb
if filereadable("./cscope.out")
cs add cscope.out
elseif $CSCOPE_DB != ""
if filereadable($CSCOPE_DB)
cs add $CSCOPE_DB
else
echo "Can't read $CSCOPE_DB; cscope add not run"
endif
endif
set csre
set csverb
endif
</pre>

== Ctags ==
Ctags generates an index of symbols that many editors can use to quickly navigate the source code. Though many Unices ship with a version installed at <code>/usr/bin/ctags</code>, <code>test_env.bash</code> presumes that [http://ctags.sourceforge.net/ Exuberant Ctags] is installed.

=== Vim Integration ===
Add this to your <code>.vimrc</code> to take advantage of Ctags integration in vim:

<pre>
" The tags path, as set below, will search the local file first, then the
" project-wide file.
if $TAGS_FILE != ""
if filereadable($TAGS_FILE)
set tags+=$TAGS_FILE
else
echo "Can't read $TAGS_FILE; tags not set"
endif
endif
</pre>

== git-new-workdir ==
Usually installed as <code>/usr/local/share/git-core/contrib/workdir/git-new-workdir</code>, this will create a new working directory for a specified branch that's linked to the original repository. This is nice when working on multiple branches in parallel, as it doesn't require committing or stashing changes before issuing a <code>git checkout</code> to switch between branches. The following <code>bash</code> function wraps <code>git-new-workdir</code> to create a working dir called <code>reponame-branchname</code> in the same directory as the original repository:

<pre>
# Creates a branch-specific git working directory in the same directory as the
# original repository.
git-new-workdir() {
if test $# -ne 2; then
echo "Usage: $FUNCNAME <git repo> <branch name>"
return 1
elif test ! -d $1; then
echo "$1 does not exist"
return 1
fi
new_workdir=$1-$2
if /usr/local/share/git-core/contrib/workdir/git-new-workdir\
$1 $new_workdir $2; then
echo "Created $new_workdir"
else
return 1
fi
}
</pre>

== tmux Terminal Multiplexer ==
[http://tmux.sourceforge.net/ The tmux terminal multiplexer] is helpful for using a single terminal window to efficiently manage different editing and build sessions, amongst many other useful functions. [http://pragprog.com/book/bhtmux/tmux tmux: Productive Mouse-Free Development] from The Pragmatic Bookshelf is a great primer.

These are some helpful config options to add to <code>~/.tmux.conf</code>, including getting GNU screen-like CTRL-a behavior:

<pre>
set-option -g prefix C-a
unbind-key C-b
bind-key C-a send-prefix

bind R source-file ~/.tmux.conf \; display "Reloaded ~/.tmux.conf"
set -g default-terminal "screen-256color"
set -g status-style fg=black,bg=colour7
setw -g window-status-style fg=default,bg=default,none
setw -g window-status-current-style fg=black,bg=colour15,bold
set -g pane-active-border-style fg=default,bg=colour7
set -g message-style fg=black,bg=colour11,bright
set -g status-utf8 on
setw -g monitor-activity on
set -g visual-activity on

bind -r H resize-pane -L 5
bind -r J resize-pane -D 5
bind -r K resize-pane -U 5
bind -r L resize-pane -R 5

# Enables nohup to work. From:
# https://github.com/ChrisJohnsen/tmux-MacOSX-pasteboard
set-option -g default-command "reattach-to-user-namespace -l $SHELL"
</pre>

Testing and Development Tools and Tips

2014-06-10T18:46:34Z

Mbland: Add note that test utils will be linked to mainline as they are integrated

This is a collection of helpful tools and tips for navigating the OpenSSL code base and managing a local git repository.

== Testing Environment and Tools ==
In [https://github.com/mbland/openssl/compare/test-util the test-util branch of Mike Bland's fork] are a few helper files that you may wish to copy or pull into your branch. As they are incorporated into the main OpenSSL repository, links to the mainline versions will be incorporated into the descriptions below. However, there may be newer versions in Mike's fork.

=== test/testutil.h ===
[http://git.openssl.org/gitweb/?p=openssl.git;a=blob;f=test/testutil.h test/testutil.h] defines the generic <code>SETUP_TEST_FIXTURE</code> and <code>EXECUTE_TEST</code> macros. The header comments describe how to define test-specific macros based on these.

=== test/new-test.sh ===
Generates a new automated test stub, following [http://mike-bland.com/2014/06/05/pseudo-xunit-pattern.html the Pseudo-xUnit Pattern] and using the macros from <code>test/testutil.h</code>. The generated stub will compile standalone.

=== test/test_env.bash ===
Environment variables, functions, and aliases to help with OpenSSL unit testing. The header comments contain documentation on each of the functions and aliases in the file.

== Cscope ==
[http://cscope.sourceforge.net/ Cscope] is a powerful C source code browser. Once you have it installed, you can use the <code>make-cscope</code> and <code>open-cscope</code> functions from <code>test/test_env.bash</code>.

=== Vim Integration ===
Once <code>$CSCOPE_DB</code> (defined in <code>test/test_env.bash</code> is built using <code>make-cscope</code>, you can take advantage of the Cscope integration in Vim by adding something like the following to your <code>.vimrc</code>:

<pre>
" Detect whether cscope features are present and whether we should add a
" connection to an existing cscope.out database file
if has("cscope")
set nocsverb
if filereadable("./cscope.out")
cs add cscope.out
elseif $CSCOPE_DB != ""
if filereadable($CSCOPE_DB)
cs add $CSCOPE_DB
else
echo "Can't read $CSCOPE_DB; cscope add not run"
endif
endif
set csre
set csverb
endif
</pre>

== Ctags ==
Ctags generates an index of symbols that many editors can use to quickly navigate the source code. Though many Unices ship with a version installed at <code>/usr/bin/ctags</code>, <code>test_env.bash</code> presumes that [http://ctags.sourceforge.net/ Exuberant Ctags] is installed.

=== Vim Integration ===
Add this to your <code>.vimrc</code> to take advantage of Ctags integration in vim:

<pre>
" The tags path, as set below, will search the local file first, then the
" project-wide file.
if $TAGS_FILE != ""
if filereadable($TAGS_FILE)
set tags+=$TAGS_FILE
else
echo "Can't read $TAGS_FILE; tags not set"
endif
endif
</pre>

== git-new-workdir ==
Usually installed as <code>/usr/local/share/git-core/contrib/workdir/git-new-workdir</code>, this will create a new working directory for a specified branch that's linked to the original repository. This is nice when working on multiple branches in parallel, as it doesn't require committing or stashing changes before issuing a <code>git checkout</code> to switch between branches. The following <code>bash</code> function wraps <code>git-new-workdir</code> to create a working dir called <code>reponame-branchname</code> in the same directory as the original repository:

<pre>
# Creates a branch-specific git working directory in the same directory as the
# original repository.
git-new-workdir() {
if test $# -ne 2; then
echo "Usage: $FUNCNAME <git repo> <branch name>"
return 1
elif test ! -d $1; then
echo "$1 does not exist"
return 1
fi
new_workdir=$1-$2
if /usr/local/share/git-core/contrib/workdir/git-new-workdir\
$1 $new_workdir $2; then
echo "Created $new_workdir"
else
return 1
fi
}
</pre>

== tmux Terminal Multiplexer ==
[http://tmux.sourceforge.net/ The tmux terminal multiplexer] is helpful for using a single terminal window to efficiently manage different editing and build sessions, amongst many other useful functions. These are some helpful config options to add to <code>~/.tmux.conf</code>, including getting GNU screen-like CTRL-a behavior:

<pre>
set-option -g prefix C-a
unbind-key C-b
bind-key C-a send-prefix

bind R source-file ~/.tmux.conf \; display "Reloaded ~/.tmux.conf"
set -g default-terminal "screen-256color"
set -g status-style fg=black,bg=colour7
setw -g window-status-style fg=default,bg=default,none
setw -g window-status-current-style fg=black,bg=colour15,bold
set -g pane-active-border-style fg=default,bg=colour7
set -g message-style fg=black,bg=colour11,bright
set -g status-utf8 on
setw -g monitor-activity on
set -g visual-activity on

bind -r H resize-pane -L 5
bind -r J resize-pane -D 5
bind -r K resize-pane -U 5
bind -r L resize-pane -R 5

# Enables nohup to work. From:
# https://github.com/ChrisJohnsen/tmux-MacOSX-pasteboard
set-option -g default-command "reattach-to-user-namespace -l $SHELL"
</pre>

How To Write Unit Tests For OpenSSL

2014-06-10T18:40:54Z

Mbland: Update "Use SETUP_TEST_FIXTURE() and EXECUTE_TEST() from test/testutil.h" after openssl/master commit 3ead9f3798cb6de93b9824d6f833da9f00d8f5d7

Testing and Development Tools and Tips

2014-06-10T18:38:36Z

Mbland: Update test/testutil.h now that's it's merged into openssl/master as commit 3ead9f3798cb6de93b9824d6f833da9f00d8f5d7

This is a collection of helpful tools and tips for navigating the OpenSSL code base and managing a local git repository.

== Testing Environment and Tools ==
In [https://github.com/mbland/openssl/compare/test-util the test-util branch of Mike Bland's fork] are a few helper files that you may wish to copy or pull into your branch.

=== test/testutil.h ===
[http://git.openssl.org/gitweb/?p=openssl.git;a=blob;f=test/testutil.h test/testutil.h] defines the generic <code>SETUP_TEST_FIXTURE</code> and <code>EXECUTE_TEST</code> macros. The header comments describe how to define test-specific macros based on these.

=== test/new-test.sh ===
Generates a new automated test stub, following [http://mike-bland.com/2014/06/05/pseudo-xunit-pattern.html the Pseudo-xUnit Pattern] and using the macros from <code>test/testutil.h</code>. The generated stub will compile standalone.

=== test/test_env.bash ===
Environment variables, functions, and aliases to help with OpenSSL unit testing. The header comments contain documentation on each of the functions and aliases in the file.

== Cscope ==
[http://cscope.sourceforge.net/ Cscope] is a powerful C source code browser. Once you have it installed, you can use the <code>make-cscope</code> and <code>open-cscope</code> functions from <code>test/test_env.bash</code>.

=== Vim Integration ===
Once <code>$CSCOPE_DB</code> (defined in <code>test/test_env.bash</code> is built using <code>make-cscope</code>, you can take advantage of the Cscope integration in Vim by adding something like the following to your <code>.vimrc</code>:

<pre>
" Detect whether cscope features are present and whether we should add a
" connection to an existing cscope.out database file
if has("cscope")
set nocsverb
if filereadable("./cscope.out")
cs add cscope.out
elseif $CSCOPE_DB != ""
if filereadable($CSCOPE_DB)
cs add $CSCOPE_DB
else
echo "Can't read $CSCOPE_DB; cscope add not run"
endif
endif
set csre
set csverb
endif
</pre>

== Ctags ==
Ctags generates an index of symbols that many editors can use to quickly navigate the source code. Though many Unices ship with a version installed at <code>/usr/bin/ctags</code>, <code>test_env.bash</code> presumes that [http://ctags.sourceforge.net/ Exuberant Ctags] is installed.

=== Vim Integration ===
Add this to your <code>.vimrc</code> to take advantage of Ctags integration in vim:

<pre>
" The tags path, as set below, will search the local file first, then the
" project-wide file.
if $TAGS_FILE != ""
if filereadable($TAGS_FILE)
set tags+=$TAGS_FILE
else
echo "Can't read $TAGS_FILE; tags not set"
endif
endif
</pre>

== git-new-workdir ==
Usually installed as <code>/usr/local/share/git-core/contrib/workdir/git-new-workdir</code>, this will create a new working directory for a specified branch that's linked to the original repository. This is nice when working on multiple branches in parallel, as it doesn't require committing or stashing changes before issuing a <code>git checkout</code> to switch between branches. The following <code>bash</code> function wraps <code>git-new-workdir</code> to create a working dir called <code>reponame-branchname</code> in the same directory as the original repository:

<pre>
# Creates a branch-specific git working directory in the same directory as the
# original repository.
git-new-workdir() {
if test $# -ne 2; then
echo "Usage: $FUNCNAME <git repo> <branch name>"
return 1
elif test ! -d $1; then
echo "$1 does not exist"
return 1
fi
new_workdir=$1-$2
if /usr/local/share/git-core/contrib/workdir/git-new-workdir\
$1 $new_workdir $2; then
echo "Created $new_workdir"
else
return 1
fi
}
</pre>

== tmux Terminal Multiplexer ==
[http://tmux.sourceforge.net/ The tmux terminal multiplexer] is helpful for using a single terminal window to efficiently manage different editing and build sessions, amongst many other useful functions. These are some helpful config options to add to <code>~/.tmux.conf</code>, including getting GNU screen-like CTRL-a behavior:

<pre>
set-option -g prefix C-a
unbind-key C-b
bind-key C-a send-prefix

bind R source-file ~/.tmux.conf \; display "Reloaded ~/.tmux.conf"
set -g default-terminal "screen-256color"
set -g status-style fg=black,bg=colour7
setw -g window-status-style fg=default,bg=default,none
setw -g window-status-current-style fg=black,bg=colour15,bold
set -g pane-active-border-style fg=default,bg=colour7
set -g message-style fg=black,bg=colour11,bright
set -g status-utf8 on
setw -g monitor-activity on
set -g visual-activity on

bind -r H resize-pane -L 5
bind -r J resize-pane -D 5
bind -r K resize-pane -U 5
bind -r L resize-pane -R 5

# Enables nohup to work. From:
# https://github.com/ChrisJohnsen/tmux-MacOSX-pasteboard
set-option -g default-command "reattach-to-user-namespace -l $SHELL"
</pre>

How To Write Unit Tests For OpenSSL

2014-06-10T17:13:50Z

Mbland: Add "Keep your repo up-to-date" and "Send a pull request" subsections

How To Write Unit Tests For OpenSSL

2014-06-10T17:04:03Z

Mbland: Add Check out the Tools and Tips page subsection

Testing and Development Tools and Tips

2014-06-10T17:00:42Z

Mbland: Add tmux section

This is a collection of helpful tools and tips for navigating the OpenSSL code base and managing a local git repository.

== Testing Environment and Tools ==
In [https://github.com/mbland/openssl/compare/test-util the test-util branch of Mike Bland's fork] are a few helper files that you may wish to copy or pull into your branch.

=== test/testutil.h ===
Defines the generic <code>SETUP_TEST_FIXTURE</code> and <code>EXECUTE_TEST</code> macros. Should be pulled into the mainline pending acceptance of [https://github.com/openssl/openssl/pull/126 pull request #126].

=== test/new-test.sh ===
Generates a new automated test stub, following [http://mike-bland.com/2014/06/05/pseudo-xunit-pattern.html the Pseudo-xUnit Pattern] and using the macros from <code>test/testutil.h</code>. The generated stub will compile standalone.

=== test/test_env.bash ===
Environment variables, functions, and aliases to help with OpenSSL unit testing. The header comments contain documentation on each of the functions and aliases in the file.

== Cscope ==
[http://cscope.sourceforge.net/ Cscope] is a powerful C source code browser. Once you have it installed, you can use the <code>make-cscope</code> and <code>open-cscope</code> functions from <code>test/test_env.bash</code>.

=== Vim Integration ===
Once <code>$CSCOPE_DB</code> (defined in <code>test/test_env.bash</code> is built using <code>make-cscope</code>, you can take advantage of the Cscope integration in Vim by adding something like the following to your <code>.vimrc</code>:

<pre>
" Detect whether cscope features are present and whether we should add a
" connection to an existing cscope.out database file
if has("cscope")
set nocsverb
if filereadable("./cscope.out")
cs add cscope.out
elseif $CSCOPE_DB != ""
if filereadable($CSCOPE_DB)
cs add $CSCOPE_DB
else
echo "Can't read $CSCOPE_DB; cscope add not run"
endif
endif
set csre
set csverb
endif
</pre>

== Ctags ==
Ctags generates an index of symbols that many editors can use to quickly navigate the source code. Though many Unices ship with a version installed at <code>/usr/bin/ctags</code>, <code>test_env.bash</code> presumes that [http://ctags.sourceforge.net/ Exuberant Ctags] is installed.

=== Vim Integration ===
Add this to your <code>.vimrc</code> to take advantage of Ctags integration in vim:

<pre>
" The tags path, as set below, will search the local file first, then the
" project-wide file.
if $TAGS_FILE != ""
if filereadable($TAGS_FILE)
set tags+=$TAGS_FILE
else
echo "Can't read $TAGS_FILE; tags not set"
endif
endif
</pre>

== git-new-workdir ==
Usually installed as <code>/usr/local/share/git-core/contrib/workdir/git-new-workdir</code>, this will create a new working directory for a specified branch that's linked to the original repository. This is nice when working on multiple branches in parallel, as it doesn't require committing or stashing changes before issuing a <code>git checkout</code> to switch between branches. The following <code>bash</code> function wraps <code>git-new-workdir</code> to create a working dir called <code>reponame-branchname</code> in the same directory as the original repository:

<pre>
# Creates a branch-specific git working directory in the same directory as the
# original repository.
git-new-workdir() {
if test $# -ne 2; then
echo "Usage: $FUNCNAME <git repo> <branch name>"
return 1
elif test ! -d $1; then
echo "$1 does not exist"
return 1
fi
new_workdir=$1-$2
if /usr/local/share/git-core/contrib/workdir/git-new-workdir\
$1 $new_workdir $2; then
echo "Created $new_workdir"
else
return 1
fi
}
</pre>

== tmux Terminal Multiplexer ==
[http://tmux.sourceforge.net/ The tmux terminal multiplexer] is helpful for using a single terminal window to efficiently manage different editing and build sessions, amongst many other useful functions. These are some helpful config options to add to <code>~/.tmux.conf</code>, including getting GNU screen-like CTRL-a behavior:

<pre>
set-option -g prefix C-a
unbind-key C-b
bind-key C-a send-prefix

bind R source-file ~/.tmux.conf \; display "Reloaded ~/.tmux.conf"
set -g default-terminal "screen-256color"
set -g status-style fg=black,bg=colour7
setw -g window-status-style fg=default,bg=default,none
setw -g window-status-current-style fg=black,bg=colour15,bold
set -g pane-active-border-style fg=default,bg=colour7
set -g message-style fg=black,bg=colour11,bright
set -g status-utf8 on
setw -g monitor-activity on
set -g visual-activity on

bind -r H resize-pane -L 5
bind -r J resize-pane -D 5
bind -r K resize-pane -U 5
bind -r L resize-pane -R 5

# Enables nohup to work. From:
# https://github.com/ChrisJohnsen/tmux-MacOSX-pasteboard
set-option -g default-command "reattach-to-user-namespace -l $SHELL"
</pre>

Testing and Development Tools and Tips

2014-06-10T16:56:02Z

Mbland: Flesh out git-new-workdir section

Testing and Development Tools and Tips

2014-06-10T16:51:27Z

Mbland: Flesh out Ctags section

Testing and Development Tools and Tips

2014-06-10T16:48:03Z

Mbland: Flesh out Cscope section

Testing and Development Tools and Tips

2014-06-10T16:43:56Z

Mbland: Flesh out Testing Environment and Tools

Testing and Development Tools and Tips

2014-06-10T16:27:27Z

Mbland: Created page, initial sections

This is a collection of helpful tools and tips for navigating the OpenSSL code base and managing a local git repository.

== Testing Environment and Tools ==

== Cscope ==
=== Vim Integration ===

== Ctags ==
=== Vim Integration ===

== git-new-workdir ==

Unit Testing

2014-06-10T16:24:39Z

Mbland: /* Howtos and Style Guides */

This page will track the effort to improve unit test and other automated test coverage for OpenSSL. Everyone is encouraged to help with this effort, but this page will track which community members have accepted specific responsibilities, and what tasks are currently underway.

== Goals ==

The goals of this project are:

* Establish a written contribution/code review policy that requires:
** tests for every new code change, except for very trivial changes that do not directly impact program logic (e.g. documentation fixes, string constant updates, build script tweaks); this can be limited to an agreed-upon subset of the code to start, but would hopefully be expanded to cover most, if not all, of the code base
** tests for every bugfix
** smaller, well-tested changes building up to a complete feature (as opposed to large and/or untested changes that implement a large, complex feature all at once)
* Establish a schedule of Build Cops and Tutors to begin enforcing rollbacks/fixes and to help shepherd new contributors through the testing process.
* Set up a series of publicly-accessible continuous integration servers using whatever system the community prefers (e.g. Jenkins, Buildbot, etc.). These can be dedicated hardware or possibly Amazon Web Services-based. Establish a written policy that these remain green at all times, and that breakages be rolled-back or fixed within no more than 30 minutes.

We can add more concrete goals to this list after we make progress on these. Getting people to own roles, setting up build infrastructure, and setting and enforcing policy are important first steps. Upon that foundation, we can begin to make serious progress towards improved test coverage, code quality, and defect prevention.

== Discussion List ==
[https://groups.google.com/d/forum/openssl-testing openssl-testing] is the mailing list dedicated to the unit/automated testing effort, to avoid clogging openssl-dev. Membership is open to whoever wishes to join, even if only to lurk. Ideally all testing discussions will eventually move to openssl-dev once we have processes, tools, conventions, etc. in place.

== Team ==

We can define new roles as needed. The success of this effort will be inversely proportional to the number of roles with any one individual's name next to them. Each role can have more than one person filling it, but needs specific people to fill them.

The point is not to tell people what to do and not do: it’s to establish clear responsibilities so everyone can run with them right away and begin applying their creative energy to solutions, rather than waste it figuring out what the hell is going on, what they should do to contribute, and who they need to coordinate with to get things done.

=== Organizer===

Contact: [mailto:mbland@acm.org Mike Bland]

Ensures the overall effort is making progress by facilitating communication, removing obstacles, and making hands-on contributions when necessary.

=== Tech Writer ===

Contacts: [mailto:mbland@acm.org Mike Bland], Sandeep Mankar, Chelsea Komlo, Graham Brooks

Maintains documentation of tools, techniques, outstanding issues, etc. May maintain a code review/testing style guide based on the community consensus. Also documents who currently fills each role, as well as who might've filled it in the past.

=== Buildmaster ===

Contacts: [mailto:mbland@acm.org Mike Bland], Isaac Truett, Dileep Bapat, Graham Brooks

Sets up and maintains automated builds and any infrastructure used to monitor them and report breakage.

=== Toolsmith ===

Contact: Tony Aiuto

Maintains any tools and frameworks that are a part of the core build process; proposes, integrates, and/or retires existing tools.

=== Tutor ===

Contacts: [mailto:mbland@acm.org Mike Bland], Darshan Mody, Tom Francis, Brian N. Makin, Chelsea Komlo, Edward Knapp, Dileep Bapat, Graham Brooks

Helps contributors, particularly new contributors, with the details of writing unit or other automated tests. May be assigned by the reviewer of a patch to help a contributor get a new change under test. Submission of the patch should require the approval of the tutor in most cases. (Should ideally be more than one person.)

== Tasks ==
The current list is posted in the [http://goo.gl/5u9nbw OpenSSL Testing Tasks Google Spreadsheet].

(Would be nice to enable the [http://www.mediawiki.org/wiki/Extension:Widget MediaWiki Widget extension] and the [http://www.mediawikiwidgets.org/Google_Spreadsheet Google Spreadsheet widget].)

== Howtos and Style Guides ==
Documents that provide background on specific testing issues and idioms should be linked from here.

* [[How To Write Unit Tests For OpenSSL]]
* [http://mike-bland.com/2014/06/05/pseudo-xunit-pattern.html The Pseudo-xUnit Pattern]
* [[Testing and Development Tools and Tips]]

== Discussion Topics ==
These are discussions related to some of the current testing tasks.

=== First Area of Focus ===
Matt Caswell: One criterion might be ease of testing. If you were going down that route you might choose some of the low level packages within libcrypto which are relatively self-contained. I'm not talking about the crypto algorithms themselves (there are at least *some* tests for those), I'm thinking more about bio, pem, x509, x509v3, etc.

Another criterion might be where you get the most value. I suspect we will get the most value from a really good set of unit tests for libssl. I suspect this might be a big ask to start off with (you would probably have to mock out a lot of stuff).

=== Build Systems/Machines ===
Where can we get the machines? Is someone willing to donate them? What system should we use?

=== Get Unit Tests to Build on Windows ===
Steve Henson on building ssl/heartbeat_test.c on Windows: The first problem is that __func__ isn't understood. Changing that to __FUNCTION__ resolves that easily enough.

The second problem is more serious. Windows (and a couple of other platforms IIRC) is strict about only exporting approved symbols from global headers in shared libraries and wont export internal only functions.

That means you get linker errors:

heartbeat_test.obj : error LNK2019: unresolved external symbol _ssl3_setup_buffe
rs referenced in function _set_up
heartbeat_test.obj : error LNK2019: unresolved external symbol _ssl_init_wbio_bu
ffer referenced in function _set_up
heartbeat_test.obj : error LNK2019: unresolved external symbol _tls1_process_hea
rtbeat referenced in function _set_up_tls
heartbeat_test.obj : error LNK2019: unresolved external symbol _dtls1_process_he
artbeat referenced in function _set_up_dtls
out32dll\heartbeat_test.exe : fatal error LNK1120: 4 unresolved externals
NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual Studio 11.0
\VC\BIN\link.EXE"' : return code '0x460'

Potential solutions:
* http://stackoverflow.com/questions/7759777/is-it-possible-to-test-internal-class-from-a-c-dll-using-mstest
* http://stackoverflow.com/questions/19934714/googletest-c-unit-testing-linking-error
* http://stackoverflow.com/questions/14010627/unresolved-externals-when-compiling-unit-tests-for-visual-c-2012

ifdef'd out (for now): https://github.com/openssl/openssl/commit/7ce79a5bfdbcd53ae75f85e94eed665a05b5dea3

=== How to Manage Private Symbols ===
Steve Henson: As this also affects other platforms a general solution might be in order.

One option is to link to static libraries as has been mentioned. If you have a
lot of unit tests you could end up with a lot of large binaries. Would need some
changes to the dreaded Windows build system too.

A different option would be to export the unit test functions through a
structure pointer which a function returns. So you'd have something like..

unit_test = SSL_get_unit_test_funcs();

unit_test.some_internal_funcction(...);

Where the actual structure is opaque to applications (so they aren't tempted to
poke around in internals) but visible to the unit test code. Then the unit test
code doesn't call any internal functions not defined in there and doesn't need
to load private headers (unless it needs them for some definitions).

Mike Bland: This is an interesting idea, though I'd expect the unit tests will
still need access to internal headers.

But I have a question: Why do any of the symbols need to be private?
Is that degree of encapsulation necessary, and does it really
discourage irresponsible clients? The source code is open, so people
can always build their own copy and depend on internals if they really
want to, and the default UNIX/Linux/OS X build (i.e. ./configure &&
make && make test) doesn't lock down internal symbols. Have
dependencies on internal APIs ever been a problem in practice?

I'm not saying I feel strongly either way, but it seems that clients
should be clear that the public API is what's available in
openssl/include, and shouldn't depend on anything else. I just don't
understand what making symbols private buys anyone in the context of
OpenSSL, in light of how it appears to complicate the ability to unit
test.

Not trying to be argumentative, I just want to understand the
rationale. I'll work with whatever environment I have to.

Tom Francis: Making the symbols private allows the link editors on Windows and pretty much every UNIX system that doesn't use GNU ld to:
1) Run much, much faster at link time (not a minor consideration for most of the software I've worked on); and
2) optimize the final binary to load much faster.
Last time I checked, there was a minor improvement for GNU ld for #1, but it's trivial compared to Windows and HP's link editor, e.g. (I can't overstate the difference there). The runtime improvement is also dramatic on Windows, HP-UX, and AIX (we're talking tens for milliseconds for libraries that are much smaller than OpenSSL -- never tried it with OpenSSL, too lazy to mark everything, after having just done so for the library I was really interested in).

If you want to avoid a bunch of small binaries, you could combine all the unit tests into a single test application, and use multiple runs of it for each test, similar to how the openssl binary is created; if you link the object files directly (or a static library), you avoid the private symbols issue. Of course, this may be an issue for some of the embedded systems, where the loading a large binary for running the tests might not be acceptable (but might be preferable to a bunch of binaries?). Mike Bland was concerned about parallelization, and for any kind of automated tests on commits, that's a big point, if you can guarantee that each system only compiles/executes a limited number of tests (it could be useful for OS vendors, too). For the average OpenSSL user, that's not really a consideration; an average user is more likely concerned with how long (wall clock) it takes to compile and run the tests, and depending on the size of a single binary v the size of each smaller binary, it'll vary from system to system (generally, it'll be many times faster to create a single large binary than a few hundred smaller binaries, but the larger the binary, the longer it takes to load -- at some point, the load time might well outweigh the hundreds of smaller binaries (assuming you execute the large binary several times, rather than have a method for it to execute all tests in a single go). That point is probably > 100MB for larger systems, but for an embedded system, it could be much smaller.

For Steve's second suggestion, I think my biggest concerns would be how to logically separate out a unit test method without including the test framework (maybe throw it all in there?). I kinda like the idea of keeping the test inside the libraries, but I'm not sure why -- maybe 'cause I recently spent a few months back in a java world, where it's normal to throw all your unit tests into the same jar for production. I'm not sure I like that, though. The overhead of a single function per "module" added to the exports is also unlikely to drastically decrease link-time or run-time performance, too (compared with exporting everything).

=== Improved Patch/Cherry-pick Process ===
Geoff Thorpe: (a) patches that are not applied/cherry-picked to all the branches they apply to, or (b) applied to more branches than they should be. A complication of (a) is that some patches may need to be reworked in order to apply (or in order to not break binary compatibility of a stable branch). The difficulty of (b) is determining when the nature of the change either breaks binary compatibility or causes some other behavioral deviation that is not in keeping with the "promises" of stable branches.

http://ispras.linuxbase.org/index.php/ABI_compliance_checker
https://github.com/lvc/abi-compliance-checker/

Geoff on 2014-05-30: ...we might move to a scheme whereby we only maintain one most-recent stable branch for production releases, with snapshots or whatever of the development head. Ie. two branches, one for maximum-compatibility/bug-fixes-only, and the other for more active development. The number of "stable" branches would thus be 1 rather than 'n'. The idea is that we may defer to downstream "slow-moving" users (eg. long-term releases of linux distros) the opportunity to step up and do the work of maintaining (cherry-picking, back-porting, ...) older branches. Ie. what they typically already do in-house, but the opportunity to do it under our roof and thus combine their efforts with any other similarly-interested parties. If there's noone prepared to do that, the logical conclusion is that there is probably no need for it (or so the argument goes).

=== Improve/Standardize Algorithm Testing ===
Steve Henson: While the algorithms are partly tested I don't think they're anything like comprehensive enough. The way they are handled is also haphazard and spread across multiple tests covering all sorts of odd APIs, some of which are deprecated.

In fact I'd say they're (with the odd exception) an example of how *not* to do things.

Examples rc2test tests RC2 it has various test vectors hard coded inconsistent indentation and use of low level APIs. The rc4test is a similar story and uses a different format to rc2test. The same tale is repeated all over the place with ideatest, rmdtest etc etc.

Then we come to the code which I think is closest to how things should be which is evp_test. It uses the standard high level EVP API the format is consistent (if undocumented) and adding new tests can be done simply by editing a text file.

It currently however only works for some algorithms and the text file format could be better. It currently doesn't handle public key algorithms or test variants of the same algorithm (e.g. AES-NI versus non AES-NI) either.

=== Develop SSL/TLS Stack and Other Tools ===
Steve Henson: The heartbeat bug is an example where you can perform some testing without a complete SSL/TLS stack. However looking through the code in heartbeat_test.c it uses a lot of functionality which would normally be strictly off-limits for an application.

Some bugs are far more subtle and harder to test. For example some involve doing *almost* everything right and then slipping in some illegal parameter or badly formatted message. If you want to do that in a standalone bit of code you need a near complete SSL/TLS stack.

I can think of a couple of ways to handle that. One is to have some kind of hooks in libssl so testing code can misbehave and send out illegal messages in controlled ways (I had a prototype "bad heartbeat" API which only added a couple of lines to libssl). An alternative would be to write some kind of tiny SSL/TLS code used only by the test programs and carefully tailored to permit such things.

Unit Testing

2014-06-10T16:23:51Z

Mbland: Link to new Testing and Development Tools and Tips page

This page will track the effort to improve unit test and other automated test coverage for OpenSSL. Everyone is encouraged to help with this effort, but this page will track which community members have accepted specific responsibilities, and what tasks are currently underway.

== Goals ==

The goals of this project are:

* Establish a written contribution/code review policy that requires:
** tests for every new code change, except for very trivial changes that do not directly impact program logic (e.g. documentation fixes, string constant updates, build script tweaks); this can be limited to an agreed-upon subset of the code to start, but would hopefully be expanded to cover most, if not all, of the code base
** tests for every bugfix
** smaller, well-tested changes building up to a complete feature (as opposed to large and/or untested changes that implement a large, complex feature all at once)
* Establish a schedule of Build Cops and Tutors to begin enforcing rollbacks/fixes and to help shepherd new contributors through the testing process.
* Set up a series of publicly-accessible continuous integration servers using whatever system the community prefers (e.g. Jenkins, Buildbot, etc.). These can be dedicated hardware or possibly Amazon Web Services-based. Establish a written policy that these remain green at all times, and that breakages be rolled-back or fixed within no more than 30 minutes.

We can add more concrete goals to this list after we make progress on these. Getting people to own roles, setting up build infrastructure, and setting and enforcing policy are important first steps. Upon that foundation, we can begin to make serious progress towards improved test coverage, code quality, and defect prevention.

== Discussion List ==
[https://groups.google.com/d/forum/openssl-testing openssl-testing] is the mailing list dedicated to the unit/automated testing effort, to avoid clogging openssl-dev. Membership is open to whoever wishes to join, even if only to lurk. Ideally all testing discussions will eventually move to openssl-dev once we have processes, tools, conventions, etc. in place.

== Team ==

We can define new roles as needed. The success of this effort will be inversely proportional to the number of roles with any one individual's name next to them. Each role can have more than one person filling it, but needs specific people to fill them.

The point is not to tell people what to do and not do: it’s to establish clear responsibilities so everyone can run with them right away and begin applying their creative energy to solutions, rather than waste it figuring out what the hell is going on, what they should do to contribute, and who they need to coordinate with to get things done.

=== Organizer===

Contact: [mailto:mbland@acm.org Mike Bland]

Ensures the overall effort is making progress by facilitating communication, removing obstacles, and making hands-on contributions when necessary.

=== Tech Writer ===

Contacts: [mailto:mbland@acm.org Mike Bland], Sandeep Mankar, Chelsea Komlo, Graham Brooks

Maintains documentation of tools, techniques, outstanding issues, etc. May maintain a code review/testing style guide based on the community consensus. Also documents who currently fills each role, as well as who might've filled it in the past.

=== Buildmaster ===

Contacts: [mailto:mbland@acm.org Mike Bland], Isaac Truett, Dileep Bapat, Graham Brooks

Sets up and maintains automated builds and any infrastructure used to monitor them and report breakage.

=== Toolsmith ===

Contact: Tony Aiuto

Maintains any tools and frameworks that are a part of the core build process; proposes, integrates, and/or retires existing tools.

=== Tutor ===

Contacts: [mailto:mbland@acm.org Mike Bland], Darshan Mody, Tom Francis, Brian N. Makin, Chelsea Komlo, Edward Knapp, Dileep Bapat, Graham Brooks

Helps contributors, particularly new contributors, with the details of writing unit or other automated tests. May be assigned by the reviewer of a patch to help a contributor get a new change under test. Submission of the patch should require the approval of the tutor in most cases. (Should ideally be more than one person.)

== Tasks ==
The current list is posted in the [http://goo.gl/5u9nbw OpenSSL Testing Tasks Google Spreadsheet].

(Would be nice to enable the [http://www.mediawiki.org/wiki/Extension:Widget MediaWiki Widget extension] and the [http://www.mediawikiwidgets.org/Google_Spreadsheet Google Spreadsheet widget].)

== Howtos and Style Guides ==
Documents that provide background on specific testing issues and idioms should be linked from here.

* [[How To Write Unit Tests For OpenSSL]]
* [http://mike-bland.com/2014/06/05/pseudo-xunit-pattern.html The Pseudo-xUnit Pattern]
* [[Testing And Development Tools and Tips]]

== Discussion Topics ==
These are discussions related to some of the current testing tasks.

=== First Area of Focus ===
Matt Caswell: One criterion might be ease of testing. If you were going down that route you might choose some of the low level packages within libcrypto which are relatively self-contained. I'm not talking about the crypto algorithms themselves (there are at least *some* tests for those), I'm thinking more about bio, pem, x509, x509v3, etc.

Another criterion might be where you get the most value. I suspect we will get the most value from a really good set of unit tests for libssl. I suspect this might be a big ask to start off with (you would probably have to mock out a lot of stuff).

=== Build Systems/Machines ===
Where can we get the machines? Is someone willing to donate them? What system should we use?

=== Get Unit Tests to Build on Windows ===
Steve Henson on building ssl/heartbeat_test.c on Windows: The first problem is that __func__ isn't understood. Changing that to __FUNCTION__ resolves that easily enough.

The second problem is more serious. Windows (and a couple of other platforms IIRC) is strict about only exporting approved symbols from global headers in shared libraries and wont export internal only functions.

That means you get linker errors:

heartbeat_test.obj : error LNK2019: unresolved external symbol _ssl3_setup_buffe
rs referenced in function _set_up
heartbeat_test.obj : error LNK2019: unresolved external symbol _ssl_init_wbio_bu
ffer referenced in function _set_up
heartbeat_test.obj : error LNK2019: unresolved external symbol _tls1_process_hea
rtbeat referenced in function _set_up_tls
heartbeat_test.obj : error LNK2019: unresolved external symbol _dtls1_process_he
artbeat referenced in function _set_up_dtls
out32dll\heartbeat_test.exe : fatal error LNK1120: 4 unresolved externals
NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual Studio 11.0
\VC\BIN\link.EXE"' : return code '0x460'

Potential solutions:
* http://stackoverflow.com/questions/7759777/is-it-possible-to-test-internal-class-from-a-c-dll-using-mstest
* http://stackoverflow.com/questions/19934714/googletest-c-unit-testing-linking-error
* http://stackoverflow.com/questions/14010627/unresolved-externals-when-compiling-unit-tests-for-visual-c-2012

ifdef'd out (for now): https://github.com/openssl/openssl/commit/7ce79a5bfdbcd53ae75f85e94eed665a05b5dea3

=== How to Manage Private Symbols ===
Steve Henson: As this also affects other platforms a general solution might be in order.

One option is to link to static libraries as has been mentioned. If you have a
lot of unit tests you could end up with a lot of large binaries. Would need some
changes to the dreaded Windows build system too.

A different option would be to export the unit test functions through a
structure pointer which a function returns. So you'd have something like..

unit_test = SSL_get_unit_test_funcs();

unit_test.some_internal_funcction(...);

Where the actual structure is opaque to applications (so they aren't tempted to
poke around in internals) but visible to the unit test code. Then the unit test
code doesn't call any internal functions not defined in there and doesn't need
to load private headers (unless it needs them for some definitions).

Mike Bland: This is an interesting idea, though I'd expect the unit tests will
still need access to internal headers.

But I have a question: Why do any of the symbols need to be private?
Is that degree of encapsulation necessary, and does it really
discourage irresponsible clients? The source code is open, so people
can always build their own copy and depend on internals if they really
want to, and the default UNIX/Linux/OS X build (i.e. ./configure &&
make && make test) doesn't lock down internal symbols. Have
dependencies on internal APIs ever been a problem in practice?

I'm not saying I feel strongly either way, but it seems that clients
should be clear that the public API is what's available in
openssl/include, and shouldn't depend on anything else. I just don't
understand what making symbols private buys anyone in the context of
OpenSSL, in light of how it appears to complicate the ability to unit
test.

Not trying to be argumentative, I just want to understand the
rationale. I'll work with whatever environment I have to.

Tom Francis: Making the symbols private allows the link editors on Windows and pretty much every UNIX system that doesn't use GNU ld to:
1) Run much, much faster at link time (not a minor consideration for most of the software I've worked on); and
2) optimize the final binary to load much faster.
Last time I checked, there was a minor improvement for GNU ld for #1, but it's trivial compared to Windows and HP's link editor, e.g. (I can't overstate the difference there). The runtime improvement is also dramatic on Windows, HP-UX, and AIX (we're talking tens for milliseconds for libraries that are much smaller than OpenSSL -- never tried it with OpenSSL, too lazy to mark everything, after having just done so for the library I was really interested in).

If you want to avoid a bunch of small binaries, you could combine all the unit tests into a single test application, and use multiple runs of it for each test, similar to how the openssl binary is created; if you link the object files directly (or a static library), you avoid the private symbols issue. Of course, this may be an issue for some of the embedded systems, where the loading a large binary for running the tests might not be acceptable (but might be preferable to a bunch of binaries?). Mike Bland was concerned about parallelization, and for any kind of automated tests on commits, that's a big point, if you can guarantee that each system only compiles/executes a limited number of tests (it could be useful for OS vendors, too). For the average OpenSSL user, that's not really a consideration; an average user is more likely concerned with how long (wall clock) it takes to compile and run the tests, and depending on the size of a single binary v the size of each smaller binary, it'll vary from system to system (generally, it'll be many times faster to create a single large binary than a few hundred smaller binaries, but the larger the binary, the longer it takes to load -- at some point, the load time might well outweigh the hundreds of smaller binaries (assuming you execute the large binary several times, rather than have a method for it to execute all tests in a single go). That point is probably > 100MB for larger systems, but for an embedded system, it could be much smaller.

For Steve's second suggestion, I think my biggest concerns would be how to logically separate out a unit test method without including the test framework (maybe throw it all in there?). I kinda like the idea of keeping the test inside the libraries, but I'm not sure why -- maybe 'cause I recently spent a few months back in a java world, where it's normal to throw all your unit tests into the same jar for production. I'm not sure I like that, though. The overhead of a single function per "module" added to the exports is also unlikely to drastically decrease link-time or run-time performance, too (compared with exporting everything).

=== Improved Patch/Cherry-pick Process ===
Geoff Thorpe: (a) patches that are not applied/cherry-picked to all the branches they apply to, or (b) applied to more branches than they should be. A complication of (a) is that some patches may need to be reworked in order to apply (or in order to not break binary compatibility of a stable branch). The difficulty of (b) is determining when the nature of the change either breaks binary compatibility or causes some other behavioral deviation that is not in keeping with the "promises" of stable branches.

http://ispras.linuxbase.org/index.php/ABI_compliance_checker
https://github.com/lvc/abi-compliance-checker/

Geoff on 2014-05-30: ...we might move to a scheme whereby we only maintain one most-recent stable branch for production releases, with snapshots or whatever of the development head. Ie. two branches, one for maximum-compatibility/bug-fixes-only, and the other for more active development. The number of "stable" branches would thus be 1 rather than 'n'. The idea is that we may defer to downstream "slow-moving" users (eg. long-term releases of linux distros) the opportunity to step up and do the work of maintaining (cherry-picking, back-porting, ...) older branches. Ie. what they typically already do in-house, but the opportunity to do it under our roof and thus combine their efforts with any other similarly-interested parties. If there's noone prepared to do that, the logical conclusion is that there is probably no need for it (or so the argument goes).

=== Improve/Standardize Algorithm Testing ===
Steve Henson: While the algorithms are partly tested I don't think they're anything like comprehensive enough. The way they are handled is also haphazard and spread across multiple tests covering all sorts of odd APIs, some of which are deprecated.

In fact I'd say they're (with the odd exception) an example of how *not* to do things.

Examples rc2test tests RC2 it has various test vectors hard coded inconsistent indentation and use of low level APIs. The rc4test is a similar story and uses a different format to rc2test. The same tale is repeated all over the place with ideatest, rmdtest etc etc.

Then we come to the code which I think is closest to how things should be which is evp_test. It uses the standard high level EVP API the format is consistent (if undocumented) and adding new tests can be done simply by editing a text file.

It currently however only works for some algorithms and the text file format could be better. It currently doesn't handle public key algorithms or test variants of the same algorithm (e.g. AES-NI versus non AES-NI) either.

=== Develop SSL/TLS Stack and Other Tools ===
Steve Henson: The heartbeat bug is an example where you can perform some testing without a complete SSL/TLS stack. However looking through the code in heartbeat_test.c it uses a lot of functionality which would normally be strictly off-limits for an application.

Some bugs are far more subtle and harder to test. For example some involve doing *almost* everything right and then slipping in some illegal parameter or badly formatted message. If you want to do that in a standalone bit of code you need a near complete SSL/TLS stack.

I can think of a couple of ways to handle that. One is to have some kind of hooks in libssl so testing code can misbehave and send out illegal messages in controlled ways (I had a prototype "bad heartbeat" API which only added a couple of lines to libssl). An alternative would be to write some kind of tiny SSL/TLS code used only by the test programs and carefully tailored to permit such things.

How To Write Unit Tests For OpenSSL

2014-06-08T19:21:01Z

Mbland: /* Building and Running the Test */

How To Write Unit Tests For OpenSSL

2014-06-07T18:49:57Z

Mbland: /* Use the Test Template Generator */

How To Write Unit Tests For OpenSSL

2014-06-07T17:45:48Z

Mbland: /* Style */

How To Write Unit Tests For OpenSSL

2014-06-07T17:25:41Z

Mbland: /* Building and Running the Test */

Unit Testing

2014-06-06T17:52:05Z

Mbland:

This page will track the effort to improve unit test and other automated test coverage for OpenSSL. Everyone is encouraged to help with this effort, but this page will track which community members have accepted specific responsibilities, and what tasks are currently underway.

== Discussion List ==
[https://groups.google.com/d/forum/openssl-testing openssl-testing] is the mailing list dedicated to the unit/automated testing effort, to avoid clogging openssl-dev. Membership is open to whoever wishes to join, even if only to lurk. Ideally all testing discussions will eventually move to openssl-dev once we have processes, tools, conventions, etc. in place.

== Team ==
We can define new roles as needed. The success of this effort will be inversely proportional to the number of roles with any one individual's name next to them. Each role can have more than one person filling it, but needs specific people to fill them.

The point is not to tell people what to do and not do: it’s to establish clear responsibilities so everyone can run with them right away and begin applying their creative energy to solutions, rather than waste it figuring out what the hell is going on, what they should do to contribute, and who they need to coordinate with to get things done.

'''Organizer''': [mailto:mbland@acm.org Mike Bland] 
Ensures the overall effort is making progress by facilitating communication, removing obstacles, and making hands-on contributions when necessary.

'''Tech Writer''': [mailto:mbland@acm.org Mike Bland], Sandeep Mankar, Chelsea Komlo, Graham Brooks 
Maintains documentation of tools, techniques, outstanding issues, etc. May maintain a code review/testing style guide based on the community consensus. Also documents who currently fills each role, as well as who might've filled it in the past.

'''Buildmaster''': [mailto:mbland@acm.org Mike Bland], Isaac Truett, Dileep Bapat, Graham Brooks 
Sets up and maintains automated builds and any infrastructure used to monitor them and report breakage.

'''Toolsmith''': Tony Aiuto
Maintains any tools and frameworks that are a part of the core build process; proposes, integrates, and/or retires existing tools.

'''Tutor''': [mailto:mbland@acm.org Mike Bland], Darshan Mody, Tom Francis, Brian N. Makin, Chelsea Komlo, Edward Knapp, Dileep Bapat, Graham Brooks 
Helps contributors, particularly new contributors, with the details of writing unit or other automated tests. May be assigned by the reviewer of a patch to help a contributor get a new change under test. Submission of the patch should require the approval of the tutor in most cases. (Should ideally be more than one person.)

== Goals ==
We can add more concrete goals to this list after we make progress on these. Getting people to own roles, setting up build infrastructure, and setting and enforcing policy are important first steps. Upon that foundation, we can begin to make serious progress towards improved test coverage, code quality, and defect prevention.
* Establish a written contribution/code review policy that requires:
** tests for every new code change, except for very trivial changes that do not directly impact program logic (e.g. documentation fixes, string constant updates, build script tweaks); this can be limited to an agreed-upon subset of the code to start, but would hopefully be expanded to cover most, if not all, of the code base
** tests for every bugfix
** smaller, well-tested changes building up to a complete feature (as opposed to large and/or untested changes that implement a large, complex feature all at once)
* Establish a schedule of Build Cops and Tutors to begin enforcing rollbacks/fixes and to help shepherd new contributors through the testing process.
* Set up a series of publicly-accessible continuous integration servers using whatever system the community prefers (e.g. Jenkins, Buildbot, etc.). These can be dedicated hardware or possibly Amazon Web Services-based. Establish a written policy that these remain green at all times, and that breakages be rolled-back or fixed within no more than 30 minutes.

== Tasks ==
The current list is posted in the [http://goo.gl/5u9nbw OpenSSL Testing Tasks Google Spreadsheet].

(Would be nice to enable the [http://www.mediawiki.org/wiki/Extension:Widget MediaWiki Widget extension] and the [http://www.mediawikiwidgets.org/Google_Spreadsheet Google Spreadsheet widget].)

== Howtos and Style Guides ==
Documents that provide background on specific testing issues and idioms should be linked from here.

* [[How To Write Unit Tests For OpenSSL]]
* [http://mike-bland.com/2014/06/05/pseudo-xunit-pattern.html The Pseudo-xUnit Pattern]

== Discussion Topics ==
These are discussions related to some of the current testing tasks.

=== First Area of Focus ===
Matt Caswell: One criterion might be ease of testing. If you were going down that route you might choose some of the low level packages within libcrypto which are relatively self-contained. I'm not talking about the crypto algorithms themselves (there are at least *some* tests for those), I'm thinking more about bio, pem, x509, x509v3, etc.

Another criterion might be where you get the most value. I suspect we will get the most value from a really good set of unit tests for libssl. I suspect this might be a big ask to start off with (you would probably have to mock out a lot of stuff).

=== Build Systems/Machines ===
Where can we get the machines? Is someone willing to donate them? What system should we use?

=== Get Unit Tests to Build on Windows ===
Steve Henson on building ssl/heartbeat_test.c on Windows: The first problem is that __func__ isn't understood. Changing that to __FUNCTION__ resolves that easily enough.

The second problem is more serious. Windows (and a couple of other platforms IIRC) is strict about only exporting approved symbols from global headers in shared libraries and wont export internal only functions.

That means you get linker errors:

heartbeat_test.obj : error LNK2019: unresolved external symbol _ssl3_setup_buffe
rs referenced in function _set_up
heartbeat_test.obj : error LNK2019: unresolved external symbol _ssl_init_wbio_bu
ffer referenced in function _set_up
heartbeat_test.obj : error LNK2019: unresolved external symbol _tls1_process_hea
rtbeat referenced in function _set_up_tls
heartbeat_test.obj : error LNK2019: unresolved external symbol _dtls1_process_he
artbeat referenced in function _set_up_dtls
out32dll\heartbeat_test.exe : fatal error LNK1120: 4 unresolved externals
NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual Studio 11.0
\VC\BIN\link.EXE"' : return code '0x460'

Potential solutions:
* http://stackoverflow.com/questions/7759777/is-it-possible-to-test-internal-class-from-a-c-dll-using-mstest
* http://stackoverflow.com/questions/19934714/googletest-c-unit-testing-linking-error
* http://stackoverflow.com/questions/14010627/unresolved-externals-when-compiling-unit-tests-for-visual-c-2012

ifdef'd out (for now): https://github.com/openssl/openssl/commit/7ce79a5bfdbcd53ae75f85e94eed665a05b5dea3

=== How to Manage Private Symbols ===
Steve Henson: As this also affects other platforms a general solution might be in order.

One option is to link to static libraries as has been mentioned. If you have a
lot of unit tests you could end up with a lot of large binaries. Would need some
changes to the dreaded Windows build system too.

A different option would be to export the unit test functions through a
structure pointer which a function returns. So you'd have something like..

unit_test = SSL_get_unit_test_funcs();

unit_test.some_internal_funcction(...);

Where the actual structure is opaque to applications (so they aren't tempted to
poke around in internals) but visible to the unit test code. Then the unit test
code doesn't call any internal functions not defined in there and doesn't need
to load private headers (unless it needs them for some definitions).

Mike Bland: This is an interesting idea, though I'd expect the unit tests will
still need access to internal headers.

But I have a question: Why do any of the symbols need to be private?
Is that degree of encapsulation necessary, and does it really
discourage irresponsible clients? The source code is open, so people
can always build their own copy and depend on internals if they really
want to, and the default UNIX/Linux/OS X build (i.e. ./configure &&
make && make test) doesn't lock down internal symbols. Have
dependencies on internal APIs ever been a problem in practice?

I'm not saying I feel strongly either way, but it seems that clients
should be clear that the public API is what's available in
openssl/include, and shouldn't depend on anything else. I just don't
understand what making symbols private buys anyone in the context of
OpenSSL, in light of how it appears to complicate the ability to unit
test.

Not trying to be argumentative, I just want to understand the
rationale. I'll work with whatever environment I have to.

=== Improved Patch/Cherry-pick Process ===
Geoff Thorpe: (a) patches that are not applied/cherry-picked to all the branches they apply to, or (b) applied to more branches than they should be. A complication of (a) is that some patches may need to be reworked in order to apply (or in order to not break binary compatibility of a stable branch). The difficulty of (b) is determining when the nature of the change either breaks binary compatibility or causes some other behavioral deviation that is not in keeping with the "promises" of stable branches.

http://ispras.linuxbase.org/index.php/ABI_compliance_checker
https://github.com/lvc/abi-compliance-checker/

Geoff on 2014-05-30: ...we might move to a scheme whereby we only maintain one most-recent stable branch for production releases, with snapshots or whatever of the development head. Ie. two branches, one for maximum-compatibility/bug-fixes-only, and the other for more active development. The number of "stable" branches would thus be 1 rather than 'n'. The idea is that we may defer to downstream "slow-moving" users (eg. long-term releases of linux distros) the opportunity to step up and do the work of maintaining (cherry-picking, back-porting, ...) older branches. Ie. what they typically already do in-house, but the opportunity to do it under our roof and thus combine their efforts with any other similarly-interested parties. If there's noone prepared to do that, the logical conclusion is that there is probably no need for it (or so the argument goes).

=== Improve/Standardize Algorithm Testing ===
Steve Henson: While the algorithms are partly tested I don't think they're anything like comprehensive enough. The way they are handled is also haphazard and spread across multiple tests covering all sorts of odd APIs, some of which are deprecated.

In fact I'd say they're (with the odd exception) an example of how *not* to do things.

Examples rc2test tests RC2 it has various test vectors hard coded inconsistent indentation and use of low level APIs. The rc4test is a similar story and uses a different format to rc2test. The same tale is repeated all over the place with ideatest, rmdtest etc etc.

Then we come to the code which I think is closest to how things should be which is evp_test. It uses the standard high level EVP API the format is consistent (if undocumented) and adding new tests can be done simply by editing a text file.

It currently however only works for some algorithms and the text file format could be better. It currently doesn't handle public key algorithms or test variants of the same algorithm (e.g. AES-NI versus non AES-NI) either.

=== Develop SSL/TLS Stack and Other Tools ===
Steve Henson: The heartbeat bug is an example where you can perform some testing without a complete SSL/TLS stack. However looking through the code in heartbeat_test.c it uses a lot of functionality which would normally be strictly off-limits for an application.

Some bugs are far more subtle and harder to test. For example some involve doing *almost* everything right and then slipping in some illegal parameter or badly formatted message. If you want to do that in a standalone bit of code you need a near complete SSL/TLS stack.

I can think of a couple of ways to handle that. One is to have some kind of hooks in libssl so testing code can misbehave and send out illegal messages in controlled ways (I had a prototype "bad heartbeat" API which only added a couple of lines to libssl). An alternative would be to write some kind of tiny SSL/TLS code used only by the test programs and carefully tailored to permit such things.

Unit Testing

2014-06-06T17:39:53Z

Mbland:

This page will track the effort to improve unit test and other automated test coverage for OpenSSL. Everyone is encouraged to help with this effort, but this page will track which community members have accepted specific responsibilities, and what tasks are currently underway.

== Discussion List ==
[https://groups.google.com/forum/#!forum/openssl-testing openssl-testing] is the mailing list dedicated to the unit/automated testing effort, to avoid clogging openssl-dev. Membership is open to whoever wishes to join, even if only to lurk. Ideally all testing discussions will eventually move to openssl-dev once we have processes, tools, conventions, etc. in place.

== Team ==
We can define new roles as needed. The success of this effort will be inversely proportional to the number of roles with any one individual's name next to them. Each role can have more than one person filling it, but needs specific people to fill them.

The point is not to tell people what to do and not do: it’s to establish clear responsibilities so everyone can run with them right away and begin applying their creative energy to solutions, rather than waste it figuring out what the hell is going on, what they should do to contribute, and who they need to coordinate with to get things done.

'''Organizer''': [mailto:mbland@acm.org Mike Bland] 
Ensures the overall effort is making progress by facilitating communication, removing obstacles, and making hands-on contributions when necessary.

'''Tech Writer''': [mailto:mbland@acm.org Mike Bland], Sandeep Mankar, Chelsea Komlo, Graham Brooks 
Maintains documentation of tools, techniques, outstanding issues, etc. May maintain a code review/testing style guide based on the community consensus. Also documents who currently fills each role, as well as who might've filled it in the past.

'''Buildmaster''': [mailto:mbland@acm.org Mike Bland], Isaac Truett, Dileep Bapat, Graham Brooks 
Sets up and maintains automated builds and any infrastructure used to monitor them and report breakage.

'''Toolsmith''': Tony Aiuto
Maintains any tools and frameworks that are a part of the core build process; proposes, integrates, and/or retires existing tools.

'''Tutor''': [mailto:mbland@acm.org Mike Bland], Darshan Mody, Tom Francis, Brian N. Makin, Chelsea Komlo, Edward Knapp, Dileep Bapat, Graham Brooks 
Helps contributors, particularly new contributors, with the details of writing unit or other automated tests. May be assigned by the reviewer of a patch to help a contributor get a new change under test. Submission of the patch should require the approval of the tutor in most cases. (Should ideally be more than one person.)

== Goals ==
We can add more concrete goals to this list after we make progress on these. Getting people to own roles, setting up build infrastructure, and setting and enforcing policy are important first steps. Upon that foundation, we can begin to make serious progress towards improved test coverage, code quality, and defect prevention.
* Establish a written contribution/code review policy that requires:
** tests for every new code change, except for very trivial changes that do not directly impact program logic (e.g. documentation fixes, string constant updates, build script tweaks); this can be limited to an agreed-upon subset of the code to start, but would hopefully be expanded to cover most, if not all, of the code base
** tests for every bugfix
** smaller, well-tested changes building up to a complete feature (as opposed to large and/or untested changes that implement a large, complex feature all at once)
* Establish a schedule of Build Cops and Tutors to begin enforcing rollbacks/fixes and to help shepherd new contributors through the testing process.
* Set up a series of publicly-accessible continuous integration servers using whatever system the community prefers (e.g. Jenkins, Buildbot, etc.). These can be dedicated hardware or possibly Amazon Web Services-based. Establish a written policy that these remain green at all times, and that breakages be rolled-back or fixed within no more than 30 minutes.

== Tasks ==
The current list is posted in the [http://goo.gl/5u9nbw OpenSSL Testing Tasks Google Spreadsheet].

(Would be nice to enable the [http://www.mediawiki.org/wiki/Extension:Widget MediaWiki Widget extension] and the [http://www.mediawikiwidgets.org/Google_Spreadsheet Google Spreadsheet widget].)

== Howtos and Style Guides ==
Documents that provide background on specific testing issues and idioms should be linked from here.

* [[How To Write Unit Tests For OpenSSL]]
* [http://mike-bland.com/2014/06/05/pseudo-xunit-pattern.html The Pseudo-xUnit Pattern]

== Discussion Topics ==
These are discussions related to some of the current testing tasks.

=== First Area of Focus ===
Matt Caswell: One criterion might be ease of testing. If you were going down that route you might choose some of the low level packages within libcrypto which are relatively self-contained. I'm not talking about the crypto algorithms themselves (there are at least *some* tests for those), I'm thinking more about bio, pem, x509, x509v3, etc.

Another criterion might be where you get the most value. I suspect we will get the most value from a really good set of unit tests for libssl. I suspect this might be a big ask to start off with (you would probably have to mock out a lot of stuff).

=== Build Systems/Machines ===
Where can we get the machines? Is someone willing to donate them? What system should we use?

=== Get Unit Tests to Build on Windows ===
Steve Henson on building ssl/heartbeat_test.c on Windows: The first problem is that __func__ isn't understood. Changing that to __FUNCTION__ resolves that easily enough.

The second problem is more serious. Windows (and a couple of other platforms IIRC) is strict about only exporting approved symbols from global headers in shared libraries and wont export internal only functions.

That means you get linker errors:

heartbeat_test.obj : error LNK2019: unresolved external symbol _ssl3_setup_buffe
rs referenced in function _set_up
heartbeat_test.obj : error LNK2019: unresolved external symbol _ssl_init_wbio_bu
ffer referenced in function _set_up
heartbeat_test.obj : error LNK2019: unresolved external symbol _tls1_process_hea
rtbeat referenced in function _set_up_tls
heartbeat_test.obj : error LNK2019: unresolved external symbol _dtls1_process_he
artbeat referenced in function _set_up_dtls
out32dll\heartbeat_test.exe : fatal error LNK1120: 4 unresolved externals
NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual Studio 11.0
\VC\BIN\link.EXE"' : return code '0x460'

Potential solutions:
* http://stackoverflow.com/questions/7759777/is-it-possible-to-test-internal-class-from-a-c-dll-using-mstest
* http://stackoverflow.com/questions/19934714/googletest-c-unit-testing-linking-error
* http://stackoverflow.com/questions/14010627/unresolved-externals-when-compiling-unit-tests-for-visual-c-2012

ifdef'd out (for now): https://github.com/openssl/openssl/commit/7ce79a5bfdbcd53ae75f85e94eed665a05b5dea3

=== How to Manage Private Symbols ===
Steve Henson: As this also affects other platforms a general solution might be in order.

One option is to link to static libraries as has been mentioned. If you have a
lot of unit tests you could end up with a lot of large binaries. Would need some
changes to the dreaded Windows build system too.

A different option would be to export the unit test functions through a
structure pointer which a function returns. So you'd have something like..

unit_test = SSL_get_unit_test_funcs();

unit_test.some_internal_funcction(...);

Where the actual structure is opaque to applications (so they aren't tempted to
poke around in internals) but visible to the unit test code. Then the unit test
code doesn't call any internal functions not defined in there and doesn't need
to load private headers (unless it needs them for some definitions).

Mike Bland: This is an interesting idea, though I'd expect the unit tests will
still need access to internal headers.

But I have a question: Why do any of the symbols need to be private?
Is that degree of encapsulation necessary, and does it really
discourage irresponsible clients? The source code is open, so people
can always build their own copy and depend on internals if they really
want to, and the default UNIX/Linux/OS X build (i.e. ./configure &&
make && make test) doesn't lock down internal symbols. Have
dependencies on internal APIs ever been a problem in practice?

I'm not saying I feel strongly either way, but it seems that clients
should be clear that the public API is what's available in
openssl/include, and shouldn't depend on anything else. I just don't
understand what making symbols private buys anyone in the context of
OpenSSL, in light of how it appears to complicate the ability to unit
test.

Not trying to be argumentative, I just want to understand the
rationale. I'll work with whatever environment I have to.

=== Improved Patch/Cherry-pick Process ===
Geoff Thorpe: (a) patches that are not applied/cherry-picked to all the branches they apply to, or (b) applied to more branches than they should be. A complication of (a) is that some patches may need to be reworked in order to apply (or in order to not break binary compatibility of a stable branch). The difficulty of (b) is determining when the nature of the change either breaks binary compatibility or causes some other behavioral deviation that is not in keeping with the "promises" of stable branches.

http://ispras.linuxbase.org/index.php/ABI_compliance_checker
https://github.com/lvc/abi-compliance-checker/

Geoff on 2014-05-30: ...we might move to a scheme whereby we only maintain one most-recent stable branch for production releases, with snapshots or whatever of the development head. Ie. two branches, one for maximum-compatibility/bug-fixes-only, and the other for more active development. The number of "stable" branches would thus be 1 rather than 'n'. The idea is that we may defer to downstream "slow-moving" users (eg. long-term releases of linux distros) the opportunity to step up and do the work of maintaining (cherry-picking, back-porting, ...) older branches. Ie. what they typically already do in-house, but the opportunity to do it under our roof and thus combine their efforts with any other similarly-interested parties. If there's noone prepared to do that, the logical conclusion is that there is probably no need for it (or so the argument goes).

=== Improve/Standardize Algorithm Testing ===
Steve Henson: While the algorithms are partly tested I don't think they're anything like comprehensive enough. The way they are handled is also haphazard and spread across multiple tests covering all sorts of odd APIs, some of which are deprecated.

In fact I'd say they're (with the odd exception) an example of how *not* to do things.

Examples rc2test tests RC2 it has various test vectors hard coded inconsistent indentation and use of low level APIs. The rc4test is a similar story and uses a different format to rc2test. The same tale is repeated all over the place with ideatest, rmdtest etc etc.

Then we come to the code which I think is closest to how things should be which is evp_test. It uses the standard high level EVP API the format is consistent (if undocumented) and adding new tests can be done simply by editing a text file.

It currently however only works for some algorithms and the text file format could be better. It currently doesn't handle public key algorithms or test variants of the same algorithm (e.g. AES-NI versus non AES-NI) either.

=== Develop SSL/TLS Stack and Other Tools ===
Steve Henson: The heartbeat bug is an example where you can perform some testing without a complete SSL/TLS stack. However looking through the code in heartbeat_test.c it uses a lot of functionality which would normally be strictly off-limits for an application.

Some bugs are far more subtle and harder to test. For example some involve doing *almost* everything right and then slipping in some illegal parameter or badly formatted message. If you want to do that in a standalone bit of code you need a near complete SSL/TLS stack.

I can think of a couple of ways to handle that. One is to have some kind of hooks in libssl so testing code can misbehave and send out illegal messages in controlled ways (I had a prototype "bad heartbeat" API which only added a couple of lines to libssl). An alternative would be to write some kind of tiny SSL/TLS code used only by the test programs and carefully tailored to permit such things.

How To Write Unit Tests For OpenSSL

2014-06-06T16:13:19Z

Mbland:

How To Write Unit Tests For OpenSSL

2014-06-06T16:02:05Z

Mbland:

Unit Testing

2014-06-06T15:57:30Z

Mbland:

This page will track the effort to improve unit test and other automated test coverage for OpenSSL. Everyone is encouraged to help with this effort, but this page will track which community members have accepted specific responsibilities, and what tasks are currently underway.

== Team ==
We can define new roles as needed. The success of this effort will be inversely proportional to the number of roles with any one individual's name next to them. Each role can have more than one person filling it, but needs specific people to fill them.

The point is not to tell people what to do and not do: it’s to establish clear responsibilities so everyone can run with them right away and begin applying their creative energy to solutions, rather than waste it figuring out what the hell is going on, what they should do to contribute, and who they need to coordinate with to get things done.

'''Organizer''': [mailto:mbland@acm.org Mike Bland] 
Ensures the overall effort is making progress by facilitating communication, removing obstacles, and making hands-on contributions when necessary.

'''Tech Writer''': [mailto:mbland@acm.org Mike Bland], Sandeep Mankar, Chelsea Komlo, Graham Brooks 
Maintains documentation of tools, techniques, outstanding issues, etc. May maintain a code review/testing style guide based on the community consensus. Also documents who currently fills each role, as well as who might've filled it in the past.

'''Buildmaster''': [mailto:mbland@acm.org Mike Bland], Isaac Truett, Dileep Bapat, Graham Brooks 
Sets up and maintains automated builds and any infrastructure used to monitor them and report breakage.

'''Toolsmith''': Tony Aiuto
Maintains any tools and frameworks that are a part of the core build process; proposes, integrates, and/or retires existing tools.

'''Tutor''': [mailto:mbland@acm.org Mike Bland], Darshan Mody, Tom Francis, Brian N. Makin, Chelsea Komlo, Edward Knapp, Dileep Bapat, Graham Brooks 
Helps contributors, particularly new contributors, with the details of writing unit or other automated tests. May be assigned by the reviewer of a patch to help a contributor get a new change under test. Submission of the patch should require the approval of the tutor in most cases. (Should ideally be more than one person.)

== Goals ==
We can add more concrete goals to this list after we make progress on these. Getting people to own roles, setting up build infrastructure, and setting and enforcing policy are important first steps. Upon that foundation, we can begin to make serious progress towards improved test coverage, code quality, and defect prevention.
* Establish a written contribution/code review policy that requires:
** tests for every new code change, except for very trivial changes that do not directly impact program logic (e.g. documentation fixes, string constant updates, build script tweaks); this can be limited to an agreed-upon subset of the code to start, but would hopefully be expanded to cover most, if not all, of the code base
** tests for every bugfix
** smaller, well-tested changes building up to a complete feature (as opposed to large and/or untested changes that implement a large, complex feature all at once)
* Establish a schedule of Build Cops and Tutors to begin enforcing rollbacks/fixes and to help shepherd new contributors through the testing process.
* Set up a series of publicly-accessible continuous integration servers using whatever system the community prefers (e.g. Jenkins, Buildbot, etc.). These can be dedicated hardware or possibly Amazon Web Services-based. Establish a written policy that these remain green at all times, and that breakages be rolled-back or fixed within no more than 30 minutes.

== Tasks ==
The current list is posted in the [http://goo.gl/5u9nbw OpenSSL Testing Tasks Google Spreadsheet].

(Would be nice to enable the [http://www.mediawiki.org/wiki/Extension:Widget MediaWiki Widget extension] and the [http://www.mediawikiwidgets.org/Google_Spreadsheet Google Spreadsheet widget].)

== Howtos and Style Guides ==
Documents that provide background on specific testing issues and idioms should be linked from here.

* [[How To Write Unit Tests For OpenSSL]]
* [http://mike-bland.com/2014/06/05/pseudo-xunit-pattern.html The Pseudo-xUnit Pattern]

== Discussion Topics ==
These are discussions related to some of the current testing tasks.

=== First Area of Focus ===
Matt Caswell: One criterion might be ease of testing. If you were going down that route you might choose some of the low level packages within libcrypto which are relatively self-contained. I'm not talking about the crypto algorithms themselves (there are at least *some* tests for those), I'm thinking more about bio, pem, x509, x509v3, etc.

Another criterion might be where you get the most value. I suspect we will get the most value from a really good set of unit tests for libssl. I suspect this might be a big ask to start off with (you would probably have to mock out a lot of stuff).

=== Build Systems/Machines ===
Where can we get the machines? Is someone willing to donate them? What system should we use?

=== Get Unit Tests to Build on Windows ===
Steve Henson on building ssl/heartbeat_test.c on Windows: The first problem is that __func__ isn't understood. Changing that to __FUNCTION__ resolves that easily enough.

The second problem is more serious. Windows (and a couple of other platforms IIRC) is strict about only exporting approved symbols from global headers in shared libraries and wont export internal only functions.

That means you get linker errors:

heartbeat_test.obj : error LNK2019: unresolved external symbol _ssl3_setup_buffe
rs referenced in function _set_up
heartbeat_test.obj : error LNK2019: unresolved external symbol _ssl_init_wbio_bu
ffer referenced in function _set_up
heartbeat_test.obj : error LNK2019: unresolved external symbol _tls1_process_hea
rtbeat referenced in function _set_up_tls
heartbeat_test.obj : error LNK2019: unresolved external symbol _dtls1_process_he
artbeat referenced in function _set_up_dtls
out32dll\heartbeat_test.exe : fatal error LNK1120: 4 unresolved externals
NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual Studio 11.0
\VC\BIN\link.EXE"' : return code '0x460'

Potential solutions:
* http://stackoverflow.com/questions/7759777/is-it-possible-to-test-internal-class-from-a-c-dll-using-mstest
* http://stackoverflow.com/questions/19934714/googletest-c-unit-testing-linking-error
* http://stackoverflow.com/questions/14010627/unresolved-externals-when-compiling-unit-tests-for-visual-c-2012

ifdef'd out (for now): https://github.com/openssl/openssl/commit/7ce79a5bfdbcd53ae75f85e94eed665a05b5dea3

=== How to Manage Private Symbols ===
Steve Henson: As this also affects other platforms a general solution might be in order.

One option is to link to static libraries as has been mentioned. If you have a
lot of unit tests you could end up with a lot of large binaries. Would need some
changes to the dreaded Windows build system too.

A different option would be to export the unit test functions through a
structure pointer which a function returns. So you'd have something like..

unit_test = SSL_get_unit_test_funcs();

unit_test.some_internal_funcction(...);

Where the actual structure is opaque to applications (so they aren't tempted to
poke around in internals) but visible to the unit test code. Then the unit test
code doesn't call any internal functions not defined in there and doesn't need
to load private headers (unless it needs them for some definitions).

Mike Bland: This is an interesting idea, though I'd expect the unit tests will
still need access to internal headers.

But I have a question: Why do any of the symbols need to be private?
Is that degree of encapsulation necessary, and does it really
discourage irresponsible clients? The source code is open, so people
can always build their own copy and depend on internals if they really
want to, and the default UNIX/Linux/OS X build (i.e. ./configure &&
make && make test) doesn't lock down internal symbols. Have
dependencies on internal APIs ever been a problem in practice?

I'm not saying I feel strongly either way, but it seems that clients
should be clear that the public API is what's available in
openssl/include, and shouldn't depend on anything else. I just don't
understand what making symbols private buys anyone in the context of
OpenSSL, in light of how it appears to complicate the ability to unit
test.

Not trying to be argumentative, I just want to understand the
rationale. I'll work with whatever environment I have to.

=== Improved Patch/Cherry-pick Process ===
Geoff Thorpe: (a) patches that are not applied/cherry-picked to all the branches they apply to, or (b) applied to more branches than they should be. A complication of (a) is that some patches may need to be reworked in order to apply (or in order to not break binary compatibility of a stable branch). The difficulty of (b) is determining when the nature of the change either breaks binary compatibility or causes some other behavioral deviation that is not in keeping with the "promises" of stable branches.

http://ispras.linuxbase.org/index.php/ABI_compliance_checker
https://github.com/lvc/abi-compliance-checker/

Geoff on 2014-05-30: ...we might move to a scheme whereby we only maintain one most-recent stable branch for production releases, with snapshots or whatever of the development head. Ie. two branches, one for maximum-compatibility/bug-fixes-only, and the other for more active development. The number of "stable" branches would thus be 1 rather than 'n'. The idea is that we may defer to downstream "slow-moving" users (eg. long-term releases of linux distros) the opportunity to step up and do the work of maintaining (cherry-picking, back-porting, ...) older branches. Ie. what they typically already do in-house, but the opportunity to do it under our roof and thus combine their efforts with any other similarly-interested parties. If there's noone prepared to do that, the logical conclusion is that there is probably no need for it (or so the argument goes).

=== Improve/Standardize Algorithm Testing ===
Steve Henson: While the algorithms are partly tested I don't think they're anything like comprehensive enough. The way they are handled is also haphazard and spread across multiple tests covering all sorts of odd APIs, some of which are deprecated.

In fact I'd say they're (with the odd exception) an example of how *not* to do things.

Examples rc2test tests RC2 it has various test vectors hard coded inconsistent indentation and use of low level APIs. The rc4test is a similar story and uses a different format to rc2test. The same tale is repeated all over the place with ideatest, rmdtest etc etc.

Then we come to the code which I think is closest to how things should be which is evp_test. It uses the standard high level EVP API the format is consistent (if undocumented) and adding new tests can be done simply by editing a text file.

It currently however only works for some algorithms and the text file format could be better. It currently doesn't handle public key algorithms or test variants of the same algorithm (e.g. AES-NI versus non AES-NI) either.

=== Develop SSL/TLS Stack and Other Tools ===
Steve Henson: The heartbeat bug is an example where you can perform some testing without a complete SSL/TLS stack. However looking through the code in heartbeat_test.c it uses a lot of functionality which would normally be strictly off-limits for an application.

Some bugs are far more subtle and harder to test. For example some involve doing *almost* everything right and then slipping in some illegal parameter or badly formatted message. If you want to do that in a standalone bit of code you need a near complete SSL/TLS stack.

I can think of a couple of ways to handle that. One is to have some kind of hooks in libssl so testing code can misbehave and send out illegal messages in controlled ways (I had a prototype "bad heartbeat" API which only added a couple of lines to libssl). An alternative would be to write some kind of tiny SSL/TLS code used only by the test programs and carefully tailored to permit such things.

Unit Testing

2014-06-06T15:50:46Z

Mbland:

How To Write Unit Tests For OpenSSL

2014-06-06T15:50:35Z

Mbland:

How To Write Unit Tests For OpenSSL

2014-06-06T15:49:22Z

Mbland:

How To Write Unit Tests For OpenSSL

2014-06-06T15:21:46Z

Mbland:

How To Write Unit Tests For OpenSSL

2014-06-06T15:10:51Z

Mbland: