Difference between revisions of "Self Test Failures"

From OpenSSLWiki
Jump to navigationJump to search
m (Add info on OpenSSL 1.1.x)
 
(11 intermediate revisions by the same user not shown)
Line 1: Line 1:
 
The OpenSSL library includes a test suite that exercises components from both <tt>libcrypto.a</tt> and <tt>libssl.a</tt>. Ensuring the library builds and executes its tests properly is a keystone to using the library.
 
The OpenSSL library includes a test suite that exercises components from both <tt>libcrypto.a</tt> and <tt>libssl.a</tt>. Ensuring the library builds and executes its tests properly is a keystone to using the library.
  
This page will show you how to isolate the test and the cause under many conditions. The more information you can supply with a bug report or pull request, the easier it is for the team to identify the failure or accept proposed changes.
+
This page will show you how to isolate the test and the cause under many conditions. The more information you can supply with a bug report or pull request, the easier it is for the team to identify the failure or accept proposed changes. Also see [http://www.openssl.org/docs/faq.html#BUILD17 Item 17. I think I've found a bug, what should I do?] in the OpenSSL FAQ.
  
 
Generally speaking, there are two generations of self tests. The first generation is from OpenSSL 1.0.2 and below. The second generation is from OpenSSL 1.1.0 and above. This page focuses on the second generation, or OpenSSL 1.1.0 and above.
 
Generally speaking, there are two generations of self tests. The first generation is from OpenSSL 1.0.2 and below. The second generation is from OpenSSL 1.1.0 and above. This page focuses on the second generation, or OpenSSL 1.1.0 and above.
  
==Debug Build==
+
You can sometimes isolate a bug with <tt>no-asm</tt>, and sometimes with <tt>-O0</tt> or <tt>-O1</tt>. Ideally the bug will be present with no optimizations (<tt>-O0</tt>), but sometimes it takes some compiler analysis to tickle it (<tt>-O1</tt>).
 +
 
 +
== OpenSSL 1.1.0 ==
 +
 
 +
===Build the Library===
 +
 
 +
The first thing you should do after verifying the bug is to create a debug build of the OpenSSL library and attempt to reproduce it.
 +
 
 +
TODO: provide instrcutions.
 +
 
 +
===Build the Test===
 +
 
 +
You typically run <tt>make test</tt> to validate the library after <tt>make</tt>'ing.
 +
 
 +
<pre>$ make test
 +
...
 +
 
 +
make[1]: Entering directory '/home/test_user/openssl-1.1.1d'
 +
make[1]: Leaving directory '/home/test_user/openssl-1.1.1d'
 +
make[1]: Entering directory '/home/test_user/openssl-1.1.1d'
 +
( cd test; \
 +
  mkdir -p test-runs; \
 +
  SRCTOP=../. \
 +
  BLDTOP=../. \
 +
  RESULT_D=test-runs \
 +
  PERL="/usr/bin/perl" \
 +
  EXE_EXT= \
 +
  OPENSSL_ENGINES=`cd .././engines 2>/dev/null && pwd` \
 +
  OPENSSL_DEBUG_MEMORY=on \
 +
    /usr/bin/perl .././test/run_tests.pl  )
 +
../test/recipes/01-test_abort.t .................... ok
 +
../test/recipes/01-test_sanity.t ................... ok
 +
...
 +
../test/recipes/15-test_dh.t ....................... ok
 +
../test/recipes/15-test_dsa.t ...................... ok
 +
../test/recipes/15-test_ec.t ....................... Dubious, test returned 1 (wstat 256, 0x100)
 +
Failed 1/5 subtests
 +
../test/recipes/15-test_ecdsa.t .................... Dubious, test returned 1 (wstat 256, 0x100)
 +
Failed 1/1 subtests
 +
../test/recipes/15-test_ecparam.t .................. Dubious, test returned 2 (wstat 512, 0x200)
 +
Failed 2/134 subtests
 +
../test/recipes/15-test_genrsa.t ................... ok
 +
../test/recipes/15-test_mp_rsa.t ................... ok 
 +
../test/recipes/15-test_out_option.t ............... ok
 +
../test/recipes/15-test_rsa.t ...................... ok
 +
../test/recipes/15-test_rsapss.t ................... ok
 +
...</pre>
 +
 
 +
=== Debug the Test ===
 +
 
 +
You can run a specific test by setting <tt>PERL5LIB</tt> and <tt>TOP</tt> variables as shown below.
 +
 
 +
<pre># From the root of the OpenSSL directory
 +
$ PERL5LIB=util/perl TOP=$PWD perl test/recipes/15-test_ec.t
 +
1..5
 +
ok 1 - require '/home/test_user/openssl-1.1.1d/test/recipes/tconversion.pl';
 +
    # Subtest: /home/test_user/openssl-1.1.1d/test/ectest
 +
    1..11
 +
crypto/ec/ecp_nistp521.c:144:15: runtime error: load of misaligned address 0x7ffec57b10b7 for type 'limb', which requires 8 byte alignment
 +
0x7ffec57b10b7: note: pointer points here
 +
c2 31 7e 7e f9  9b 42 6a 85 c1 b3 48 33  de a8 ff a2 27 c1 1d fe  28 59 e7 ef 77 5e 4b a1  ba 3d 4d
 +
            ^
 +
/home/test_user/openssl-1.1.1d/util/shlib_wrap.sh /home/test_user/openssl-1.1.1d/test/ectest => 1
 +
not ok 2 - running ectest
 +
#  Failed test 'running ectest'
 +
#  at test/recipes/15-test_ec.t line 23.
 +
# Subtest: ec conversions -- private key
 +
    1..10
 +
    ok 1 - initializing
 +
...</pre>
 +
 
 +
=== Debug an Engine ===
 +
 
 +
TODO: provide instructions
 +
 
 +
== OpenSSL 1.0.2 ==
 +
 
 +
The instructions that follow help determine a failue in OpenSSL 1.0.2 and below. As of January 2020, OpenSSL 1.0.2 is end of life. You should move to OpenSSL 1.1.x as soon as possible.
 +
 
 +
===Build the Library===
  
 
The first thing you should do after verifying the bug is to create a debug build of the OpenSSL library and attempt to reproduce it. Creating a debug build is as easy as using <tt>-d</tt> configure argument. In the output below, notice <tt>-d</tt> caused the library to reduce optimizations (<tt>-O0</tt>) and increase debugging information (<tt>-g</tt>). However, the debugging information will lack symbolic constants because <tt>-g3</tt> was not used:
 
The first thing you should do after verifying the bug is to create a debug build of the OpenSSL library and attempt to reproduce it. Creating a debug build is as easy as using <tt>-d</tt> configure argument. In the output below, notice <tt>-d</tt> caused the library to reduce optimizations (<tt>-O0</tt>) and increase debugging information (<tt>-g</tt>). However, the debugging information will lack symbolic constants because <tt>-g3</tt> was not used:
  
<pre>./config -d
+
<pre>$ ./config -d
 
Operating system: i86pc-whatever-solaris2
 
Operating system: i86pc-whatever-solaris2
 
Configuring for solaris64-x86_64-gcc
 
Configuring for solaris64-x86_64-gcc
Line 29: Line 108:
 
After configuring, run <tt>make</tt> as usual.
 
After configuring, run <tt>make</tt> as usual.
  
==Build the Test==
+
===Build the Test===
  
Typically after <tt>make</tt>'ing the library, you run <tt>make test</tt> to validate it. Since you already know there's a problem, you can only build the problem test with the following. The command below builds the HMAC test suite
+
You typically run <tt>make test</tt> to validate the library after <tt>make</tt>'ing. Since you already know there's a problem, you can build the problem test with the following. The command below is an example for tracking down a failure in the HMAC test suite
  
<pre>VERBOSE=1 make TESTS="test_hmac" test
+
<pre>$ VERBOSE=1 make TESTS="test_hmac" test
 
...
 
...
  
Line 59: Line 138:
 
Result: PASS
 
Result: PASS
 
`test' is up to date.</pre>
 
`test' is up to date.</pre>
 +
 +
=== Debug the Test ===
  
 
The executable for <tt>test_hmac</tt> is created from the source <tt>05-test_hmac.t</tt>, and it will be located at <tt>./test/buildtest_hmac</tt>:
 
The executable for <tt>test_hmac</tt> is created from the source <tt>05-test_hmac.t</tt>, and it will be located at <tt>./test/buildtest_hmac</tt>:
Line 64: Line 145:
 
<pre>$ file ./test/buildtest_hmac
 
<pre>$ file ./test/buildtest_hmac
 
./test/buildtest_hmac:  ELF 64-bit LSB executable AMD64 Version 1, dynamically linked, not stripped</pre>
 
./test/buildtest_hmac:  ELF 64-bit LSB executable AMD64 Version 1, dynamically linked, not stripped</pre>
 +
 +
You can manually execute the test with the following. Note that <tt>$PWD</tt> is the root of the OpenSSL distribution, and the libraries are put on path using <tt>LD_LIBRARY_PATH</tt>.
 +
 +
<pre>$ LD_LIBRARY_PATH=".:$LD_LIBRARY_PATH" test/hmactest
 +
^C
 +
$</pre>
 +
 +
You can run the test under a debugger with the following. As can be seen, the execution results in a loop between lines 184 and 189.
 +
 +
<pre>$ LD_LIBRARY_PATH=".:$LD_LIBRARY_PATH" gdb test/hmactest
 +
GNU gdb (GDB) 7.6
 +
...
 +
Reading symbols from .../openssl/test/hmactest...done.
 +
 +
(gdb) r
 +
Starting program: /export/home/jwalton/openssl/test/hmactest
 +
[Thread debugging using libthread_db enabled]
 +
 +
^C
 +
Program received signal SIGINT, Interrupt.
 +
OPENSSL_cleanse () at crypto/x86_64cpuid.s:184
 +
184            testq  $7,%rdi
 +
 +
(gdb) where
 +
#0  OPENSSL_cleanse () at crypto/x86_64cpuid.s:184
 +
#1  0xffff80ffba0eabec in hmac_ctx_cleanup (ctx=0x412ac0)
 +
    at crypto/hmac/hmac.c:144
 +
#2  0xffff80ffba0eac6f in HMAC_CTX_reset (ctx=0x412ac0)
 +
    at crypto/hmac/hmac.c:160
 +
#3  0xffff80ffba0eab68 in HMAC_CTX_new () at crypto/hmac/hmac.c:129
 +
#4  0xffff80ffba0eae2e in HMAC (evp_md=0xffff80ffba1afec0 <md5_md>,
 +
    key=0x412700 <test>, key_len=0,
 +
    d=0x412714 <test+20> "More text test vectors to stuff up EBCDIC machines :-)", n=54, md=0xffff80ffba1c8100 <m.9561> "", md_len=0x0)
 +
    at crypto/hmac/hmac.c:209
 +
#5  0x0000000000401a65 in main (argc=1, argv=0xffff80ffbffffa28)
 +
    at test/hmactest.c:105
 +
 +
(gdb) s
 +
185            jz      .Laligned
 +
(gdb)
 +
186            movb    %al,(%rdi)
 +
(gdb)
 +
187            leaq    -0(%rsi),%rsi
 +
(gdb)
 +
188            leaq    0(%rdi),%rdi
 +
(gdb)
 +
189            jmp    .Lot
 +
(gdb)
 +
184            testq  $7,%rdi
 +
(gdb)
 +
185            jz      .Laligned
 +
(gdb)
 +
186            movb    %al,(%rdi)
 +
(gdb)
 +
187            leaq    -0(%rsi),%rsi
 +
(gdb)
 +
188            leaq    0(%rdi),%rdi
 +
(gdb)
 +
189            jmp    .Lot</pre>
 +
 +
=== Debug an Engine ===
 +
 +
If you are testing an OpenSSL engine, then the procedures and paths are slightly different.
 +
 +
You manually build the engine with:
 +
 +
<pre>$ cd test
 +
$ make test/afalgtest</pre>
 +
 +
You can run the engine test under a debugger with the following:
 +
 +
<pre>$ OPENSSL_ENGINES=../engines/afalg gdb ./afalgtest</pre>

Latest revision as of 16:17, 9 February 2020

The OpenSSL library includes a test suite that exercises components from both libcrypto.a and libssl.a. Ensuring the library builds and executes its tests properly is a keystone to using the library.

This page will show you how to isolate the test and the cause under many conditions. The more information you can supply with a bug report or pull request, the easier it is for the team to identify the failure or accept proposed changes. Also see Item 17. I think I've found a bug, what should I do? in the OpenSSL FAQ.

Generally speaking, there are two generations of self tests. The first generation is from OpenSSL 1.0.2 and below. The second generation is from OpenSSL 1.1.0 and above. This page focuses on the second generation, or OpenSSL 1.1.0 and above.

You can sometimes isolate a bug with no-asm, and sometimes with -O0 or -O1. Ideally the bug will be present with no optimizations (-O0), but sometimes it takes some compiler analysis to tickle it (-O1).

OpenSSL 1.1.0[edit]

Build the Library[edit]

The first thing you should do after verifying the bug is to create a debug build of the OpenSSL library and attempt to reproduce it.

TODO: provide instrcutions.

Build the Test[edit]

You typically run make test to validate the library after make'ing.

$ make test
...

make[1]: Entering directory '/home/test_user/openssl-1.1.1d'
make[1]: Leaving directory '/home/test_user/openssl-1.1.1d'
make[1]: Entering directory '/home/test_user/openssl-1.1.1d'
( cd test; \
  mkdir -p test-runs; \
  SRCTOP=../. \
  BLDTOP=../. \
  RESULT_D=test-runs \
  PERL="/usr/bin/perl" \
  EXE_EXT= \
  OPENSSL_ENGINES=`cd .././engines 2>/dev/null && pwd` \
  OPENSSL_DEBUG_MEMORY=on \
    /usr/bin/perl .././test/run_tests.pl  )
../test/recipes/01-test_abort.t .................... ok
../test/recipes/01-test_sanity.t ................... ok
...
../test/recipes/15-test_dh.t ....................... ok
../test/recipes/15-test_dsa.t ...................... ok
../test/recipes/15-test_ec.t ....................... Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/5 subtests 
../test/recipes/15-test_ecdsa.t .................... Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/1 subtests 
../test/recipes/15-test_ecparam.t .................. Dubious, test returned 2 (wstat 512, 0x200)
Failed 2/134 subtests 
../test/recipes/15-test_genrsa.t ................... ok
../test/recipes/15-test_mp_rsa.t ................... ok  
../test/recipes/15-test_out_option.t ............... ok
../test/recipes/15-test_rsa.t ...................... ok
../test/recipes/15-test_rsapss.t ................... ok
...

Debug the Test[edit]

You can run a specific test by setting PERL5LIB and TOP variables as shown below.

# From the root of the OpenSSL directory
$ PERL5LIB=util/perl TOP=$PWD perl test/recipes/15-test_ec.t
1..5
ok 1 - require '/home/test_user/openssl-1.1.1d/test/recipes/tconversion.pl';
    # Subtest: /home/test_user/openssl-1.1.1d/test/ectest
    1..11
crypto/ec/ecp_nistp521.c:144:15: runtime error: load of misaligned address 0x7ffec57b10b7 for type 'limb', which requires 8 byte alignment
0x7ffec57b10b7: note: pointer points here
 c2 31 7e 7e f9  9b 42 6a 85 c1 b3 48 33  de a8 ff a2 27 c1 1d fe  28 59 e7 ef 77 5e 4b a1  ba 3d 4d
             ^ 
/home/test_user/openssl-1.1.1d/util/shlib_wrap.sh /home/test_user/openssl-1.1.1d/test/ectest => 1
not ok 2 - running ectest
#   Failed test 'running ectest'
#   at test/recipes/15-test_ec.t line 23.
# Subtest: ec conversions -- private key
    1..10
    ok 1 - initializing
...

Debug an Engine[edit]

TODO: provide instructions

OpenSSL 1.0.2[edit]

The instructions that follow help determine a failue in OpenSSL 1.0.2 and below. As of January 2020, OpenSSL 1.0.2 is end of life. You should move to OpenSSL 1.1.x as soon as possible.

Build the Library[edit]

The first thing you should do after verifying the bug is to create a debug build of the OpenSSL library and attempt to reproduce it. Creating a debug build is as easy as using -d configure argument. In the output below, notice -d caused the library to reduce optimizations (-O0) and increase debugging information (-g). However, the debugging information will lack symbolic constants because -g3 was not used:

$ ./config -d
Operating system: i86pc-whatever-solaris2
Configuring for solaris64-x86_64-gcc
...
CC            =gcc
CFLAG         =-m64 -Wall -DL_ENDIAN -O0 -g -pthread -DFILIO_H  -Wa,--noexecstack
...

To ensure most of your preferred flags are used, perform the following. Not all flags will be honored, but its an improvement over -d. The -DNDEBUG is used to ensure Posix's assert does not crash your program while under the debugger.

$ ./config no-asm -DNDEBUG -g3 -O0 -fno-omit-frame-pointer
Operating system: i86pc-whatever-solaris2
Configuring for solaris64-x86_64-gcc
...
CC            =gcc
CFLAG         =-m64 -Wall -DL_ENDIAN -O3 -pthread -DFILIO_H  -g3 -O0 -fno-omit-frame-pointer
...

After configuring, run make as usual.

Build the Test[edit]

You typically run make test to validate the library after make'ing. Since you already know there's a problem, you can build the problem test with the following. The command below is an example for tracking down a failure in the HMAC test suite

$ VERBOSE=1 make TESTS="test_hmac" test
...

( cd test; \
  SRCTOP=../. \
  BLDTOP=../. \
  PERL="/usr/local/bin/perl" \
  EXE_EXT= \
  OPENSSL_ENGINES=.././engines \
    /usr/local/bin/perl .././test/run_tests.pl test_hmac )
../test/recipes/05-test_hmac.t .. 
1..1
test 0 ok
test 1 ok
test 2 ok
test 3 ok
test 4 ok
test 5 ok
test 6 ok
../util/shlib_wrap.sh ./hmactest => 0
ok 1 - running hmactest
ok
All tests successful.
Files=1, Tests=1,  0 wallclock secs ( 0.03 usr  0.01 sys +  0.06 cusr  0.02 csys =  0.12 CPU)
Result: PASS
`test' is up to date.

Debug the Test[edit]

The executable for test_hmac is created from the source 05-test_hmac.t, and it will be located at ./test/buildtest_hmac:

$ file ./test/buildtest_hmac
./test/buildtest_hmac:  ELF 64-bit LSB executable AMD64 Version 1, dynamically linked, not stripped

You can manually execute the test with the following. Note that $PWD is the root of the OpenSSL distribution, and the libraries are put on path using LD_LIBRARY_PATH.

$ LD_LIBRARY_PATH=".:$LD_LIBRARY_PATH" test/hmactest
^C
$

You can run the test under a debugger with the following. As can be seen, the execution results in a loop between lines 184 and 189.

$ LD_LIBRARY_PATH=".:$LD_LIBRARY_PATH" gdb test/hmactest
GNU gdb (GDB) 7.6
...
Reading symbols from .../openssl/test/hmactest...done.

(gdb) r
Starting program: /export/home/jwalton/openssl/test/hmactest
[Thread debugging using libthread_db enabled]

^C
Program received signal SIGINT, Interrupt.
OPENSSL_cleanse () at crypto/x86_64cpuid.s:184
184             testq   $7,%rdi

(gdb) where
#0  OPENSSL_cleanse () at crypto/x86_64cpuid.s:184
#1  0xffff80ffba0eabec in hmac_ctx_cleanup (ctx=0x412ac0)
    at crypto/hmac/hmac.c:144
#2  0xffff80ffba0eac6f in HMAC_CTX_reset (ctx=0x412ac0)
    at crypto/hmac/hmac.c:160
#3  0xffff80ffba0eab68 in HMAC_CTX_new () at crypto/hmac/hmac.c:129
#4  0xffff80ffba0eae2e in HMAC (evp_md=0xffff80ffba1afec0 <md5_md>, 
    key=0x412700 <test>, key_len=0, 
    d=0x412714 <test+20> "More text test vectors to stuff up EBCDIC machines :-)", n=54, md=0xffff80ffba1c8100 <m.9561> "", md_len=0x0)
    at crypto/hmac/hmac.c:209
#5  0x0000000000401a65 in main (argc=1, argv=0xffff80ffbffffa28)
    at test/hmactest.c:105

(gdb) s
185             jz      .Laligned
(gdb) 
186             movb    %al,(%rdi)
(gdb) 
187             leaq    -0(%rsi),%rsi
(gdb) 
188             leaq    0(%rdi),%rdi
(gdb) 
189             jmp     .Lot
(gdb) 
184             testq   $7,%rdi
(gdb) 
185             jz      .Laligned
(gdb) 
186             movb    %al,(%rdi)
(gdb) 
187             leaq    -0(%rsi),%rsi
(gdb) 
188             leaq    0(%rdi),%rdi
(gdb) 
189             jmp     .Lot

Debug an Engine[edit]

If you are testing an OpenSSL engine, then the procedures and paths are slightly different.

You manually build the engine with:

$ cd test
$ make test/afalgtest

You can run the engine test under a debugger with the following:

$ OPENSSL_ENGINES=../engines/afalg gdb ./afalgtest