Cryptogams SHA

Cryptogams is Andy Polyakov's project used to develop high speed cryptographic primitives and share them with other developers. This wiki article will show you how to use Cryptogams ARMv4 SHA-1 implementation. According to the head notes the ARMv4 implementation runs around 6.5 cycles per byte (cpb). Typical C/C++ implementations run around 10 to 20 cpb and Andy's routines should outperform all of them.

Andy's Cryptogam implementations are provided by OpenSSL, but they are also available stand alone under a BSD license. The BSD style license is permissive and allows developers to use Andy's high speed cryptography without an OpenSSL dependency or licensing terms.

There are 6 steps to the process. The first step obtains the sources. The second step creates an ASM source file. The third step compiles and assembles the source file into an object file. The fourth steps determines the API. The fifth step creates a C header file. The final step integrates the object file into a program. Once you create the files sha1-armv4.h and sha1-armv4.S you can use sed to restore symbols back to their Cryptogams name with sed -i 's|OPENSSL|CRYPTOGAMS|g' sha1-armv4.h sha1-armv4.S.

A few cautions before you begin. First, you are going to examine undocumented features of the OpenSSL library to learn how to work with the Cryptogam's sources. The Cryptogam sources are stable but things could change over time. Second, the ARMv4 implementation hashes full SHA blocks. You are responsible for things like padding and side channel counter-measures.

If you experience "unexpected reloc type 0x03" when building a shared object then see What does unexpected reloc type 0x03 mean? on the Binutils mailing list.

Obtain Source Files

There are two source files you need for Cryptogams SHA. The first is arm-xlate.pl and the second is sha1-armv4.pl. They are available in the OpenSSL sources. The following commands fetch OpenSSL and then peels off the two Cryptogams files of interest.

# Clone OpenSSL for the latest Cryptogams sources
git clone https://github.com/openssl/openssl.git

mkdir cryptogams/

cp ./openssl/crypto/perlasm/arm-xlate.pl ./cryptogams/
cp ./openssl/crypto/sha/asm/sha1-armv4-large.pl ./cryptogams/
cp ./openssl/crypto/arm_arch.h cryptogams/

cd cryptogams/

Create ASM File

The second step is to run sha1-armv4-large.pl to produce an assembly language source file that can be consumed by GCC. sha1-armv4-large.pl internally calls arm-xlate.pl. linux32 is the flavor used by the translate program. sha1-armv4.S is the output filename. In the command below note the *.S file extension, which is a capitol S. Do not use a lowercase s because GCC must drive the compile and assemble step.

perl sha1-armv4-large.pl linux32 sha1-armv4.S

GCC is needed to drive the process because there are C macros in the source file. Some Cryptogam source files have this requirement, while some others do not. sha1-armv4 happens to have the requirement.

$ cat sha1-armv4.S
@ Copyright 2007-2018 The OpenSSL Project Authors. All Rights Reserved.
...

#ifndef __KERNEL__
# include "arm_arch.h"
#else
# define __ARM_ARCH__ __LINUX_ARM_ARCH__
#endif
...

At this point there is an ASM file but it needs two small fixups. First, arm_arch.h is an OpenSSL source file so the dependency must be removed. Second, GCC defines __ARM_ARCH instead of __ARM_ARCH__ so a sed is needed.

To fixup the source files execute the following two commands:

# Remove OpenSSL include
sed -i 's/# include "arm_arch.h"//g' sha1-armv4.S

# Fix GCC defines
sed -i 's/__ARM_ARCH__/__ARM_ARCH/g' sha1-armv4.S

Alternately, instead of the two sed's, you can open arm_arch.h, copy the defines and paste them directly into sha1-armv4.S. Take care when using arm_arch.h as it carries the OpenSSL license.

After the two fixups sha1-armv4.S is ready to be compiled by GCC.

Compile Source File

The source file is ready to be compiled and assembled. At this point there are two choices. First, you can use ARMv5t or higher which includes Thumb instructions. The following compiles the source file with ARMv5t.

$ gcc -march=armv5t -c sha1-armv4.S

The second choice uses ARMv4 and avoids Thumb instructions. If you want to avoid Thumb then add -marm to you compile command.

$ gcc -march=armv4 -marm -c sha1-armv4.S

Using ARMv5t as an example you now have an object file with the following symbols. Symbols with a capitol T are public and exported. Symbols with a lower t are private and should not be used.

$ gcc -march=armv4 -marm -c sha1-armv4.S
$ nm sha1-armv4.o
00000000 T sha1_block_data_order

And you can inspect the generated code with objdump.

$ objdump --disassemble sha1-armv4.o
sha1-armv4.o:     file format elf32-littlearm

Disassembly of section .text:

00000000 <sha1_block_data_order>:
   0:   e92d5ff0        push    {r4, r5, r6, r7, r8, r9, sl, fp, ip, lr}
   4:   e0812302        add     r2, r1, r2, lsl #6
   8:   e89000f8        ldm     r0, {r3, r4, r5, r6, r7}
   c:   e59f858c        ldr     r8, [pc, #1420] ; 5a0 <sha1_block_data_order+0x5a0>
  10:   e1a0e00d        mov     lr, sp
  14:   e24dd03c        sub     sp, sp, #60     ; 0x3c
  18:   e1a05f65        ror     r5, r5, #30
  1c:   e1a06f66        ror     r6, r6, #30

...

Determine API

The next step is determine the API so you can call it from a C program. Unfortunately the API is not documented and you have to dig around the OpenSSL sources. Fortunately there is one function of interest called sha1_block_data_order.

A quick grep of OpenSSL sources reveals the following for sha1_block_data_order.

openssl$ grep -nIR sha1_block_data_order | grep '\.c'
crypto/evp/e_sha_cbc_hmac_sha1.c:95:    void sha1_block_data_order(void *c, const void *p, size_t len);
crypto/evp/e_sha_cbc_hmac_sha1.c:115:        sha1_block_data_order(c, ptr, len / SHA_CBLOCK);
crypto/evp/e_sha_cbc_hmac_sha1.c:615:        sha1_block_data_order(&key->md, data, 1);
crypto/evp/e_sha_cbc_hmac_sha1.c:631:        sha1_block_data_order(&key->md, data, 1);
...

We need several more symbols, and and they are OPENSSL_armcap_P, ARMV7_NEON and ARMV8_SHA1.

$ grep -nIR OPENSSL_armcap_P
...
crypto/armcap.c:20:unsigned int OPENSSL_armcap_P = 0;

Lather, rinse, repeat for ARMV7_NEON and ARMV8_SHA1.

Create C Header

The fifth step creates a C header file based on information from Determine API. The header file is needed for two reasons. First, it removes the OpenSSL dependency from your project. Second, it avoids OpenSSL licensing violations.

Below is the C Header file you can use. While it is not obvious, the len parameter from Determine API is a block count, not a byte count.

/* Header file for use with Cryptogam's ARMv4 SHA1.    */
/* Also see http://www.openssl.org/~appro/cryptogams/  */
/* https://wiki.openssl.org/index.php/Cryptogams_SHA.  */

#ifndef CRYPTOGAMS_SHA1_ARMV4_H
#define CRYPTOGAMS_SHA1_ARMV4_H

#ifdef __cplusplus
extern "C" {
#endif

extern unsigned int OPENSSL_armcap_P;
void sha1_block_data_order(void *state, const void *data, size_t blocks);

/* Auxval caps */
#ifndef HWCAP_NEON
# define HWCAP_NEON (1 << 12)
#endif
#ifndef HWCAP_SHA1
# define HWCAP_SHA1 (1 << 5)
#endif

/* OpenSSL caps */
#define ARMV7_NEON (1<<0)
#define ARMV8_SHA1 (1<<3)

#ifdef __cplusplus
}
#endif

#endif  /* CRYPTOGAMS_SHA1_ARMV4_H */

Test Program

The final step is to test the integration of Cryptogam's SHA with your program.

$ gcc -std=c99 sha1-armv4-test.c ./sha1-armv4.o -o sha1-armv4-test.exe
$ ./sha1-armv4-test.exe
SHA1 hash of empty message: DA39A3EE5E6B4B0D...
Success!

And the test program is shown below.

#define _GNU_SOURCE
#include <stdio.h>
#include <stdint.h>
#include <string.h>
#include <sys/auxv.h>
#include "sha1-armv4.h"

/* processor caps */
unsigned int OPENSSL_armcap_P = 0;

int main(int argc, char* argv[])
{
    /* processor caps */
    if (getauxval(AT_HWCAP) & HWCAP_NEON)
        OPENSSL_armcap_P |= ARMV7_NEON;
    if (getauxval(AT_HWCAP) & HWCAP_SHA1)
        OPENSSL_armcap_P |= ARMV8_SHA1;

    /* empty message with padding */
    uint8_t message[64];
    memset(message, 0x00, sizeof(message));
    message[0] = 0x80;

    /* initial state */
    uint32_t state[5] = {0x67452301, 0xEFCDAB89, 0x98BADCFE, 0x10325476, 0xC3D2E1F0};

    sha1_block_data_order(state, message, 1);

    const uint8_t b1 = (uint8_t)(state[0] >> 24);
    const uint8_t b2 = (uint8_t)(state[0] >> 16);
    const uint8_t b3 = (uint8_t)(state[0] >>  8);
    const uint8_t b4 = (uint8_t)(state[0] >>  0);
    const uint8_t b5 = (uint8_t)(state[1] >> 24);
    const uint8_t b6 = (uint8_t)(state[1] >> 16);
    const uint8_t b7 = (uint8_t)(state[1] >>  8);
    const uint8_t b8 = (uint8_t)(state[1] >>  0);

    /* DA39A3EE5E6B4B0D... */
    printf("SHA1 hash of empty message: ");
    printf("%02X%02X%02X%02X%02X%02X%02X%02X...\n",
        b1, b2, b3, b4, b5, b6, b7, b8);

    int success = ((b1 == 0xDA) && (b2 == 0x39) && (b3 == 0xA3) && (b4 == 0xEE) &&
                    (b5 == 0x5E) && (b6 == 0x6B) && (b7 == 0x4B) && (b8 == 0x0D));

    if (success)
        printf("Success!\n");
    else
        printf("Failure!\n");

    return (success != 0 ? 0 : 1);
}

Symbol Names

The article used the same names as they appeared in the Cryptogams source code. For example, sha1_block_data_order is the names of function in the source code, and they will show up in the object file and when compiled and in the library when linked.

It is possible the function and date names will collide if you also link to OpenSSL, either directly or indirectly. If you plan on using Cryptogams code in a shared object then you should rename all symbols to avoid collisions. To rename symbols for SHA-1 you should rename sha1_block_data_order and OPENSSL_armcap_P. Assuming you are using MYLIB as a prefix the following sed should do the job.

sed -i 's/OPENSSL/MYLIB/g' sha1_armv4.h sha1_armv4.S
sed -i 's/sha1_block_data_order/MYLIB_sha1_block_data_order/g' sha1_armv4.h sha1_armv4.S

You can verify public symbols were renamed with nm aes-armv4.o. Generally speaking, all symbols with capitol letters like T (public function), B (uninitialized data), C (common data), D (initialized data), and R (read-only data) should be renamed.

Benchmarks

You can perform a rough benchmark using the code shown below. Prior to executing the benchmark program you should move the CPU from on-demand or powersave to performance mode.

#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
#include <time.h>
#include <unistd.h>
#include <string.h>
#include <sys/auxv.h>
#include "sha1-armv4.h"

/* processor caps */
unsigned int OPENSSL_armcap_P = 0;

int main(int argc, char* argv[])
{
    /* set processor caps */
    if (getauxval(AT_HWCAP) & HWCAP_NEON)
        OPENSSL_armcap_P |= ARMV7_NEON;
    if (getauxval(AT_HWCAP) & HWCAP_SHA1)
        OPENSSL_armcap_P |= ARMV8_SHA1;

    const unsigned int STEPS = 128;
    uint8_t* buf = (uint8_t*)malloc(STEPS*64+64);
    memset(buf, 0x00, 16);

    double elapsed = 0.0;
    size_t total = 0;

    struct timespec start, end;
    clock_gettime(CLOCK_PROCESS_CPUTIME_ID, &start);

    uint32_t state[5] = {0x67452301, 0xEFCDAB89, 0x98BADCFE, 0x10325476, 0xC3D2E1F0};

    do
    {
        size_t idx = 0;
        for (unsigned int i=0; i<STEPS; ++i)
            sha1_block_data_order(state, buf, idx+1);
        total += 64*STEPS;
        
        clock_gettime(CLOCK_PROCESS_CPUTIME_ID, &end);
        elapsed = (end.tv_sec-start.tv_sec);
    }
    while (elapsed < 3 /* seconds */);

    /* Increase precision of elapsed time */
    elapsed = ((double)end.tv_sec-start.tv_sec) +
              ((double)end.tv_nsec-start.tv_nsec) / 1000 / 1000 / 1000;

    /* CPU freq of 1 GHz */
    const double cpuFreq = 1000.0*1000*1000;

    const double bytes = total;
    const double ghz = cpuFreq / 1000 / 1000 / 1000;
    const double mbs = bytes / elapsed / 1024 / 1024;
    const double cpb = elapsed * cpuFreq / bytes;
    
    printf("%.0f bytes\n", bytes);
    printf("%.02f mbs\n", mbs);
    printf("%.02f cpb\n", cpb);
    
    free(buf);
    
    return 0;
}

The results below are from a Libre Computer Tritium H3 with a Cortex-A7 Sun7i SoC running at 1 GHz. A C/C++ SHA implementation runs about 22 cpb on the dev-board. Notice sha1-armv4.S was compiled with -march=armv7.

$ gcc -std=c99 -march=armv7 -c sha1-armv4.S -o sha1-armv7.o
$ gcc -O3 -std=c99 sha1-armv7-test.c sha1-armv7.o -o sha1-armv7-test.exe
$ ./sha1-armv7-test.exe
180994048 bytes
57.59 mbs
16.56 cpb

iOS Builds

sha1-armv4 can be configured for iOS. Simply use ios32 or ios64 instead of linux32 as shown below.

$ perl sha1-armv4-large.pl ios32 sha1-armv4.S
$ clang -arch armv7 sha1-armv4.S -c

And then:

$ nm sha1-armv4.o
000012d0 s OPENSSL_armcap_P
00000004 C _OPENSSL_armcap_P
00000000 T _sha1_block_data_order
00001100 t sha1_block_data_order_armv8
00000560 t sha1_block_data_order_neon

$ otool -tV sha1-armv4.o
sha1-armv4.o:
(__TEXT,__text) section
_sha1_block_data_order:
00000000        f8dfc4ec        ldr.w   r12, [pc, #0x4ec]
00000004        f2af0308        subw    r3, pc, #0x8
00000008        f853c00c        ldr.w   r12, [r3, r12]
0000000c        f8dcc000        ldr.w   r12, [r12]
00000010        f01c0f08        tst.w   r12, #0x8
00000014        f0418074        bne.w   sha1_block_data_order_armv8
00000018        f01c0f01        tst.w   r12, #0x1
0000001c        f04082a0        bne.w   sha1_block_data_order_neon
00000020        e92d5ff0        push.w  {r4, r5, r6, r7, r8, r9, r10, r11, r12, lr}
...

ARMv8 Builds

If you need SHA for ARMv8 devices, which includes aarch64 and aarch32, then use sha1-armv8.pl. You would use something like this in your scripts:

if ! perl sha1-armv8.pl linux64 sha1-armv8.S; then
    echo "Failed to translate SHA source file"
    exit 1
fi

sha1-armv8.pl provides the following function:

• sha1_block_armv8

If you are building for an Apple M1 you may need to use flavor ios64. The translate script does not handle osx64 properly.