The TLS session with EdDSA host does not work

[Jakuje]

Describe the bug
The LTS does not work with EdDSA keys.

To Reproduce
Adjusting the test tests/ttls with

+    if [[ -n "$EDBASEURI" ]]; then
+        title PARA "Run sanity test with default values (EdDSA)"
+        run_test "$EDPRIURI" "$EDCRTURI"
+    fi

works as long as the pkcs11 provider is not forced. When we force the operations to the pkcs11 provider, we got failure even before accepting the connection such as:

Using default temp DH parameters
error setting certificate
80925F20587F0000:error:03000072:digital envelope routines:X509_PUBKEY_get0:decode error:crypto/x509/x_pubkey.c:464:
80925F20587F0000:error:0A00018F:SSL routines:SSL_CTX_use_certificate:ee key too small:ssl/ssl_rsa.c:239:

The debug log shows the following messages, where we process the OSSL_PARAMS from OpenSSL and we are unable to parse them:

[../src/keymgmt.c:1230] p11prov_ec_new(): ec new
[../src/keymgmt.c:1406] p11prov_ec_import(): ec import 0x5555559a3ff0
[../src/objects.c:2800] prep_ec_find(): Error: 0x00000067; Unable to decode ec group

This comes in this context:

(gdb) bt
#0  prep_ec_find (ctx=0x5555556530f0, params=0x5555559a3f70, findctx=0x7fffffffaef0) at ../src/objects.c:2775
#1  0x00007ffff77b52fc in p11prov_obj_import_public_key (key=0x5555559a3ff0, type=3, params=0x5555559a3f70)
    at ../src/objects.c:3077
#2  0x00007ffff77b71e8 in p11prov_obj_import_key (key=0x5555559a3ff0, type=3, class=2, params=0x5555559a3f70)
    at ../src/objects.c:3699
#3  0x00007ffff77a1bed in p11prov_ec_import (keydata=0x5555559a3ff0, selection=134, params=0x5555559a3f70)
    at ../src/keymgmt.c:1429
#4  0x00007ffff7b4ce8f in evp_keymgmt_import (keymgmt=<optimized out>, keydata=<optimized out>, 
    selection=<optimized out>, params=0x5555559a3f70) at crypto/evp/keymgmt_meth.c:470
#5  evp_keymgmt_util_try_import (params=0x5555559a3f70, arg=0x7fffffffb0d0) at crypto/evp/keymgmt_lib.c:50
#6  0x00007ffff7c47159 in ecx_export (keydata=0x5555559a3ed0, selection=<optimized out>, 
    param_cb=0x7ffff7b4cde0 <evp_keymgmt_util_try_import>, cbarg=0x7fffffffb0d0)
    at providers/implementations/keymgmt/ecx_kmgmt.c:262
#7  0x00007ffff7b169d4 in decoder_construct_pkey (decoder_inst=<optimized out>, params=<optimized out>, 
    construct_data=0x55555599a2d0) at crypto/encode_decode/decoder_pkey.c:167
#8  0x00007ffff7b155d4 in decoder_process (params=0x7fffffffb270, arg=0x7fffffffb3e0)
    at crypto/encode_decode/decoder_lib.c:774
#9  0x00007ffff7c22006 in der2key_decode (vctx=0x5555559a3370, cin=<optimized out>, selection=<optimized out>, 
    data_cb=0x7ffff7b15530 <decoder_process>, data_cbarg=0x7fffffffb3e0, pw_cb=<optimized out>, 
    pw_cbarg=0x5555559a2e08) at providers/implementations/encode_decode/decode_der2key.c:325
#10 0x00007ffff7b15811 in decoder_process (params=<optimized out>, arg=0x7fffffffb6a0)
    at crypto/encode_decode/decoder_lib.c:1000
#11 0x00007ffff7c23cd4 in spki2typespki_decode (vctx=<optimized out>, cin=<optimized out>, 
    selection=<optimized out>, data_cb=0x7ffff7b15530 <decoder_process>, data_cbarg=0x7fffffffb6a0, 
    pw_cb=<optimized out>, pw_cbarg=0x5555559a2e08)
    at providers/implementations/encode_decode/decode_spki2typespki.c:136
#12 0x00007ffff7b15811 in decoder_process (params=params@entry=0x0, arg=arg@entry=0x7fffffffb730)
    at crypto/encode_decode/decoder_lib.c:1000
--Type <RET> for more, q to quit, c to continue without paging--
#13 0x00007ffff7b15aeb in OSSL_DECODER_from_bio (ctx=ctx@entry=0x5555559a2dd0, in=in@entry=0x5555559a3ae0)
    at crypto/encode_decode/decoder_lib.c:82
#14 0x00007ffff7b15def in OSSL_DECODER_from_data (ctx=0x5555559a2dd0, pdata=0x7fffffffb7f8, 
    pdata_len=0x7fffffffb7f0) at crypto/encode_decode/decoder_lib.c:157
#15 0x00007ffff7c05109 in x509_pubkey_ex_d2i_ex (pval=<optimized out>, in=<optimized out>, len=<optimized out>, 
    it=<optimized out>, tag=<optimized out>, aclass=<optimized out>, opt=0 '\000', ctx=0x7fffffffbd00, libctx=0x0, 
    propq=0x0) at crypto/x509/x_pubkey.c:217
#16 0x00007ffff7a6a613 in asn1_item_embed_d2i (pval=pval@entry=0x5555556a5970, in=in@entry=0x7fffffffb9d0, 
    len=<optimized out>, len@entry=176, it=0x7ffff7e78ac0 <local_it>, tag=tag@entry=-1, aclass=aclass@entry=0, 
    opt=0 '\000', ctx=0x7fffffffbd00, depth=3, libctx=0x0, propq=0x0) at crypto/asn1/tasn_dec.c:262
#17 0x00007ffff7a6c5b2 in asn1_template_noexp_d2i (val=0x5555556a5970, in=0x7fffffffbac0, len=<optimized out>, 
    tt=tt@entry=0x7ffff7ea8190 <X509_CINF_seq_tt+240>, opt=<optimized out>, ctx=0x7fffffffbd00, depth=2, 
    libctx=0x0, propq=0x0) at crypto/asn1/tasn_dec.c:682
#18 0x00007ffff7a6c95d in asn1_template_ex_d2i (val=val@entry=0x5555556a5970, in=in@entry=0x7fffffffbac0, 
    inlen=inlen@entry=176, tt=tt@entry=0x7ffff7ea8190 <X509_CINF_seq_tt+240>, opt=<optimized out>, 
    ctx=ctx@entry=0x7fffffffbd00, depth=<optimized out>, libctx=<optimized out>, propq=<optimized out>)
    at crypto/asn1/tasn_dec.c:558
#19 0x00007ffff7a6a9dd in asn1_item_embed_d2i (pval=pval@entry=0x7fffffffbb88, in=in@entry=0x7fffffffbb80, 
    len=<optimized out>, len@entry=584, it=0x7ffff7e78a00 <local_it.1.lto_priv>, tag=<optimized out>, tag@entry=-1, 
    aclass=<optimized out>, aclass@entry=0, opt=0 '\000', ctx=0x7fffffffbd00, depth=2, libctx=0x0, propq=0x0)
    at crypto/asn1/tasn_dec.c:422
#20 0x00007ffff7a6c5b2 in asn1_template_noexp_d2i (val=0x7fffffffbb88, in=0x7fffffffbc70, len=<optimized out>, 
    tt=tt@entry=0x7ffff7ea8020 <X509_seq_tt>, opt=<optimized out>, ctx=0x7fffffffbd00, depth=1, libctx=0x0, 
    propq=0x0) at crypto/asn1/tasn_dec.c:682
#21 0x00007ffff7a6c95d in asn1_template_ex_d2i (val=val@entry=0x5555556a5920, in=in@entry=0x7fffffffbc70, 
    inlen=inlen@entry=584, tt=tt@entry=0x7ffff7ea8020 <X509_seq_tt>, opt=<optimized out>, 
    ctx=ctx@entry=0x7fffffffbd00, depth=<optimized out>, libctx=<optimized out>, propq=<optimized out>)
    at crypto/asn1/tasn_dec.c:558
#22 0x00007ffff7a6a9dd in asn1_item_embed_d2i (pval=pval@entry=0x7fffffffbe40, in=0x7fffffffbd60, 
    len=<optimized out>, it=it@entry=0x7ffff7e789c0 <local_it.0.lto_priv>, tag=<optimized out>, tag@entry=-1, 
    aclass=<optimized out>, aclass@entry=0, opt=0 '\000', ctx=0x7fffffffbd00, depth=1, libctx=0x0, propq=0x0)
    at crypto/asn1/tasn_dec.c:422
#23 0x00007ffff7a6b0c0 in asn1_item_ex_d2i_intern (pval=0x7fffffffbe40, in=<optimized out>, len=<optimized out>, 
    it=<optimized out>, tag=-1, aclass=0, opt=0 '\000', ctx=0x7fffffffbd00, libctx=<optimized out>, 
    propq=<optimized out>) at crypto/asn1/tasn_dec.c:118
#24 ASN1_item_d2i_ex (pval=0x7fffffffbe40, in=<optimized out>, len=<optimized out>, 
    it=0x7ffff7e789c0 <local_it.0.lto_priv>, libctx=<optimized out>, propq=<optimized out>)
    at crypto/asn1/tasn_dec.c:144
#25 0x00007ffff7c0232a in d2i_X509_AUX (a=0x7fffffffbe40, pp=0x7fffffffbe10, length=588) at crypto/x509/x_x509.c:201
#26 0x00007ffff7bccc30 in try_cert (data=0x7fffffffbdf0, v=<optimized out>, 
    libctx=0x7ffff7eacc60 <default_context_int.lto_priv>, propq=0x0) at crypto/store/store_result.c:481
#27 ossl_store_handle_load_result (params=<optimized out>, arg=<optimized out>) at crypto/store/store_result.c:138
#28 0x00007ffff77cb989 in p11prov_store_load (pctx=0x5555556a56e0, 
    object_cb=0x7ffff7bcc750 <ossl_store_handle_load_result>, object_cbarg=0x7fffffffc070, 
    pw_cb=0x7ffff7b83010 <ossl_pw_passphrase_callback_dec>, pw_cbarg=0x5555556a0ce8) at ../src/store.c:412
#29 0x00007ffff7bcb2cf in OSSL_STORE_load (ctx=ctx@entry=0x5555556a0ca0) at crypto/store/store_lib.c:447
#30 0x00005555555d4c24 in load_key_certs_crls (uri=<optimized out>, 
    uri@entry=0x7fffffffceb3 "pkcs11:type=cert;object=edCert", format=format@entry=0, 
    maybe_stdin=maybe_stdin@entry=1, pass=pass@entry=0x0, desc=desc@entry=0x5555555f5985 "server certificate", 
    quiet=quiet@entry=0, ppkey=<optimized out>, ppubkey=<optimized out>, pparams=<optimized out>, 
    pcert=<optimized out>, pcerts=0x0, pcrl=<optimized out>, pcrls=0x0) at apps/lib/apps.c:1021
#31 0x00005555555d57d7 in load_cert_pass (uri=0x7fffffffceb3 "pkcs11:type=cert;object=edCert", format=0, 
    maybe_stdin=1, pass=0x0, desc=0x5555555f5985 "server certificate") at apps/lib/apps.c:497
#32 0x00005555555be37a in s_server_main (argc=<optimized out>, argv=<optimized out>) at apps/s_server.c:1795
#33 0x00005555555a070e in do_cmd (prog=prog@entry=0x55555564fb20, argc=argc@entry=10, 
    argv=argv@entry=0x7fffffffc870) at apps/openssl.c:426
#34 0x000055555557dd88 in main (argc=<optimized out>, argv=<optimized out>) at apps/openssl.c:307
(gdb) info locals
group = 0x1
point = 0x5555559a3ff0
bn_ctx = 0x7fffffffaed0
tmp = {
  key = 0x7ffff7b4cde0 <evp_keymgmt_util_try_import> "\363\017\036\372UH\211\345AUATI\211\374SH\211\363H\203\354\bH\203~\b", data_type = 4294947024, data = 0x7fffffffae80, data_size = 140737346393489, return_size = 1}
p = 0x7fffffffaeb0
pub_key = {{key = 0x55555564cd00 "\204<\255\373\005", data_type = 4154413136, data = 0x5555556542c0, 
    data_size = 70, return_size = 140737488334432}, {key = 0x7ffff789c6fd <new_do_write+93> "\017\267\273\200", 
    data_type = 4152188800, data = 0x46, data_size = 140737347801168, return_size = 93824996701552}}
pub_data = "559a3ff0\340\255\377\377\377\177\000\000\253Z}\367\377\177\000\000\000\257\377\377\377\177\000\000\240\255\377\377\377\177\000\000\214\272\206\367\377\177", '\000' <repeats 11 times>, "\315dUUU\000\000Ю\377\377\377\177\000\000\033s\207\367\377\177", '\000' <repeats 11 times>, "\217A\033\232-\017\247\360\256\377\377\377\177\000\000а\377\377\377\177\000\000p?\232UUU\000\000pu\231UUU\000\000\005CeUUU\000\000\005CeUUU\000\000\300ReUUU"
digest_data = {{data = 0x5555556552c0 "", length = 21}, {
    data = 0x555500000004 <error: Cannot access memory at address 0x555500000004>, length = 93824993250560}, {
    data = 0x5555556552c0 "", length = 18}, {data = 0x4 <error: Cannot access memory at address 0x4>, 
    length = 93824993250560}, {data = 0x5 <error: Cannot access memory at address 0x5>, 
    length = 3834029163820351424}}
digest = {
  data = 0x5555556542ed "ec import 0x5555559a3ff0\n (slotid=1, ret=e0)\n", 'f' <repeats 12 times>, ", rw=false)\n9551615\n\n=1\nn Len Range: 4-255\n  Public  Memory  Total: 18446744073709551615  Free: 18446744073709551615\n  Private Memory  Total: 1"..., length = 93824993280749}
curve_name = 0x7ffff79f5050 <_IO_file_jumps> ""
curve_nid = 0
ecparams = 0x7fffffffaec8 "\374R{\367\377\177"
len = 32767
i = 21845
rv = 12037890468890119936
__func__ = "prep_ec_find"
(gdb) info args 
ctx = 0x5555556530f0
params = 0x5555559a3f70
findctx = 0x7fffffffaef0

Expected behavior
The TLS context is possible to establish with EdDSA keys.

Operating environment (please complete the following information):

• OS: Fedora
• Version 40

Token and application used (please complete the following information):

• Device: SoftHSM
• PKCS11 Driver version: softhsm-2.6.1-9.fc40.x86_64
• Application openssl
• Version openssl-3.2.2-3.fc40.x86_64

There is a WIP branch with the test modified to demonstrate the issue and attempt to fix it, but even though I managed to get further to import the eddsa key, it fails later when attempting to do the EdDSA signatures:

https://github.com/Jakuje/pkcs11-provider/tree/tls-eddsa

90: C_Sign
P:624000; T:0x139785848611456 2024-11-04 16:40:17.479
[in] hSession = 0x2
[in] pData[ulDataLen] 00007ffffe00f140 / 146
    00000000  20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20                  
    00000010  20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20                  
    00000020  20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20                  
    00000030  20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20                  
    00000040  54 4C 53 20 31 2E 33 2C 20 73 65 72 76 65 72 20  TLS 1.3, server 
    00000050  43 65 72 74 69 66 69 63 61 74 65 56 65 72 69 66  CertificateVerif
    00000060  79 00 2F 84 2D 55 FB D1 0E 00 85 7E C2 44 77 3D  y./.-U.....~.Dw=
    00000070  00 2A 33 52 D7 A2 61 47 E9 EA 8B 8F C2 EE 3E 33  .*3R..aG......>3
    00000080  F4 19 EC B9 D5 F9 21 8E F3 C1 0D 5C A4 B5 08 57  ......!....\...W
    00000090  F6 95                                            ..              
Returned:  5 CKR_GENERAL_ERROR

I will investigate this further.

[Jakuje]

Turns out the server operations work ok, but the client has an issue when it has the OPENSSL_CONF forcing all the operation in the token, causing infinite recursion while trying to verify the EdDSA signatures. The provider asks the softhsm for the signature verification and it goes through the OpenSSL EVP API back to the pkcs11 provider, until we exhaust stack. We do not hit this issue in other key types, because they go through the old deprecated API, which is not redirected to the pkcs11 provider.

Ways out after some brainstorming with @beldmit

• Fix softhsm not to use global context -- not great
• Do not force the operations on the client -- this removes some test coverage we care about
• Use the -propquery '?provider=pkcs11' switch on the s_client CLI. This makes the propquery limited to the s_client context and not propagated to the softhsm.

This should be fixed in #465.