SSL / TLS incompatibility when using ecdsa keys introduced in 26.5.2.3 (and 27.x)

[sstrollo]

Describe the bug
We are currently running distributed nodes using tls, and based on Erlang/OTP version 26.2.5 in production. When I tried to upgrade to 27.1 I ran into a problem where an old node couldn't talk to a new node (which is what we need to do as part of our upgrade). The connection fails with (on the node running 26.2.5):

=NOTICE REPORT==== 8-Oct-2024::17:24:15.648674 ===
TLS server: In state hello at ssl_handshake.erl:1740 generated SERVER ALERT: Fatal - Insufficient Security
 - no_suitable_signature_algorithm

and on the node running version 26.2.5.3 it fails with:

=NOTICE REPORT==== 8-Oct-2024::17:30:06.636603 ===
TLS client: In state hello received SERVER ALERT: Fatal - Insufficient Security

After some experimenting I realized that it happens because we are using ecdsa keys using the prime256v1 curve (everything works fine if I use rsa keys or e.g. ED25519) but ecdsa is what we are using in production...

To Reproduce
Start a distributed node using version 26.2.5, and another using 26.2.5.3 and try to setup a connection between them. I have created a repository otp-dist-test which contains exact steps how to reproduce in the README file all you need are two Erlang installations - everything else is provided in the reproduction repository.

Expected behavior
That the nodes can talk to each other :)

Affected versions
26.2.5.3 and 27.1

Additional context
Looking at the diff git diff OTP-26.2.5 OTP-26.2.5.3 -- lib/ssl/src it seems the filtering of digest algos has changed unfortunately I'm not quite able to figure out what would need to be done to be able to accept a connection with 26.2.5. (Actually I just realized that all three versions post 26.2.5: 26.2.5.1, 26.2.5.2, and 26.2.5.3 have changes in them - I haven't been able to pinpoint which of these versions introduced the incompatibility).

[sstrollo]

How to reproduce using the otp-dist-test repository:

Terminal A, with OTP 26.2.5 installed running on my host nephele:

nephele% cd /tmp

nephele% git clone https://github.com/sstrollo/otp-dist-test.git
Cloning into 'otp-dist-test'...
remote: Enumerating objects: 26, done.
remote: Counting objects: 100% (26/26), done.
remote: Compressing objects: 100% (16/16), done.
remote: Total 26 (delta 11), reused 24 (delta 9), pack-reused 0 (from 0)
Receiving objects: 100% (26/26), 4.71 KiB | 4.71 MiB/s, done.
Resolving deltas: 100% (11/11), done.

nephele% cd otp-dist-test

nephele% make start-a
mkdir -p certs/
openssl ecparam -name prime256v1 -genkey -noout -out certs/dist-cert.key
openssl req -new -key certs/dist-cert.key -out certs/dist-cert.csr \
	    -subj "/C=SE/L=Stockholm/O=Erlang/CN=nephele"
mkdir -p certs/
openssl ecparam -name prime256v1 -genkey -noout -out certs/dist-ca.key
openssl req -x509 -new -nodes -key certs/dist-ca.key -sha256 -days 730 -out certs/dist-ca.pem \
	    -subj "/C=SE/L=Stockholm/O=Erlang CA/CN=example.com"
openssl x509 -req -in certs/dist-cert.csr \
	    -CA certs/dist-ca.pem  -CAkey certs/dist-ca.key -CAcreateserial \
	    -out certs/dist-cert.pem -outform pem -days 365 -sha256 -extfile certs/dist-cert.cfg
Certificate request self-signature ok
subject=C=SE, L=Stockholm, O=Erlang, CN=nephele
/Applications/Xcode.app/Contents/Developer/usr/bin/make node-a
mkdir node-a
escript ./genrel.escript > node-a/a.rel
cd node-a && erlc a.rel
cd node-a && \
	  erl -boot ./a -proto_dist inet_tls -sname a \
	      -ssl_dist_optfile ../ssl_dist.conf
Erlang/OTP 26 [erts-14.2.5] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit]

Eshell V14.2.5 (press Ctrl+G to abort, type help(). for help)
(a@nephele)1> 

Terminal B, activating OTP 26.2.5.3 instead:

nephele% cd /tmp/otp-dist-test

nephele% kerl_deactivate

nephele% source ~/.kerl/installed/OTP-26.2.5.3/activate

nephele% make start-b
mkdir node-b
escript ./genrel.escript > node-b/b.rel
cd node-b && erlc b.rel
cd node-b && \
	  erl -boot ./b -proto_dist inet_tls -sname b \
	      -ssl_dist_optfile ../ssl_dist.conf
Erlang/OTP 26 [erts-14.2.5.3] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit]

Eshell V14.2.5.3 (press Ctrl+G to abort, type help(). for help)
(b@nephele)1> net:ping(a@nephele).
=NOTICE REPORT==== 8-Oct-2024::21:22:03.362989 ===
TLS client: In state hello received SERVER ALERT: Fatal - Insufficient Security

pang
(b@nephele)2> 

And the output in terminal A is:

=NOTICE REPORT==== 8-Oct-2024::21:22:03.352136 ===
TLS server: In state hello at ssl_handshake.erl:1740 generated SERVER ALERT: Fatal - Insufficient Security
 - no_suitable_signature_algorithm
(a@nephele)1> 

[IngelaAndin]

Well that curve should be supported, we call it by its other name secp256r1. Does your certificates use sha (sha1) as hash function in its signature?

[sstrollo]

Thanks Ingela!

Well that curve should be supported, we call it by its other name secp256r1.

Thanks Ingela - right, the curve works, but just to be clear: it works as long as I use 26.2.5.3 (or 27.1) to talk to each other - it is only when a node running an older version (26.2.5 or older) trying to talk to one of the newer nodes that it doesn't work. Could it be as simple as it having different names in the different versions?

Does your certificates use sha (sha1) as hash function in its signature?

I think so :) all the commands to generate the certificates are in the reproduction repository, I guess the -sha256 in this line: https://github.com/sstrollo/otp-dist-test/blob/7bcedf5c39cb57e46bdc2fc4aea7f960f9661378/Makefile#L40 is the signature algo?

[sstrollo]

Could it be as simple as it having different names in the different versions?

Sorry, no, that's not it :)

Presumably the error message from the node (B in the example above) running 26.2.5:

=NOTICE REPORT==== 8-Oct-2024::21:22:03.352136 ===
TLS server: In state hello at ssl_handshake.erl:1740 generated SERVER ALERT: Fatal - Insufficient Security
 - no_suitable_signature_algorithm

comes from ssl_handshake:do_select_hashsign() - so it seems the version of that function in 26.2.5 can't agree with the version of that function in 26.2.5.3....?

[IngelaAndin]

Well as you observed I do not think it is a name conflict. The name that you used is not used at all by our code. But you provide the parameters. Anyway it is not a SHA1 - problem either as sha256 is part of SHA2 and supported by default. I think there existed some bug related to ECDSA signature algorithms, I will try to check a little later today when that was fixed.

[IngelaAndin]

This change made in OTP-26.2.5.1 might be relevant.

 OTP-19107    Application(s): ssl
               Related Id(s): GH-8376

               If both TLS-1.3 and TLS-1.2 is supported and TLS-1.2
               negotiated convert TLS-1.3 ECDSA schemes to TLS-1.2
               hash and signature pairs for increased
               interoperability.

[IngelaAndin]

If you do not configure the signature algorithms to be used yourselves you can inspect defaults using for instances
ssl:signature_algs(default, 'tlsv1.3').

[jbevemyr]

Btw, the problem appears to have been introduced between 26.2.5.1 and 26.2.5.2. Ping works between 26.2.5 and 26.2.5.1, but not between 26.2.5.1 and 26.2.5.2.

[jbevemyr]

The ssl:signature_algs(default, 'tlsv1.3') gives the same result on both 26.2.5.1 and 26.2.5.2.

[eddsa_ed25519,eddsa_ed448,ecdsa_secp521r1_sha512,
 ecdsa_secp384r1_sha384,ecdsa_secp256r1_sha256,
 rsa_pss_pss_sha512,rsa_pss_pss_sha384,rsa_pss_pss_sha256,
 rsa_pss_rsae_sha512,rsa_pss_rsae_sha384,rsa_pss_rsae_sha256,
 rsa_pkcs1_sha512,rsa_pkcs1_sha384,rsa_pkcs1_sha256,
 {sha512,ecdsa},
 {sha384,ecdsa},
 {sha256,ecdsa}]

and the ssl_dist.conf we use is very basic, just points to the certificates.
Do you think the problem can be worked around by setting some parameter in the ssl_dist.conf?

%% -*- mode: erlang -*-
[
{client,
    [{cacertfile,"../certs/dist-ca.pem"},
     {certfile,"../certs/dist-cert.pem"},
     {reuse_sessions,false},
     {keyfile,"../certs/dist-cert.key"},
%     {password,"foobar"},
     {reuse_sessions,false},
     {verify,verify_peer},
     {secure_renegotiate,true}]}
,
{server,
    [{certfile,"../certs/dist-cert.pem"},
     {keyfile,"../certs/dist-cert.key"},
%     {password,"foobar"},
     {reuse_sessions,false},
     {cacertfile,"../certs/dist-ca.pem"},
     {reuse_sessions,false},
     {verify,verify_peer},
     {fail_if_no_peer_cert,true},
     {secure_renegotiate,true}]}
].

[IngelaAndin]

The only change between those releases to the ssl application is:

'OTP-19152 Application(s): ssl
Related Id(s): GH-8588

           When the default value for signature_algs is used,
           default the signature_algs_cert to the default value +
           rsa_pkcs1_sha1 to allow this algorithms for
           certificates but not for the TLS protocol. This is for
           better interoperability. If signature_algs is set
           explicitly signature_algs_cert must also be set
           explicitly if they should be different.

´´´'

which should not have any effect on you ECDSA-certs. Do you run different versions of cryptolib ? Could you verify what is the signature algorithm of your cert? You can decode it using public_key functions.

[jbevemyr]

Nope, it is all built on the same machine with the same openssl. The replication repo that Sebastian provided makes it really easy to reproduce, btw.

[jbevemyr]

The replication repo creates the certifidcates using openssl, very basic usage.

[IngelaAndin]

Well, I will try that but I may not have time to try until end of next week as I have a conference to prepare and attend.

[jbevemyr]

$ openssl x509 -in dist-cert.pem -text -noout
Certificate:
    Data:
        Version: 1 (0x0)
        Serial Number:
            16:c8:25:54:72:d5:17:25:9e:92:d2:ed:24:28:6e:df:1c:01:0a:2e
        Signature Algorithm: ecdsa-with-SHA256
        Issuer: C = SE, L = Stockholm, O = Erlang CA, CN = example.com
        Validity
            Not Before: Oct 10 08:47:16 2024 GMT
            Not After : Oct 10 08:47:16 2025 GMT
        Subject: C = SE, L = Stockholm, O = Erlang, CN = tio
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
                pub:
                    04:b5:bc:8c:3b:0e:07:97:2c:b4:0d:cc:7c:68:64:
                    12:24:15:34:89:87:58:8d:d2:d5:ea:57:05:28:48:
                    35:a9:be:52:7d:dd:83:41:97:1e:3e:48:54:08:c7:
                    09:f2:e5:64:b6:63:ea:62:7b:f2:4e:c8:ca:00:d2:
                    81:fc:25:41:58
                ASN1 OID: prime256v1
                NIST CURVE: P-256
    Signature Algorithm: ecdsa-with-SHA256
    Signature Value:
        30:44:02:20:45:36:c8:28:51:26:aa:36:f0:df:9f:e4:1c:94:
        4f:c7:d4:0d:58:92:af:6c:7b:8e:f2:8e:df:51:46:da:12:d9:
        02:20:23:ff:8c:d9:4c:29:35:18:2e:f0:cf:d2:5a:7c:2c:1a:
        12:b4:03:13:ea:86:6e:e6:c3:f1:47:15:6f:59:37:dc

{ok, B} = file:read_file("../certs/dist-cert.pem").
[{_, C, _}] = public_key:pem_decode(B).
rp(public_key:pkix_decode_cert(C, otp)).
#'OTPCertificate'{
    tbsCertificate =
        #'OTPTBSCertificate'{
            version = 0,
            serialNumber =
                130061197888767310875680520578826563719536904750,
            signature =
                #'SignatureAlgorithm'{
                    algorithm = {1,2,840,10045,4,3,2},
                    parameters = asn1_NOVALUE},
            issuer =
                {rdnSequence,
                    [[#'AttributeTypeAndValue'{type = {2,5,4,6},value = "SE"}],
                     [#'AttributeTypeAndValue'{
                          type = {2,5,4,7},
                          value = {utf8String,<<"Stockholm">>}}],
                     [#'AttributeTypeAndValue'{
                          type = {2,5,4,10},
                          value = {utf8String,<<"Erlang CA">>}}],
                     [#'AttributeTypeAndValue'{
                          type = {2,5,4,3},
                          value = {utf8String,<<"example.com">>}}]]},
            validity =
                #'Validity'{
                    notBefore = {utcTime,"241010084716Z"},
                    notAfter = {utcTime,"251010084716Z"}},
            subject =
                {rdnSequence,
                    [[#'AttributeTypeAndValue'{type = {2,5,4,6},value = "SE"}],
                     [#'AttributeTypeAndValue'{
                          type = {2,5,4,7},
                          value = {utf8String,<<"Stockholm">>}}],
                     [#'AttributeTypeAndValue'{
                          type = {2,5,4,10},
                          value = {utf8String,<<"Erlang">>}}],
                     [#'AttributeTypeAndValue'{
                          type = {2,5,4,3},
                          value = {utf8String,<<"tio">>}}]]},
            subjectPublicKeyInfo =
                #'OTPSubjectPublicKeyInfo'{
                    algorithm =
                        #'PublicKeyAlgorithm'{
                            algorithm = {1,2,840,10045,2,1},
                            parameters = {namedCurve,{1,2,840,10045,3,1,7}}},
                    subjectPublicKey =
                        #'ECPoint'{
                            point =
                                <<4,181,188,140,59,14,7,151,44,180,13,204,124,104,100,
                                  18,36,21,52,137,135,88,141,210,213,234,87,5,40,72,53,
                                  169,190,82,125,221,131,65,151,30,62,72,84,8,199,9,242,
                                  229,100,182,99,234,98,123,242,78,200,202,0,210,129,
                                  252,37,65,88>>}},
            issuerUniqueID = asn1_NOVALUE,
            subjectUniqueID = asn1_NOVALUE,extensions = asn1_NOVALUE},
    signatureAlgorithm =
        #'SignatureAlgorithm'{
            algorithm = {1,2,840,10045,4,3,2},
            parameters = asn1_NOVALUE},
    signature =
        <<48,68,2,32,69,54,200,40,81,38,170,54,240,223,159,228,
          28,148,79,199,212,13,88,146,175,108,123,142,242,142,
          223,81,70,218,18,217,2,32,35,255,140,217,76,41,53,24,
          46,240,207,210,90,124,44,26,18,180,3,19,234,134,110,
          230,195,241,71,21,111,89,55,220>>}

[jbevemyr]

Another observation is the following:

1. start node a with 26.2.5.1
2. start node b with 26.2.5.2

If b tries to net:ping(a@...) then you get

=NOTICE REPORT==== 10-Oct-2024::13:01:03.436809 ===
TLS server: In state hello at ssl_handshake.erl:1740 generated SERVER ALERT: Fatal - Insufficient Security
 - no_suitable_signature_algorithm

at node a (running 26.2.5.1) and the following on node b (running 26.2.5.2)

(b@tio)1> net:ping(a@tio).
=NOTICE REPORT==== 10-Oct-2024::13:01:03.444087 ===
TLS client: In state hello received SERVER ALERT: Fatal - Insufficient Security

pang

However, if I then do the ping in the other direction, ie, node a pings node b, then it works,
and from then on b can ping a, or course.

[IngelaAndin]

Hum ... it just hit me on the way home that I just recently made a patch for a refactoring bug (related to RSA legacy signature schemes) that occurred in OTP-26.2.1 and that fix also included the following diff to handle TLS-1.3 schemes names in TLS-1.2 code.

@@ -1881,7 +1881,7 @@ client_signature_schemes(ClientHashSigns, undefined) ->
     ClientHashSigns;
 client_signature_schemes(_, #signature_algorithms_cert{
                                signature_scheme_list = ClientSignatureSchemes}) ->
-    ClientSignatureSchemes.
+    ssl_cipher:signature_schemes_1_2(ClientSignatureSchemes).
 

This might explain your problem. Well now I do not have more time for this. So hope it helps otherwise I will have to be stalled to later.

[jbevemyr]

Thanks! No, worries, I'll continue digging in the meantime. Tried the patch without success.

[jbevemyr]

I did some wireshark traces and noticed that on a protocol level the initial hello looks a bit different between 26.2.5.1 and 26.2.5.2. When 26.2.5.2 attempts to talk to 26.2.5.1 (which fails) it sends the handshake extension signature_algorithms_cert, whereas 26.2.5.1 does not when it successfully connects to a 26.2.5.2 node.

[jbevemyr]

The problem apperas to be related to this change in ssl.erl:

diff --git a/lib/ssl/src/ssl.erl b/lib/ssl/src/ssl.erl
index 4cca50bfd6..a2d85a6a25 100644
--- a/lib/ssl/src/ssl.erl
+++ b/lib/ssl/src/ssl.erl
@@ -202,6 +202,7 @@
 
 -type sign_schemes()            :: [sign_scheme()].
 
+
 -type sign_scheme()             :: eddsa_ed25519
                                  | eddsa_ed448
                                  | ecdsa_secp384r1_sha384
@@ -2130,29 +2131,80 @@ server_name_indication_default(_) ->
 
 opt_signature_algs(UserOpts, #{versions := Versions} = Opts, _Env) ->
     [TlsVersion|_] = TlsVsns = [tls_version(V) || V <- Versions],
-    SA = case get_opt_list(signature_algs, undefined, UserOpts, Opts) of
-             {default, undefined} when ?TLS_GTE(TlsVersion, ?TLS_1_2) ->
-                 DefAlgs = tls_v1:default_signature_algs(TlsVsns),
-                 handle_hashsigns_option(DefAlgs, TlsVersion);
-             {new, Algs} ->
-                 assert_version_dep(signature_algs, Versions, ['tlsv1.2', 'tlsv1.3']),
-                 SA0 = handle_hashsigns_option(Algs, TlsVersion),
-                 option_error(SA0 =:= [], no_supported_algorithms, {signature_algs, Algs}),
-                 SA0;
-             {_, Algs} ->
-                 Algs
-         end,
-    SAC = case get_opt_list(signature_algs_cert, undefined, UserOpts, Opts) of
-              {new, Schemes} ->
-                  %% Do not send by default
-                  assert_version_dep(signature_algs_cert, Versions, ['tlsv1.2', 'tlsv1.3']),
-                  SAC0 = handle_signature_algorithms_option(Schemes, TlsVersion),
-                  option_error(SAC0 =:= [], no_supported_signature_schemes, {signature_algs_cert, Schemes}),
-                  SAC0;
-              {_, Schemes} ->
-                  Schemes
-          end,
-    Opts#{signature_algs => SA, signature_algs_cert => SAC}.
+    case ?TLS_GTE(TlsVersion, ?TLS_1_2) of
+        true ->
+            opt_signature_algs_valid(UserOpts, Opts, TlsVsns);
+        false ->
+            opt_signature_algs_not_valid(UserOpts, Opts)
+    end.
+
+opt_signature_algs_valid(UserOpts, #{versions := Versions} = Opts, [TlsVersion|_] = TlsVsns)->
+    SAC1 = case get_opt_list(signature_algs_cert, undefined, UserOpts, Opts) of
+               {new, Schemes} ->
+                   assert_version_dep(signature_algs_cert, Versions, ['tlsv1.2', 'tlsv1.3']),
+                   SAC0 = handle_signature_algorithms_option(Schemes, TlsVersion),
+                   option_error(SAC0 =:= [], no_supported_signature_schemes,
+                                {signature_algs_cert, Schemes}),
+                   SAC0;
+               {_, Schemes} ->
+                   Schemes
+           end,
+
+    {SA, SAC2} =
+        case get_opt_list(signature_algs, undefined, UserOpts, Opts) of
+            {default, undefined}  ->
+                %% Smooth upgrade path allow rsa_pkcs1_sha1 for signatures_algs_cert
+                %% by default as long as signature_algs is set to default
+                DefAlgs0 = tls_v1:default_signature_algs(TlsVsns),
+                DefAlgs = handle_hashsigns_option(DefAlgs0, TlsVersion),
+                DSAC0 = case SAC1 of
+                            undefined ->
+                                [default | DefAlgs ++ sha_rsa(TlsVersion)];
+                            _ ->
+                                SAC1
+                        end,
+                {DefAlgs, DSAC0};
+            {new, Algs} ->
+                assert_version_dep(signature_algs, Versions, ['tlsv1.2', 'tlsv1.3']),
+                SA0 = handle_hashsigns_option(Algs, TlsVersion),
+                option_error(SA0 =:= [], no_supported_algorithms, {signature_algs, Algs}),
+                DSAC0 = case SAC1 of
+                            %% If user sets signature_algs, signature_algs_cert default should
+                            %% be undefined.
+                            [default |_] ->
+                                undefined;
+                            SAC1 ->
+                                SAC1
+                        end,
+                {SA0, DSAC0};
+                     {old, Algs} ->
+                {Algs, SAC1}
+        end,
+    Opts#{signature_algs => SA, signature_algs_cert => SAC2}.
+
+opt_signature_algs_not_valid(UserOpts, #{versions := Versions} = Opts0)->
+    Opts =
+        case get_opt_list(signature_algs, undefined, UserOpts, Opts0) of
+            {default, undefined} ->
+                Opts0#{signature_algs => undefined};
+            {old, _} ->
+                Opts0;
+            _ ->
+                option_incompatible([signature_algs, {versions, Versions}])
+        end,
+    case get_opt_list(signature_algs_cert, undefined, UserOpts, Opts) of
+        {default, undefined} ->
+            Opts#{signature_algs_cert => undefined};
+        {old, _} ->
+            Opts;
+        _ ->
+            option_incompatible([signature_algs_cert, {versions, Versions}])
+    end.
+
+sha_rsa(?TLS_1_2) ->
+    [{sha, rsa}];
+sha_rsa(?TLS_1_3) ->
+    [rsa_pkcs1_sha1].

If I revert this, it works.

[jbevemyr]

I found this text in the ssl dokumentation

The client will send a signature_algorithms_cert extension (in the client hello message), if TLS version 1.2 (back-ported to TLS 1.2 in 24.1) or later is used, and the signature_algs_cert option is explicitly specified. By default, only the signature_algs extension is sent with the exception of when signature_algs option is not explicitly specified, in which case it will append the rsa_pkcs1_sha1 algorithm to the default value of signature_algs and use it as value for signature_algs_cert to allow certificates to have this signature but still disallow sha1 use in the TLS protocol, since 27.0.1 and 26.2.5.2.

Which appears to reference this behaviour. I experimented by setting the signature_algs setting in the dist-ssl.conf like this:

...
{client,
    [{cacertfile,"../certs/dist-ca.pem"},
     {certfile,"../certs/dist-cert.pem"},
     {reuse_sessions,false},
     {keyfile,"../certs/dist-cert.key"},
%     {password,"foobar"},
     {reuse_sessions,false},
     {verify,verify_peer},
     {signature_algs, [{sha512,ecdsa},
                       rsa_pss_pss_sha512,rsa_pss_rsae_sha512,
                       {sha512,rsa},
                       {sha384,ecdsa},
                       rsa_pss_pss_sha384,rsa_pss_rsae_sha384,
                       {sha384,rsa},
                       {sha256,ecdsa},
                       rsa_pss_pss_sha256,rsa_pss_rsae_sha256,
                       {sha256,rsa}]},
     {secure_renegotiate,true}]}
...

Which appears to work as a workaround.

[IngelaAndin]

@jbevemyr Yes, that should work perfectly as a workaround. The reason I suggested the patch above is that it is a place where the signature_algorithm_cert extension was handled incorrectly for TLS-1.2. If that should help the patch needs to be applied to the node running the server on the older OTP-release. I am sure the problem is not with the code you referred above but with the handling code of that extension. When the signature_algoritm_cert extension is not sent it will default to the value of signature_algs.

[IngelaAndin]

@jbevemyr @sstrollo so are you content with this now? I think it should be a solved problem.

[sstrollo]

Hej @IngelaAndin - we tried explicitly listing the signature_algs in the client settings in our new version (based on 27.1.2 now) and it works - it can talk to the old version, we are happy with the workaround. From our end totally fine to close this issue - thank you!