Add PQ integration tests between s2n and AWS-LC's libssl

[alexw91]

Resolved issues:

N/A

Description of changes:

Adds hybrid post-quantum TLS integration tests between s2n and AWS-LC that is currently in a PR here: aws/aws-lc#1201

Call-outs:

This new integration test will fail in s2n's current GitHub CI until the following are completed:

1. PR 1201 is merged into AWS-LC
2. AWS-LC cuts a new release that has PR 1201 in it
3. The install_awslc.sh script is updated to use this new release version.
4. (Potentially?) s2n's CI Docker images that contain cached test dependency artifacts may need to be regenerated to contain the latest version of aws-lc in the ./test-deps directory.

Testing:

Locally removed all other tests in test_pq_handshake.py file and confirmed that these tests pass locally on my Ubuntu EC2 instance:

$ S2N_LIBCRYPTO=awslc BUILD_S2N=true TESTS=integrationv2 GCC_VERSION=9
$ source ./codebuild/bin/s2n_setup_env.sh
$ clear; TOX_TEST_NAME="test_pq_handshake" make -C tests/integrationv2
make: Entering directory '/home/ubuntu/workspace/github/s2n-tls/tests/integrationv2'
( DYLD_LIBRARY_PATH="/home/ubuntu/workspace/github/s2n-tls/test-deps/awslc/lib:$DYLD_LIBRARY_PATH" LD_LIBRARY_PATH="/home/ubuntu/workspace/github/s2n-tls/test-deps/awslc/lib:"/test-deps/openssl-1.1.1/lib":"/test-deps/gnutls37/nettle/lib":$LD_LIBRARY_PATH" S2N_INTEG_TEST=1 PATH="/bin":"/test-deps/openssl-1.1.1/bin":"/test-deps/gnutls37/bin":/home/ubuntu/workspace/github/s2n-tls/codebuild/bin:/home/ubuntu/workspace/github/s2n-tls/test-deps/openssl-1.1.1/bin:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin PYTHONNOUSERSITE=1 TOX_TEST_NAME=test_pq_handshake.py python3.9 -m tox )
py39 installed: attrs==23.1.0,cffi==1.16.0,cryptography==36.0.2,execnet==2.0.2,more-itertools==10.1.0,nassl==4.0.2,packaging==23.2,pep8==1.7.1,pluggy==0.13.1,py==1.11.0,pycparser==2.21,pydantic==1.8.2,pytest==5.3.5,pytest-forked==1.6.0,pytest-rerunfailures==12.0,pytest-xdist==1.34.0,six==1.16.0,sslyze==5.0.2,tls-parser==2.0.1,typing-extensions==4.8.0,wcwidth==0.2.8
py39 run-test-pre: PYTHONHASHSEED='2087458580'
py39 run-test: commands[0] | pytest -x -n=2 --maxfail=1 --reruns=2 --cache-clear -rpfsq -o log_cli=true --log-cli-level=INFO --provider-version=awslc --provider-criterion=off --fips-mode=0 --no-pq=0 test_pq_handshake.py
========================================================================================= test session starts =========================================================================================
platform linux -- Python 3.9.18, pytest-5.3.5, py-1.11.0, pluggy-0.13.1
cachedir: .tox/py39/.pytest_cache
rootdir: /home/ubuntu/workspace/github/s2n-tls/tests/integrationv2
plugins: forked-1.6.0, rerunfailures-12.0, xdist-1.34.0
[gw0] Python 3.9.18 (main, Aug 25 2023, 13:20:04)  -- [GCC 9.4.0]
[gw1] Python 3.9.18 (main, Aug 25 2023, 13:20:04)  -- [GCC 9.4.0]
gw0 [5] / gw1 [5]
scheduling tests via LoadScheduling

test_pq_handshake.py::test_nothing 
test_pq_handshake.py::test_s2nc_to_awslc_pq_handshake[SecP256r1Kyber768Draft00-PQ-TLS-1-3-2023-06-01] 
[gw0] [ 20%] PASSED test_pq_handshake.py::test_nothing 
test_pq_handshake.py::test_s2nc_to_awslc_pq_handshake[X25519Kyber768Draft00-PQ-TLS-1-3-2023-06-01] 
[gw1] [ 40%] PASSED test_pq_handshake.py::test_s2nc_to_awslc_pq_handshake[SecP256r1Kyber768Draft00-PQ-TLS-1-3-2023-06-01] 
test_pq_handshake.py::test_s2nd_to_awslc_pq_handshake[SecP256r1Kyber768Draft00-PQ-TLS-1-3-2023-06-01] 
[gw0] [ 60%] PASSED test_pq_handshake.py::test_s2nc_to_awslc_pq_handshake[X25519Kyber768Draft00-PQ-TLS-1-3-2023-06-01] 
test_pq_handshake.py::test_s2nd_to_awslc_pq_handshake[X25519Kyber768Draft00-PQ-TLS-1-3-2023-06-01] 
[gw1] [ 80%] PASSED test_pq_handshake.py::test_s2nd_to_awslc_pq_handshake[SecP256r1Kyber768Draft00-PQ-TLS-1-3-2023-06-01] 
[gw0] [100%] PASSED test_pq_handshake.py::test_s2nd_to_awslc_pq_handshake[X25519Kyber768Draft00-PQ-TLS-1-3-2023-06-01] 

======================================================================================= short test summary info =======================================================================================
PASSED test_pq_handshake.py::test_nothing
PASSED test_pq_handshake.py::test_s2nc_to_awslc_pq_handshake[SecP256r1Kyber768Draft00-PQ-TLS-1-3-2023-06-01]
PASSED test_pq_handshake.py::test_s2nc_to_awslc_pq_handshake[X25519Kyber768Draft00-PQ-TLS-1-3-2023-06-01]
PASSED test_pq_handshake.py::test_s2nd_to_awslc_pq_handshake[SecP256r1Kyber768Draft00-PQ-TLS-1-3-2023-06-01]
PASSED test_pq_handshake.py::test_s2nd_to_awslc_pq_handshake[X25519Kyber768Draft00-PQ-TLS-1-3-2023-06-01]
========================================================================================== 5 passed in 0.91s ==========================================================================================
_______________________________________________________________________________________________ summary _______________________________________________________________________________________________
  py39: commands succeeded
  congratulations :)
make: Leaving directory '/home/ubuntu/workspace/github/s2n-tls/tests/integrationv2'
ubuntu@ip-172-31-23-238:~/workspace/github/s2n-tls$ 

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[dougch]

1. Add HybridKeyShare support for SecP256r1Kyber768Draft00 and X25519Kyber768Draft00 aws-lc#1201 is merged into AWS-LC
2. AWS-LC cuts a new release that has PR 1201 in it
3. The install_awslc.sh script is updated to use this new release version.
4. s2n's CI Docker images that contain cached test dependency artifacts may need to be regenerated to contain the latest version of aws-lc in the ./test-deps directory.

These all appear complete, the current codebuild container was built with v1.17.4
Kicked off a retry of the failed general job.

[jmayclin]

It feels a bit odd to me that we have to manually specify these. We currently have AWS-LC integ tests, and as far as I can tell we don't have to specify these for the other tests?

Ideally we'd launch aws-lc here the same way that we do in other tests. If that's not possible, could you add a comment explaning why the extra env variables are needed?

[alexw91]

as far as I can tell we don't have to specify these for the other tests?

We do still set up environment variables when testing s2n with libOQS's fork of Openssl.

Removing all awslc_env_vars resulted in CI failure. From trial and error, it seems that setting awslc_env_vars["PATH"] = os.path.abspath("../../test-deps/awslc/bin") is the only required value for this to pass in CI.

[jmayclin]

Ideally we wouldn't be using any CI specific paths in our integration tests ("test-deps") since it makes these tests really hard to run from a local dev machine. however I've found that our integration tests are already really difficult to run from a local dev machine, so it does feel a little rich to hold you to that standard 😉.

[jmayclin]

Ideally we wouldn't be using any CI specific paths in our integration tests ("test-deps") since it makes these tests really hard to run from a local dev machine. however I've found that our integration tests are already really difficult to run from a local dev machine, so it does feel a little rich to hold you to that standard 😉.