ci: add more libcryptos for fuzz batch & follow cmake idioms

[jouho]

Resolved issues:

1. The current fuzz batch jobs are built with awslc and openssl-3.0. This PR adds additional supported libcrypto versions to increase coverage.
2. Test executables and override libraries are currently generated under the tests/fuzz and tests/fuzz/LD_PRELOAD folders. We should follow cmake build idioms (w.r.t binary location & library location)

Description of changes:

• Added openssl-1.0.2 and openssl-1.1.1 to the s2nFuzzBatch job.

• Modified CMakeLists.txt to remove logic that generates test binaries and library files in specific folders. Now, when compiling fuzz tests:

  • Overriding libraries are stored in s2n-tls/build/lib
  • Test executables are stored in s2n-tls/build/bin.

• Updated runFuzzTest.sh to adjust the file paths accordingly.

Call-outs:

I attempted to add awslc-fips to the batch job, but it failed during the compile. Link to Failed CodeBuild job
I also tried awslc-fips-2022, which compiled successfully but failed to find the libcrypto. Link to CodeBuild job

I have created new issue to investigate this: #4800

After this PR is merged, I will also update s2nOmnibus to include these libcrypto versions to ensure that the patches won’t break anything.

Testing:

Tested by overriding s2nFuzzBatch job against this PR, and they are finding correct libcrypto versions:
Link to CodeBuild job

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[dougch]

I'd like to run runFuzzTest.sh locally, using the syntax it'll be called with...

[dougch]

Are these arguments going to be updated in a future PR? or rather, can I review how this script is being called somewhere ?

[jouho]

This script is getting called in CMakeList.txt here:
https://github.com/jouho/s2n-tls/blob/cd9eaa5fd589e7e8e785203c7dc4792e25f56436/CMakeLists.txt#L673

SCRIPT_PATH is the path to runFuzzTest.sh. Does this answer your question?

[lrstewart]

Only in CMakeList.txt? No existing codebuild job is calling it?

[jouho]

This is only ran in CMake after we migrated away from make. codebuild jobs will indirectly call it through cmake.

[jouho]

I'd like to run runFuzzTest.sh locally, using the syntax it'll be called with...

You could run the same command we have in buildspec_fuzz.yml, which is

cmake . -Bbuild \
-DCMAKE_PREFIX_PATH=/usr/local/$S2N_LIBCRYPTO \
-DS2N_FUZZ_TEST=on

Then

cmake --build ./build -- -j $(nproc)

And then

cmake --build build/ --target test -- ARGS="-L fuzz --output-on-failure"

[lrstewart]

Modified CMakeLists.txt to remove logic that generates test binaries and library files in specific folders

Do you know why it was using specific folders before? Was is possibly due to concerns about name collisions / to avoid overwriting a non-fuzzing library or binary?

[jouho]

This was a result of inheriting make build configuration. CMake by default stores executables and libraries under build/ folder. In order to mimic the behavior of make, I had to specify specific folder to store these files. Now that I am making things a bit more "cmaky", I am removing that logic.

I couldn't find any mention of the original intention for where to store these files, but that is a good call-out. Should we store all the fuzz-related binaries under a specific folder, something like build/fuzz/bin for executables and build/fuzz/lib for libraries?

[jmayclin]

I'd prefer to keep the cmake standard paths, but we should make sure LD_PRELOAD is getting set to the actual libraries and not just the build/lib

[jouho]

we should make sure LD_PRELOAD is getting set to the actual libraries and not just the build/lib

What do you mean by this? Currently the .so files are being generated in build/lib, and we set the LD_PRELOAD path to be "${BUILD_DIR_PATH}/lib/lib${TEST_NAME}_overrides.so" and "${BUILD_DIR_PATH}/lib/libglobal_overrides.so". Do you suggest storing the .so files elsewhere?

[jmayclin]

Nope, that's great.

To Lindsay's point, moving everything to a generic build/lib would have been a problem if we were only specifying the LD_PRELOAD by folder (LD_PRELOAD=build/lib/), because then shenanigans would ensue, and all of the test specific overrides would be pulled in for all tests.

[dougch]

Running locally and reviewing your ad-hoc job, it looks like the tests were being run as singletons/serially. Somewhere you should have a CTEST_PARALLEL_LEVEL=$(nproc) or a -j flag to tell ctest to run in parallel.

[jouho]

Running multiple fuzz tests concurrently is a bit complicated, because we are already running individual tests multi-threaded. Even if we run multiple tests concurrently, the total duration would remain approximately the same, as long as we ensure each individual test's runtime remains consistent to maintain coverage.

If our goal is to reduce the total test time, there is a tracking issue #4748 to investigate how much code coverage fuzzing achieves over time. Total test time may be reduced if we reach peak coverage early into the fuzz test.