diff --git a/README.md b/README.md index 414553ac97a..1c7a89c2dbb 100644 --- a/README.md +++ b/README.md @@ -111,6 +111,8 @@ In addition to code reviews, s2n-tls is subject to regular static analysis, fuzz s2n-tls includes positive and negative unit tests and end-to-end test cases. +Unit test coverage can be viewed [here](https://dx1inn44oyl7n.cloudfront.net/main/index.html). Note that this represents unit coverage for a particular build. Since that build won't necessarily support all s2n-tls features, test coverage may be artificially lowered. + ##### Erase on read s2n-tls encrypts or erases plaintext data as quickly as possible. For example, decrypted data buffers are erased as they are read by the application. diff --git a/bindings/rust/bench/README.md b/bindings/rust/bench/README.md index 98926ebbbca..b541d5c0146 100644 --- a/bindings/rust/bench/README.md +++ b/bindings/rust/bench/README.md @@ -23,13 +23,13 @@ cargo bench --bench handshake --bench throughput -- --profile-time 5 rm -rf .cargo ``` -## Setup +## Setup -Setup is easy! Just have OpenSSL installed, generate Rust bindings for s2n-tls using `../generate.sh`, and generate certs using `scripts/generate-certs.sh`. +Setup is easy! Just have OpenSSL installed, generate Rust bindings for s2n-tls using `../generate.sh`, and generate certs using `scripts/generate-certs.sh`. -Dependencies are the same as with s2n-tls. Currently, this crate has only been tested on Ubuntu (both x86 and ARM), but we expect everything to work with other Unix environments. +Dependencies are the same as with s2n-tls. Currently, this crate has only been tested on Ubuntu (both x86 and ARM), but we expect everything to work with other Unix environments. -To bench with AWS-LC, Amazon's custom libcrypto implementation, first run `scripts/install-aws-lc.sh` to install AWS-LC for the bench crate. To then run the benchmarks with AWS-LC, use Cargo with either the flag `--config aws-lc-config/s2n.toml` or `--config aws-lc-config/rustls.toml` (or both). You can also append these configs to `.cargo/config.toml` to let Cargo automatically detect the settings without specifying the flags each time. +To bench with AWS-LC, Amazon's custom libcrypto implementation, first run `scripts/install-aws-lc.sh` to install AWS-LC for the bench crate. To then run the benchmarks with AWS-LC, use Cargo with either the flag `--config aws-lc-config/s2n.toml` or `--config aws-lc-config/rustls.toml` (or both). You can also append these configs to `.cargo/config.toml` to let Cargo automatically detect the settings without specifying the flags each time. ### Features @@ -37,7 +37,7 @@ Default features (`rustls` and `openssl`) can be disabled by running the benches ## Performance benchmarks -The handshake and throughput benchmarks can be run with the `cargo bench` command. Criterion will auto-generate an HTML report in `target/criterion/`. +The handshake and throughput benchmarks can be run with the `cargo bench` command. Criterion will auto-generate an HTML report in `target/criterion/`. Throughput benchmarks measure round-trip throughput with the client and server connections in the same thread for symmetry. In practice, a machine would either host only the client or only the server and use multiple threads, so throughput for a single connection could theoretically be up to ~4x higher than the values from the benchmarks (when run on the same machine). @@ -74,9 +74,24 @@ heaptrack target/release/memory (pair|client|server) (s2n-tls|rustls|openssl) To do historical benchmarks, run `scripts/bench-past.sh`. This will checkout old versions of s2n-tls back to v1.3.16 in `target/` and run benchmarks on those with the `historical-perf` feature, disabling Rustls and OpenSSL benches. +## PKI Structure +``` + ┌────root──────┐ + │ │ + │ │ + ▼ │ + branch │ + │ │ + │ │ + │ │ + ▼ ▼ + leaf client +``` +`generate-certs.sh` will generate 4 certificates for each key type, with the signing relationships that are indicated in the diagram above. This cert chain length was chosen because it matches the cert chain length used by public AWS services. + ### Caveats -The last version benched is v1.3.16, since before that, the s2n-tls Rust bindings have a different API and would thus require a different bench harness to test. +The last version benched is v1.3.16, since before that, the s2n-tls Rust bindings have a different API and would thus require a different bench harness to test. v1.3.30-1.3.37 are not benched because of depedency issues when generating the Rust bindings. However, versions before and after are benched, so the overall trend in performance can still be seen without the data from these versions. @@ -87,7 +102,7 @@ Because these benches take a longer time to generate (>30 min), we include the r Notes: - Two sets of parameters for the handshake couldn't be benched before 1.3.40, since security policies that negotiated those policies as their top choice did not exist before then. - There is no data from 1.3.30 to 1.3.37 because those versions have a dependency issue that cause the Rust bindings not to build. However, there is data before and after that period, so the performance for those versions can be inferred via interpolation. -- The improvement in throughput in 1.3.28 was most likely caused by the addition of LTO to the default Rust bindings build. +- The improvement in throughput in 1.3.28 was most likely caused by the addition of LTO to the default Rust bindings build. - Since the benches are run over a long time, noise on the machine can cause variability, and background processes can cause spikes. - The variability can be seen with throughput especially because it is calculated as the inverse of time taken. @@ -97,11 +112,11 @@ Notes: ## Implementation details -We use Rust bindings for s2n-tls and OpenSSL. All of our benchmarks are run in Rust on a single thread for consistency. +We use Rust bindings for s2n-tls and OpenSSL. All of our benchmarks are run in Rust on a single thread for consistency. ### IO -To remove external factors, we use custom IO with our benchmarks, bypassing the networking layer and having the client and server connections transfer data to each other via a local buffer. +To remove external factors, we use custom IO with our benchmarks, bypassing the networking layer and having the client and server connections transfer data to each other via a local buffer. ### Certificate generation diff --git a/bindings/rust/bench/benches/resumption.rs b/bindings/rust/bench/benches/resumption.rs index 7b0872660ff..e98887521e0 100644 --- a/bindings/rust/bench/benches/resumption.rs +++ b/bindings/rust/bench/benches/resumption.rs @@ -67,7 +67,7 @@ where /// s2n-tls?". pub fn bench_resumption(c: &mut Criterion) { // compare resumption savings across both client and server - for sig_type in [SigType::Rsa2048, SigType::Ecdsa384] { + for sig_type in [SigType::Rsa2048, SigType::Ecdsa256] { let mut bench_group = c.benchmark_group(format!("resumption-pair-{:?}", sig_type)); bench_handshake_pair::(&mut bench_group, sig_type); } diff --git a/bindings/rust/bench/certs/config/ca.cnf b/bindings/rust/bench/certs/config/ca.cnf deleted file mode 100644 index 66447a71611..00000000000 --- a/bindings/rust/bench/certs/config/ca.cnf +++ /dev/null @@ -1,13 +0,0 @@ -[ req ] -prompt = no -distinguished_name = ca_distinguished_name - -[ ca_distinguished_name ] -# country name -C = JP -# state or province -ST = Chiba -L = Chiba City -O = Tessier-Ashpool -CN = develop.localca -emailAddress = ca@develop.localca diff --git a/bindings/rust/bench/certs/config/client.cnf b/bindings/rust/bench/certs/config/client.cnf deleted file mode 100644 index 2018077e6c3..00000000000 --- a/bindings/rust/bench/certs/config/client.cnf +++ /dev/null @@ -1,24 +0,0 @@ -[req] -distinguished_name = client_distinguished_name -prompt = no - -# these fields are used to force x509 to v3 certificates -# rustls will complain otherwise -x509_extensions = v3_req -req_extensions = req_ext - -[client_distinguished_name] -countryName = US -stateOrProvinceName = Washington -localityName = Seattle -organizationName = client -commonName = client.localhost - -[v3_req] -subjectAltName = @alt_names - -[req_ext] -subjectAltName = @alt_names - -[alt_names] -DNS = localhost diff --git a/bindings/rust/bench/certs/config/server.cnf b/bindings/rust/bench/certs/config/server.cnf deleted file mode 100644 index 403932d8fcf..00000000000 --- a/bindings/rust/bench/certs/config/server.cnf +++ /dev/null @@ -1,24 +0,0 @@ -[req] -distinguished_name = server_distinguished_name -prompt = no - -# these fields are used to force x509 to v3 certificates -# rustls will complain otherwise -x509_extensions = v3_req -req_extensions = req_ext - -[server_distinguished_name] -countryName = US -stateOrProvinceName = Washington -localityName = Seattle -organizationName = server -commonName = server.localhost - -[v3_req] -subjectAltName = @alt_names - -[req_ext] -subjectAltName = @alt_names - -[alt_names] -DNS = localhost diff --git a/bindings/rust/bench/scripts/generate-certs.sh b/bindings/rust/bench/scripts/generate-certs.sh index 8fab212fc60..b2ac029a4f7 100755 --- a/bindings/rust/bench/scripts/generate-certs.sh +++ b/bindings/rust/bench/scripts/generate-certs.sh @@ -35,40 +35,108 @@ cert-gen () { mkdir -p $dir_name cd $dir_name + # The "basicConstraints" and "keyUsage" extensions are necessary for CA + # certificates that sign other certificates. Normally the openssl x509 tool + # will ignore the extensions requests in the .csr, but by using the + # copy_extensions=copyall flag we can pass the extensions from the .csr on + # to the final public certificate. + + # The advantage of manually specifying the extensions is that there is no + # dependency on any openssl config files + echo "generating CA private key and certificate" - openssl req -new -nodes -x509 -newkey $key_family -pkeyopt $argname$key_size -keyout ca-key.pem -out ca-cert.pem -days 65536 -config ../config/ca.cnf + openssl req -new -noenc -x509 \ + -newkey $key_family \ + -pkeyopt $argname$key_size \ + -keyout ca-key.pem \ + -out ca-cert.pem \ + -days 65536 \ + -subj "/C=US/CN=root" \ + -addext "basicConstraints = critical,CA:true" \ + -addext "keyUsage = critical,keyCertSign" + + echo "generating intermediate private key and CSR" + openssl req -new -noenc \ + -newkey $key_family \ + -pkeyopt $argname$key_size \ + -keyout intermediate-key.pem \ + -out intermediate.csr \ + -subj "/C=US/CN=branch" \ + -addext "basicConstraints = critical,CA:true" \ + -addext "keyUsage = critical,keyCertSign" echo "generating server private key and CSR" - openssl req -new -nodes -newkey $key_family -pkeyopt $argname$key_size -keyout server-key.pem -out server.csr -config ../config/server.cnf + openssl req -new -noenc \ + -newkey $key_family \ + -pkeyopt $argname$key_size \ + -keyout server-key.pem \ + -out server.csr \ + -subj "/C=US/CN=leaf" \ + -addext "subjectAltName = DNS:localhost" echo "generating client private key and CSR" - openssl req -new -nodes -newkey $key_family -pkeyopt $argname$key_size -keyout client-key.pem -out client.csr -config ../config/client.cnf + openssl req -new -noenc \ + -newkey $key_family \ + -pkeyopt $argname$key_size \ + -keyout client-key.pem \ + -out client.csr \ + -subj "/C=US/CN=client" \ + -addext "subjectAltName = DNS:localhost" + + echo "generating intermediate certificate and signing it" + openssl x509 -days 65536 \ + -req -in intermediate.csr \ + -CA ca-cert.pem \ + -CAkey ca-key.pem \ + -CAcreateserial \ + -out intermediate-cert.pem \ + -copy_extensions=copyall echo "generating server certificate and signing it" - openssl x509 -days 65536 -req -in server.csr -CA ca-cert.pem -CAkey ca-key.pem -CAcreateserial -out server-cert.pem -extensions req_ext -extfile ../config/server.cnf + openssl x509 -days 65536 \ + -req -in server.csr \ + -CA intermediate-cert.pem \ + -CAkey intermediate-key.pem \ + -CAcreateserial -out server-cert.pem \ + -copy_extensions=copyall echo "generating client certificate and signing it" - openssl x509 -days 65536 -req -in client.csr -CA ca-cert.pem -CAkey ca-key.pem -CAcreateserial -out client-cert.pem -extensions req_ext -extfile ../config/client.cnf - - echo "verifying generated certificates" - openssl verify -CAfile ca-cert.pem server-cert.pem + openssl x509 -days 65536 \ + -req -in client.csr \ + -CA ca-cert.pem \ + -CAkey ca-key.pem \ + -CAcreateserial -out client-cert.pem \ + -copy_extensions=copyall + + touch server-chain.pem + cat server-cert.pem >> server-chain.pem + cat intermediate-cert.pem >> server-chain.pem + cat ca-cert.pem >> server-chain.pem + + echo "verifying server certificates" + openssl verify -CAfile ca-cert.pem intermediate-cert.pem + openssl verify -CAfile ca-cert.pem -untrusted intermediate-cert.pem server-cert.pem + + echo "verifying client certificates" openssl verify -CAfile ca-cert.pem client-cert.pem echo "cleaning up temporary files" rm server.csr + rm intermediate.csr rm client.csr rm ca-key.pem cd .. } -if [[ $1 != "clean" ]] +if [[ $1 != "clean" ]] then + cert-gen ec 256 ecdsa256 cert-gen ec 384 ecdsa384 cert-gen rsa 2048 rsa2048 cert-gen rsa 3072 rsa3072 cert-gen rsa 4096 rsa4096 -else +else echo "cleaning certs" rm -rf ecdsa*/ rsa*/ fi diff --git a/bindings/rust/bench/src/harness.rs b/bindings/rust/bench/src/harness.rs index 5e7697cad80..237fbee534d 100644 --- a/bindings/rust/bench/src/harness.rs +++ b/bindings/rust/bench/src/harness.rs @@ -25,7 +25,7 @@ impl PemType { fn get_filename(&self) -> &str { match self { PemType::ServerKey => "server-key.pem", - PemType::ServerCertChain => "server-cert.pem", + PemType::ServerCertChain => "server-chain.pem", PemType::ClientKey => "client-key.pem", PemType::ClientCertChain => "client-cert.pem", PemType::CACert => "ca-cert.pem", @@ -40,6 +40,7 @@ pub enum SigType { Rsa4096, #[default] Ecdsa384, + Ecdsa256, } impl SigType { @@ -49,6 +50,7 @@ impl SigType { SigType::Rsa3072 => "rsa3072", SigType::Rsa4096 => "rsa4096", SigType::Ecdsa384 => "ecdsa384", + SigType::Ecdsa256 => "ecdsa256", } } } diff --git a/codebuild/bin/coverage_build.sh b/codebuild/bin/coverage_build.sh deleted file mode 100755 index f10d3a877d6..00000000000 --- a/codebuild/bin/coverage_build.sh +++ /dev/null @@ -1,29 +0,0 @@ -#!/bin/bash -# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"). -# You may not use this file except in compliance with the License. -# A copy of the License is located at -# -# http://aws.amazon.com/apache2.0 -# -# or in the "license" file accompanying this file. This file is distributed -# on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either -# express or implied. See the License for the specific language governing -# permissions and limitations under the License. - -set -e - -source codebuild/bin/s2n_setup_env.sh - -# LLVM complains about corrupt coverage information -# for static targets, so compile to a shared lib -# instead. -cmake . -Bbuild \ - -DCOVERAGE=ON \ - -DCMAKE_PREFIX_PATH=$LIBCRYPTO_ROOT \ - -DBUILD_SHARED_LIBS=ON - -cmake --build ./build -- -j $(nproc) - -echo "done with the build" diff --git a/codebuild/bin/coverage_upload.sh b/codebuild/bin/coverage_upload.sh deleted file mode 100755 index c29ea8761a5..00000000000 --- a/codebuild/bin/coverage_upload.sh +++ /dev/null @@ -1,25 +0,0 @@ -#!/bin/bash -# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"). -# You may not use this file except in compliance with the License. -# A copy of the License is located at -# -# http://aws.amazon.com/apache2.0 -# -# or in the "license" file accompanying this file. This file is distributed -# on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either -# express or implied. See the License for the specific language governing -# permissions and limitations under the License. -# - -set -e -echo "The codebuild source version is ${CODEBUILD_SOURCE_VERSION}" - -CLOUDFRONT_DISTRIBUTION="https://dx1inn44oyl7n.cloudfront.net" - -aws s3 sync coverage_report s3://s2n-tls-public-coverage-artifacts/${CODEBUILD_SOURCE_VERSION}/report -aws s3 cp coverage_summary.txt s3://s2n-tls-public-coverage-artifacts/${CODEBUILD_SOURCE_VERSION}/summary.txt - -echo "report : is at ${CLOUDFRONT_DISTRIBUTION}/${CODEBUILD_SOURCE_VERSION}/report/index.html" -echo "summary: is at ${CLOUDFRONT_DISTRIBUTION}/${CODEBUILD_SOURCE_VERSION}/summary.txt" diff --git a/codebuild/spec/buildspec_generalbatch.yml b/codebuild/spec/buildspec_generalbatch.yml index 09e46019343..d1e6bc6b5fe 100644 --- a/codebuild/spec/buildspec_generalbatch.yml +++ b/codebuild/spec/buildspec_generalbatch.yml @@ -335,14 +335,6 @@ batch: S2N_NO_PQ: 1 TESTS: unit identifier: s2nUnitClang15 - - identifier: s2nUnitCoverage - buildspec: codebuild/spec/buildspec_unit_coverage.yml - env: - privileged-mode: true - compute-type: BUILD_GENERAL1_LARGE - image: 024603541914.dkr.ecr.us-west-2.amazonaws.com/docker:ubuntu22codebuild - variables: - S2N_LIBCRYPTO: openssl-1.1.1 - identifier: 32BitBuildAndUnit buildspec: codebuild/spec/buildspec_32bit_cross_compile.yml env: diff --git a/codebuild/spec/buildspec_unit_coverage.yml b/codebuild/spec/buildspec_unit_coverage.yml index d8fbf3fb6df..633e491abf0 100644 --- a/codebuild/spec/buildspec_unit_coverage.yml +++ b/codebuild/spec/buildspec_unit_coverage.yml @@ -14,8 +14,6 @@ version: 0.2 env: - parameter-store: - CODECOV_TOKEN : codecov-upload-token variables: # CODEBUILD_ is a reserved namespace. CB_BIN_DIR: "./codebuild/bin" @@ -23,21 +21,25 @@ env: CXX: "/usr/bin/clang++" phases: - pre_build: - commands: - - | - if [ -d "third-party-src" ]; then - cd third-party-src; - ln -s /usr/local $CODEBUILD_SRC_DIR/third-party-src/test-deps; - fi - - ln -s /usr/local $CODEBUILD_SRC_DIR/test-deps build: on-failure: ABORT commands: - - $CB_BIN_DIR/coverage_build.sh + # LLVM complains about corrupt coverage information + # for static targets, so compile to a shared lib + # instead. + - | + cmake . -Bbuild \ + -DCOVERAGE=ON \ + -DCMAKE_PREFIX_PATH=$LIBCRYPTO_ROOT \ + -DBUILD_SHARED_LIBS=ON + - cmake --build ./build -- -j $(nproc) post_build: on-failure: ABORT commands: - - LLVM_PROFILE_FILE="ut_%p.profraw" CTEST_PARALLEL_LEVEL=$(nproc) cmake --build ./build --target test ARGS="--output-on-failure -L unit" + - LLVM_PROFILE_FILE="ut_%8m.profraw" CTEST_PARALLEL_LEVEL=$(nproc) cmake --build ./build --target test ARGS="--output-on-failure -L unit" - $CB_BIN_DIR/coverage_report.sh - - $CB_BIN_DIR/coverage_upload.sh +artifacts: + # upload all files in the coverage_report directory + files: + - '**/*' + base-directory: coverage_report