Announcement: S3 default integrity change

[aBurmeseDev]

In AWS SDK for Rust v1.69.0, we released changes to the S3 client that adopts new default integrity protections. For more information on default integrity behavior, please refer to the official SDK documentation. In SDK releases from this version on, clients default to enabling an additional checksum on all Put calls and enabling validation on Get calls.

You can disable default integrity protections for S3. We do not recommend this because checksums are important to S3 integrity posture. Integrity protections can be disabled by setting the config flag to WHEN_REQUIRED, or by using the related AWS shared config file settings or environment variables.

Disclaimer: the AWS SDKs and CLI are designed for usage with official AWS services. We may introduce and enable new features by default, such as these new default integrity protections, prior to them being supported or otherwise handled by third-party service implementations. You can disable the new behavior with the WHEN_REQUIRED value for the request_checksum_calculation and response_checksum_validation configuration options covered in Data Integrity Protections for Amazon S3.

[danprueitt]

Seems changing the default breaks AWS S3 Presigned PUTs if you don't explicitly set a checksum? I can see a checksum is added to the headers and when the client attempts an upload it gets:

Error><Code>BadDigest</Code><Message>The CRC32 you specified did not match the calculated checksum.</Message>
...

The issue was resolved by setting the config to WHEN_REQUIRED, but this feels like a breaking behavior change in this scenario. Wouldn’t it be better to manage this change behind a behavior_version in the AWS config instead?

[ysaito1001]

Hi @danprueitt, thank you for bringing that to our attention. Sounds like it's a regular S3 presigned PUT requests that are affected? Can you provide us with a reproducer so we can work backwards from it?

[landonxjames]

@danprueitt I was able to replicate your issue and working on a fix now, thanks for bringing this to our attention.

[aaronmondal]

It looks like this change added the crc64fast-nvme dependency which contains bindgen code as of awesomized/crc64fast-nvme@1ec317f. This is somewhat hard to work with in sandboxed builds (like Bazel) as it tries to write to read-only directories. IIUC the aws sdk doesn't actually make use of the bindings.

See also: awesomized/crc64fast-nvme#5

[landonxjames]

Fix was merged in smithy-lang/smithy-rs#3971, we are working to release the codegen now and the actual fix should be released in version 1.70.0 of the aws-sdk-s3 crate on Monday

[landonxjames]

@aaronmondal I've created a PR to the crs64fast-nvme crate to hopefully fix that issue (since we were experiencing it internally as well): awesomized/crc64fast-nvme#6

[nidor1998]

Thank you for the announcement. I have one question: why does the SDK use CRC32 instead of CRC64NVME?

Checking object integrity in Amazon S3 recommends CRC64NVME and boto3 also uses CRC64NVME as the default.

[aajtodd]

I have one question: why does the SDK use CRC32 instead of CRC64NVME?

Primarily because we could not enable CRC64NVME on all targets by default due to crc64fast-nvme not supporting targets without hardware acceleration. We may change the default to CRC64NVME in the future once we can sort out the target support. For now you can opt into whichever algorithm you want by explicitly setting it instead of relying on the default.

[nidor1998]

Primarily because we could not enable CRC64NVME on all targets by default due to crc64fast-nvme not supporting targets without hardware acceleration. We may change the default to CRC64NVME in the future once we can sort out the target support. For now you can opt into whichever algorithm you want by explicitly setting it instead of relying on the default

Thank you for your explanation. I've understood it well.

[landonxjames]

There have been asks of various SDKs for examples of how to revert to something like the previous behavior, especially for S3 APIs like delete_objects where there is currently no workaround besides pinning to a previous version. The code below makes use of a CustomizableOperation to insert the Content-MD5 header and remove the flexible checksum headers.

Cargol.toml

[package]
name = "md5-override"
version = "0.1.0"
edition = "2021"

[dependencies]
aws-config = "1.5.15"
aws-sdk-s3 = "1.72.0"
aws-smithy-types = "1.2.13"
md5 = "0.7.0"
tokio = { version = "1.43.0", features = ["macros", "rt-multi-thread"] }

main.rs

use aws_config::BehaviorVersion;
use aws_config::Region;
use aws_sdk_s3::config::http::HttpRequest;
use aws_sdk_s3::types::{Delete, ObjectIdentifier};
use aws_sdk_s3::Client;
use aws_smithy_types::base64;

#[tokio::main]
async fn main() {
    let region = Region::from_static("us-west-2");
    let config = aws_config::defaults(BehaviorVersion::latest())
        .region(region)
        .load()
        .await;
    let s3_client = Client::new(&config);

    let delete_objs_res = s3_client
        .delete_objects()
        .bucket("NOT_A_REAL_BUCKET")
        .delete(
            Delete::builder()
                .objects(
                    ObjectIdentifier::builder()
                        .key("NOT_A_REAL_KEY")
                        .build()
                        .unwrap(),
                )
                .build()
                .unwrap(),
        )
        .customize()
        .mutate_request(calculate_md5_checksum_and_remove_other_checksums)
        // Check that the headers were correctly mutated
        .mutate_request(|req| {
            println!("HEADERS: {:#?}", req.headers());
        })
        .send()
        .await;

    println!("RESULT: {delete_objs_res:#?}")
}

/// This function mutates the request to insert a Content-MD5 header and remove
/// any existing flexible checksum headers
fn calculate_md5_checksum_and_remove_other_checksums(http_request: &mut HttpRequest) {
    // Remove the flexibile checksum headers
    let remove_headers = http_request.headers().clone();
    let remove_headers: Vec<(&str, &str)> = remove_headers
        .iter()
        .filter(|(name, _)| {
            name.starts_with("x-amz-checksum") || name.starts_with("x-amz-sdk-checksum")
        })
        .collect();

    for (name, _) in remove_headers {
        http_request.headers_mut().remove(name);
    }

    // Check if the body is present if it isn't (streaming request) we skip adding the header
    if let Some(bytes) = http_request.body().bytes() {
        let md5 = md5::compute(bytes);
        let checksum_value = base64::encode(md5.as_slice());
        http_request
            .headers_mut()
            .append("Content-MD5", checksum_value);
    }
}

[hveiga]

Hi @landonxjames is there plans to provide a better user experience for these delete_objects calls? Maybe enabling the previous behavior through a flag in the config? Thanks.

[landonxjames]

Hi @landonxjames is there plans to provide a better user experience for these delete_objects calls? Maybe enabling the previous behavior through a flag in the config? Thanks.

There are no plans for that. The SDK's backwards compatibility guarantees only apply to AWS services and the APIs as they currently are work with S3. Happy to help customers figure out workarounds for their individual situations, but unlikely to make any features in the SDK specifically targeting deprecated behavior.