aws s3 cp does not honor request_checksum_calculation = WHEN_REQUIRED

[metadaddy]

When using the AWS CLI with a third-party S3-compatible service, such as Backblaze B2, aws s3 cp sends the x-amz-sdk-checksum-algorithm and x-amz-checksum-crc64nvme headers, even if request_checksum_calculation is set to WHEN_REQUIRED in the configuration.

Data Integrity Protections for Amazon S3 states that, when request_checksum_calculation is set to WHEN_REQUIRED, 'Checksum validation is performed only when required by the API operation.'

.aws/config

[profile b2]
endpoint_url = https://s3.us-west-004.backblazeb2.com
region = us-west-004
request_checksum_calculation = WHEN_REQUIRED
response_checksum_validation = WHEN_REQUIRED

Running the CLI:

% aws --profile b2 s3 cp ./hello.txt s3://metadaddy-private/
upload failed: ./hello.txt to s3://metadaddy-private/hello.txt An error occurred (InvalidArgument) when calling the PutObject operation: Unsupported header 'x-amz-sdk-checksum-algorithm' received for this API call.

This is happening because S3 Transfer Manager ensures that ChecksumAlgorithm is set, either to the user-specified value, or CRC64NVME (see the set_default_checksum_algorithm() function, added just a couple of days ago).

Now, in Botocore's resolve_request_checksum_algorithm() function, request_checksum_required evaluates to False, as you might expect, but, since ChecksumAlgorithm is set in the incoming params, if algorithm_member and algorithm_member in params: evaluates to True and the checksum headers are calculated and sent in the request.

I think TransferManager._add_operation_defaults() should only call set_default_checksum_algorithm() if request_checksum_calculation is set to WHEN_SUPPORTED. Something like this:

    def _add_operation_defaults(self, extra_args):
        if self.client.meta.config.request_checksum_calculation == "when_supported":
            set_default_checksum_algorithm(extra_args)

See aws/aws-cli#9214 for more context.

[jonathan343]

Hey @metadaddy,

Thank you for creating this issue!

I've opened PRs with changes very similar to yours. I've include some formatting and changelog entry information need for releasing a new patch version of s3transfer.

• Boto3 and AWS CLI v1:
  • Only set default checksum if request_checksum_calculation config is set to when_supported #329
• AWS CLI v2
  • Only set default checksum if request_checksum_calculation config is set to when_supported aws/aws-cli#9238

Our team will work on getting this PR merged and released as soon as possible!

[metadaddy]

Thanks, @jonathan343 - I closed my PR (#328).

Should I close this issue, since your PRs are both merged now, or wait until they are released in a new version of s3transfer?

[jonathan343]

Our daily automated release is currently in progress for both the version of s3transfer used by boto3/AWS CLI v1 and the version vendored in the AWS CLI v2. Once those complete, would you be able to confirm using the new versions successfully address your issue? If so, we can close this out once that's been verified.

[jonathan343]

Update:
This change has been released and is available in s3transfer-0.11.2. This will automatically be available in the latest versions of boto3 and the AWS CLI.

Please provide an update here if you're able to confirm this fix addresses the issue on your end. Thanks!

[z4kk0]

Hey there!
I also encountered this particular error.
Thanks for fixing this quickly!
Version 0.11.2 works for me!

[jonathan343]

Awesome, thanks for verifying!

I'll close this issue now that we've confirmed the issue is resolved. If there are any additional questions or concerns, feel free to reopen this issue or create a new one.

Thank you all for the help!

[metadaddy]

Thanks, @jonathan343. I can confirm that aws s3 cp in AWS CLI 2.23.5 (which incorporates s3transfer-0.11.2) does work correctly:

% aws --version                                                                    
aws-cli/2.23.5 Python/3.12.8 Darwin/24.2.0 source/arm64
% aws --profile b2 s3 cp ./hello.txt  s3://metadaddy-private/
upload: ./hello.txt to s3://metadaddy-private/hello.txt