Add Content-MD5 or checksum header for using an Object Locked S3

[trueleo]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
I am trying to use object-store crate with AWS S3 where Object lock is enabled with versioning and some retention period. When i am trying to put object it is failing with following error

... <Error><Code>InvalidRequest</Code><Message>Content-MD5 OR x-amz-checksum- HTTP header is required for Put Object requests with Object Lock parameters</Message> ...

Describe the solution you'd like
A ClientOptions method like with_content_md5 or with_checksum_header that will calculate and attach appropriate checksum header ( before sending Bytes ) so that PUT request works.

Describe alternatives you've considered
None

Additional context
This is minimal example I used for testing this out

#[tokio::main]
async fn main() {
    let client = AmazonS3Builder::new()
        .with_endpoint("https://s3.us-east-1.amazonaws.com")
        .with_access_key_id(".....")
        .with_secret_access_key(".....")
        .with_bucket_name("....")
        .with_region("us-east-1")
        .build()
        .unwrap();

    client
        .put(
            &Path::from("foo"),
            Bytes::from_static(b"content of file"),
        )
        .await
        .unwrap();
}

second notes section in AWS docs only talks about Content-MD5

[tustvold]

Makes sense to me 👍

[trueleo]

I was considering how to go about fixing this. The main objective is to make the request work with object locking, so I tried sending the x-amz-checksum-sha256 header, and it worked. For x-amz-checksum-sha256, the hash digest is encoded with base64. This can be quickly added, but internally, it seems that the object_store already calculates the sha256 digest for the request body (which is later set as x-amz-content-sha256 and encoded using hex-encode).

I have two things in mind:

• Should Content-MD5 be added as an option just for the sake of support, or should x-amz-checksum- be favored ?
  reference: https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html
• If so, can the same digest be used for x-amz-content-sha256 and x-amz-checksum-sha256? ( Still testing this out for multipart upload )

[tustvold]

Should Content-MD5 be added as an option just for the sake of support,

I don't feel strongly about this, provided it is configurable. There doesn't seem to be a huge amount of standardisation across the stores w.r.t how checksums are handled, and so it is probably we will end up with store specific logic for this.

If so, can the same digest be used for x-amz-content-sha256 and x-amz-checksum-sha256

I believe so, but should probably verify this

Edit: on a related note I filed - #3737

[trueleo]

For checksum i think something like with_checksum_algorithm(checksum::Sha256) in AmazonS3Builder makes sense. This will set x-amz-checksum- header. There are 4 of algorithms supported by s3 for integrity check CRC32, CRC32C, SHA-1, SHA-256. To use any of these one first will add them as a feature in their Cargo.toml, only then they can specify this option in the builder.