Support for AWS Requester Pays buckets

[kylebarron]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

I'm trying to work with AWS requester pays buckets, but getting an error:

GenericError: Generic S3 error: Error performing list request: Client error with status 403 Forbidden: 

<?xml version="1.0" encoding="UTF-8"?>
<Error>
    <Code>AccessDenied</Code>
    <Message>There were headers present in the request which were not signed</Message>
    <HeadersNotSigned>x-amz-request-payer</HeadersNotSigned>
    <RequestId>QEA9RD5BETE6BN7E</RequestId>
    <HostId>tfgEYvVBmWEx5EZljTRXuR6S6ocChjas5qF8eXThc0yt6bRyGEJCYjCFJ+/YOvA2Q2DT/NSqBTw=</HostId>
</Error>

The following is a Python example from obstore. It should be easy to interpret what the underlying Rust code is, but I can provide a pure-Rust example if necessary

import obstore
from obstore.store import ClientOptions, S3Store
import boto3

# boto3.Session reads credentials from disk, which then passes them to AmazonS3Builder
session = boto3.Session()

# Set x-amz-request-payer as a default header for every request
options = ClientOptions(default_headers={b"x-amz-request-payer": "requester"})

# This is a bucket exposed through the AWS open data registry
# https://registry.opendata.aws/naip/
# It's public with requester-pays turned on
store = S3Store.from_session(session, bucket="naip-visualization", client_options=options)

# Make the request
result = obstore.list(store, "ny/2022/60cm/rgb/40073/").collect()

Describe the solution you'd like

Include AWS header in signature calculation. Presumably there would need to be some way to tell object_store which default headers to include in its signing mechanism?

Describe alternatives you've considered

Additional context

This AWS docs page has a reference on requester pays and request signing

The only prior discussion I can find is on discord:

Hi everyone, if anyone is familiar with the object_store implementation in arrow-rs I had a question; is there a way to pass in arbitrary AWS (specifically S3) options via HTTP headers? For example, I don't see implicit support for requester pays buckets, which is hinted at by passing in this header value pair: x-amz-request-payer : requester so I'm wondering if there's a way to add in our own options like that.

Not on a per-request basis, as it sort of breaks the abstraction of being any object store, but you could look at overriding the headers using ClientOptions when you build the AmazonS3 client

[tustvold]

I think the issue may be when the ClientOptions headers are added to the request object, it appears it may be after signing is done as part of reqwest actually performing the request. We could workaround this, or add first-party support for request payers as an option on the builder

[kylebarron]

Default headers are applied onto the underlying reqwest::Client directly, but then it's not possible to access those default headers back out of a Client. So it's not possible to use that with requester pays because you need include the header content in the signing process.

[alamb]

label_issue.py automatically added labels {'object-store'} from #6768