Cross Bucket Copy Support

[tustvold]

Draft as needs further design, and a motivating use-case

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

Currently ObjectStore only supports copying files within the current bucket, this is despite most stores having APIs for performing efficient copies across buckets within the same cloud provider.

Describe the solution you'd like

I would like some mechanism to support copying between ObjectStore, that is able to make use of the more efficient APIs where they are possible. I don't have a good idea of how to achieve this, nor a motivating use-case for this functionality.

Describe alternatives you've considered

Additional context

[Xuanwo]

AWS S3 allows buckets in the same region to utilize the CopyObject feature, though including a bucket name directly in the API may seem odd.

[tustvold]

Agreed, I don't have a good idea for how to expose this currently. One option might be to tie into the URL parsing stuff we already have, so the user would provide the source and destination as URLs that would then get processed by a free function

[Xuanwo]

One option might be to tie into the URL parsing stuff we already have, so the user would provide the source and destination as URLs that would then get processed by a free function

Seems cool to have copy("s3://bucket_a/file", "s3://bucket_b/file").

The downside is that users might need to manually construct the URL. We might also need to provide let url = store.get_url("file")?

[tustvold]

Seems cool to have copy("s3://bucket_a/file", "s3://bucket_b/file").

Yeah something like that, not entirely sure how credentials would work, but that's broadly where my head was at.

We might also need to provide let url = store.get_url("file")?

I think perhaps the expectation here is that the user is already outside the "single-bucket" model of ObjectStore, and so probably already are constructing these URLs somehow. FWIW a get_url method has been discussed before on #4162, it's non-trivial 😅