Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

S3Path: support extremely fast bucket-to-bucket transfers #66

Open
pjbull opened this issue Sep 3, 2020 · 0 comments
Open

S3Path: support extremely fast bucket-to-bucket transfers #66

pjbull opened this issue Sep 3, 2020 · 0 comments
Labels
enhancement New feature or request

Comments

@pjbull
Copy link
Member

pjbull commented Sep 3, 2020

We've experimented with this tool for fast bucket-to-bucket transfers. It would be cool if we could leverage the logic here to make an incredibly fast bucket-to-bucket transfer:
https://github.com/generalui/s3p/

Here's a blog post with more deets:
https://medium.com/@shanebdavis/s3p-massively-parallel-s3-copying-9a9e466d0d74

I think the core pieces are:

  • async for all of the requests
  • use prefix and assumptions about list results in alphabetical order to split up the parallel "list" calls search space? (or something like that)
  • get a massive number of requests goin' at once.

(This may also be out of scope for the package, just throwing it out there as useful for our purposes)

@pjbull pjbull added the enhancement New feature or request label Oct 2, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant