Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

only buffer the first part of a large backup, stream the rest without buffering #62

Open
sbrudenell opened this issue Aug 27, 2024 · 0 comments
Assignees
Labels
enhancement New feature or request

Comments

@sbrudenell
Copy link
Owner

currently when uploading to S3, we buffer 5 GiB chunks to disk before uploading each.

It's necessary to buffer the first part as this is the only way to make the correct choice of PutObject vs CreateMultipartUpload while minimizing API usage.

however, it's probably inefficient to do this for every part. For the second part onward, we should stream the upload directly.

@sbrudenell sbrudenell added the enhancement New feature or request label Aug 27, 2024
@sbrudenell sbrudenell self-assigned this Aug 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant