Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve limit for single s3 put uploads #28542

Merged
merged 1 commit into from
Dec 29, 2021
Merged

Conversation

juliusknorr
Copy link
Member

@juliusknorr juliusknorr commented Aug 20, 2021

Follow up to #27877 to further address the comments from @kesselb

Could we use a higher value for single put operations? Like 25 or 50 MB. I understand the buffer uses the memory hence we should not pick a value to high. However most pictures have more than 5 MB nowadays.

100 MB was picked as the default, as per AWS S3 recommendation to make use of multipart upload for files larger than that:

In general, when your object size reaches 100 MB, you should consider using multipart uploads instead of uploading the object in a single operation.

https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html

Memory overhead is the actual file size which would fit our recommended php memory limit of 512 MB still fairly easy.

Base automatically changed from enh/s3-putObject to master August 20, 2021 16:54
@juliusknorr juliusknorr marked this pull request as ready for review December 29, 2021 11:06
@juliusknorr juliusknorr added 3. to review Waiting for reviews enhancement labels Dec 29, 2021
@juliusknorr juliusknorr added this to the Nextcloud 24 milestone Dec 29, 2021
@@ -35,6 +35,7 @@
use Aws\Credentials\Credentials;
use Aws\Exception\CredentialsException;
use Aws\S3\Exception\S3Exception;
use Aws\S3\MultipartUploader;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unused, it seems

Copy link
Contributor

@artonge artonge left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some php-cs warnings, else, looks good.

@juliusknorr
Copy link
Member Author

Failure unrelated

@juliusknorr juliusknorr merged commit de2b86a into master Dec 29, 2021
@juliusknorr juliusknorr deleted the enh/s3-put-limit branch December 29, 2021 13:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3. to review Waiting for reviews enhancement
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants