Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to perform multipart upload after upgrade: current throughput: 0 B/s is below minimum: 1 B/s #1146

Closed
DimanNe opened this issue May 13, 2024 · 6 comments
Labels
bug This issue is a bug. closed-for-staleness p1 This is a high priority issue response-requested Waiting on additional info and feedback. Will move to 'closing-soon' in 7 days.

Comments

@DimanNe
Copy link

DimanNe commented May 13, 2024

Describe the bug

After this upgrade:

aws-config   1.3.0 -> 1.4.0
aws-sdk-s3   1.25.0 -> 1.28.0
aws-smithy-types 1.1.8 -> 1.1.9

multi-part upload started to fail with the following messages in logs:

[2024-05-13T22:40:02.459500Z DEBUG aws_smithy_runtime::client::http::body::minimum_throughput] current throughput: 0 B/s is below minimum: 1 B/s
[2024-05-13T22:40:02.459541Z DEBUG aws_smithy_runtime::client::http::body::minimum_throughput] current throughput: 0 B/s is below minimum: 1 B/s
[2024-05-13T22:40:02.461266Z DEBUG hyper::proto::h1::dispatch] error writing: Connection reset by peer (os error 104)

[2024-05-13T22:40:02.661296Z DEBUG aws_smithy_runtime::client::http::body::minimum_throughput] current throughput: 0 B/s is below minimum: 1 B/s
[2024-05-13T22:40:02.661314Z DEBUG aws_smithy_runtime::client::http::body::minimum_throughput] current throughput: 0 B/s is below minimum: 1 B/s
[2024-05-13T22:40:02.664376Z DEBUG aws_smithy_runtime::client::http::body::minimum_throughput] current throughput: 0 B/s is below minimum: 1 B/s
[2024-05-13T22:40:02.664381Z DEBUG aws_smithy_runtime::client::http::body::minimum_throughput] grace period ended; timing out request
[2024-05-13T22:40:02.664390Z DEBUG aws_smithy_runtime::client::orchestrator] encountered orchestrator error; halting
[2024-05-13T22:40:02.664394Z DEBUG tracing::span] finally_attempt;
[2024-05-13T22:40:02.664409Z DEBUG aws_smithy_runtime::client::retries::strategy::standard] not retrying because we are out of attempts attempts=3 max_attempts=3
[2024-05-13T22:40:02.664414Z DEBUG aws_smithy_runtime::client::orchestrator] a retry is either unnecessary or not possible, exiting attempt loop
[2024-05-13T22:40:02.664417Z DEBUG tracing::span] finally_op;

Same code works perfectly fine with the older version.

Expected Behavior

see above

Current Behavior

see above

Reproduction Steps

see above

Possible Solution

No response

Additional Information/Context

No response

Version

├── aws-config v1.4.0
│   ├── aws-credential-types v1.2.0
│   │   ├── aws-smithy-async v1.2.1
│   │   ├── aws-smithy-runtime-api v1.6.0
│   │   │   ├── aws-smithy-async v1.2.1 (*)
│   │   │   ├── aws-smithy-types v1.1.9
│   │   ├── aws-smithy-types v1.1.9 (*)
│   ├── aws-runtime v1.2.2
│   │   ├── aws-credential-types v1.2.0 (*)
│   │   ├── aws-sigv4 v1.2.1
│   │   │   ├── aws-credential-types v1.2.0 (*)
│   │   │   ├── aws-smithy-eventstream v0.60.4
│   │   │   │   ├── aws-smithy-types v1.1.9 (*)
│   │   │   ├── aws-smithy-http v0.60.8
│   │   │   │   ├── aws-smithy-eventstream v0.60.4 (*)
│   │   │   │   ├── aws-smithy-runtime-api v1.6.0 (*)
│   │   │   │   ├── aws-smithy-types v1.1.9 (*)
│   │   │   ├── aws-smithy-runtime-api v1.6.0 (*)
│   │   │   ├── aws-smithy-types v1.1.9 (*)
│   │   ├── aws-smithy-async v1.2.1 (*)
│   │   ├── aws-smithy-eventstream v0.60.4 (*)
│   │   ├── aws-smithy-http v0.60.8 (*)
│   │   ├── aws-smithy-runtime-api v1.6.0 (*)
│   │   ├── aws-smithy-types v1.1.9 (*)
│   │   ├── aws-types v1.2.1
│   │   │   ├── aws-credential-types v1.2.0 (*)
│   │   │   ├── aws-smithy-async v1.2.1 (*)
│   │   │   ├── aws-smithy-runtime-api v1.6.0 (*)
│   │   │   ├── aws-smithy-types v1.1.9 (*)
│   ├── aws-sdk-sso v1.24.0
│   │   ├── aws-credential-types v1.2.0 (*)
│   │   ├── aws-runtime v1.2.2 (*)
│   │   ├── aws-smithy-async v1.2.1 (*)
│   │   ├── aws-smithy-http v0.60.8 (*)
│   │   ├── aws-smithy-json v0.60.7
│   │   │   └── aws-smithy-types v1.1.9 (*)
│   │   ├── aws-smithy-runtime v1.5.0
│   │   │   ├── aws-smithy-async v1.2.1 (*)
│   │   │   ├── aws-smithy-http v0.60.8 (*)
│   │   │   ├── aws-smithy-runtime-api v1.6.0 (*)
│   │   │   ├── aws-smithy-types v1.1.9 (*)
│   │   ├── aws-smithy-runtime-api v1.6.0 (*)
│   │   ├── aws-smithy-types v1.1.9 (*)
│   │   ├── aws-types v1.2.1 (*)
│   ├── aws-sdk-ssooidc v1.25.0
│   │   ├── aws-credential-types v1.2.0 (*)
│   │   ├── aws-runtime v1.2.2 (*)
│   │   ├── aws-smithy-async v1.2.1 (*)
│   │   ├── aws-smithy-http v0.60.8 (*)
│   │   ├── aws-smithy-json v0.60.7 (*)
│   │   ├── aws-smithy-runtime v1.5.0 (*)
│   │   ├── aws-smithy-runtime-api v1.6.0 (*)
│   │   ├── aws-smithy-types v1.1.9 (*)
│   │   ├── aws-types v1.2.1 (*)
│   ├── aws-sdk-sts v1.24.0
│   │   ├── aws-credential-types v1.2.0 (*)
│   │   ├── aws-runtime v1.2.2 (*)
│   │   ├── aws-smithy-async v1.2.1 (*)
│   │   ├── aws-smithy-http v0.60.8 (*)
│   │   ├── aws-smithy-json v0.60.7 (*)
│   │   ├── aws-smithy-query v0.60.7
│   │   │   ├── aws-smithy-types v1.1.9 (*)
│   │   ├── aws-smithy-runtime v1.5.0 (*)
│   │   ├── aws-smithy-runtime-api v1.6.0 (*)
│   │   ├── aws-smithy-types v1.1.9 (*)
│   │   ├── aws-smithy-xml v0.60.8
│   │   ├── aws-types v1.2.1 (*)
│   ├── aws-smithy-async v1.2.1 (*)
│   ├── aws-smithy-http v0.60.8 (*)
│   ├── aws-smithy-json v0.60.7 (*)
│   ├── aws-smithy-runtime v1.5.0 (*)
│   ├── aws-smithy-runtime-api v1.6.0 (*)
│   ├── aws-smithy-types v1.1.9 (*)
│   ├── aws-types v1.2.1 (*)
├── aws-sdk-s3 v1.28.0
│   ├── aws-credential-types v1.2.0 (*)
│   ├── aws-runtime v1.2.2 (*)
│   ├── aws-sigv4 v1.2.1 (*)
│   ├── aws-smithy-async v1.2.1 (*)
│   ├── aws-smithy-checksums v0.60.7
│   │   ├── aws-smithy-http v0.60.8 (*)
│   │   ├── aws-smithy-types v1.1.9 (*)
│   ├── aws-smithy-eventstream v0.60.4 (*)
│   ├── aws-smithy-http v0.60.8 (*)
│   ├── aws-smithy-json v0.60.7 (*)
│   ├── aws-smithy-runtime v1.5.0 (*)
│   ├── aws-smithy-runtime-api v1.6.0 (*)
│   ├── aws-smithy-types v1.1.9 (*)
│   ├── aws-smithy-xml v0.60.8 (*)
│   ├── aws-types v1.2.1 (*)
├── aws-smithy-types v1.1.9 (*)

Environment details (OS name and version, etc.)

Linux

Logs

No response

@DimanNe DimanNe added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels May 13, 2024
@Smotrov
Copy link

Smotrov commented May 14, 2024

Same error with SQS

@aajtodd aajtodd added p1 This is a high priority issue and removed needs-triage This issue or PR still needs to be triaged. labels May 14, 2024
@aajtodd
Copy link
Contributor

aajtodd commented May 14, 2024

Thanks for the issue, we are looking into this now and how best to address it.


As a workaround you can disable upload throughput protection:

    let config = aws_config::defaults(aws_config::BehaviorVersion::latest())
        .stalled_stream_protection(
            StalledStreamProtectionConfig::enabled()
                .upload_enabled(false)
                .build(),
        )
        .load()
        .await;

This will keep stalled stream protection for downloads enabled and only disable uploads (which was the default prior to BehaviorVersion::v2024_03_28()). If you want to disable it completely for uploads and downloads:

    let config = aws_config::defaults(aws_config::BehaviorVersion::latest())
        .stalled_stream_protection(StalledStreamProtectionConfig::disabled())
        .load()
        .await;

github-merge-queue bot pushed a commit to smithy-lang/smithy-rs that referenced this issue May 17, 2024
…lete (#3644)

## Motivation and Context
<!--- Why is this change required? What problem does it solve? -->
<!--- If it fixes an open issue, please link to the issue here -->
* awslabs/aws-sdk-rust#1141
* awslabs/aws-sdk-rust#1146
* awslabs/aws-sdk-rust#1148


## Description
* Disables stalled stream upload protection for requests with an
empty/zero length body.
* Disables stalled stream upload throughput checking once the request
body has been read and handed off to the HTTP layer.


## Testing
Additional integration tests added covering empty bodies and completed
uploads.

Tested SQS issue against latest runtime and can see it works now. The S3
`CopyObject` issue is related to downloads and will need a different
solution.

## Checklist
<!--- If a checkbox below is not applicable, then please DELETE it
rather than leaving it unchecked -->
- [x] I have updated `CHANGELOG.next.toml` if I made changes to the
smithy-rs codegen or runtime crates
- [x] I have updated `CHANGELOG.next.toml` if I made changes to the AWS
SDK, generated SDK code, or SDK runtime crates

----

_By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice._

---------

Co-authored-by: Zelda Hessler <[email protected]>
Co-authored-by: ysaito1001 <[email protected]>
@Velfi
Copy link
Contributor

Velfi commented May 21, 2024

We just released the runtime crate updates: https://github.com/smithy-lang/smithy-rs/releases/tag/release-2024-05-21
Once a release for the SDK crates runs later today (in about two hours) they'll include this fix.

@ysaito1001
Copy link
Collaborator

Let us know whether today's release has improved the behavior of your use case.

@Velfi Velfi added the response-requested Waiting on additional info and feedback. Will move to 'closing-soon' in 7 days. label May 24, 2024
Copy link

github-actions bot commented Jun 1, 2024

Greetings! It looks like this issue hasn’t been active in longer than a week. We encourage you to check if this is still an issue in the latest release. Because it has been longer than a week since the last update on this, and in the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or add an upvote to prevent automatic closure, or if the issue is already closed, please feel free to open a new one.

@github-actions github-actions bot added closing-soon This issue will automatically close in 4 days unless further comments are made. closed-for-staleness and removed closing-soon This issue will automatically close in 4 days unless further comments are made. labels Jun 1, 2024
@github-actions github-actions bot closed this as completed Jun 5, 2024
@kolupaev
Copy link

Hi Folks, unfortunately latest update didn't resolve the issue. Large multi-part uploads still fail in the same way on the slow-ish internet:


RUST_LOG=debug cargo run --  -o text -i ~/Downloads/not_so_humongous_video.mp4

2024-08-16T16:00:21Z DEBUG hyper::proto::h1::io] flushed 4096 bytes
[2024-08-16T16:00:21Z DEBUG hyper::proto::h1::io] flushed 4096 bytes
[2024-08-16T16:00:21Z DEBUG hyper::proto::h1::io] flushed 4096 bytes
[2024-08-16T16:00:21Z DEBUG hyper::proto::h1::io] flushed 4096 bytes
[2024-08-16T16:00:21Z DEBUG hyper::proto::h1::io] flushed 4096 bytes
[2024-08-16T16:00:21Z DEBUG hyper::proto::h1::io] flushed 3732 bytes
⠦ Using bucket region us-east-1[2024-08-16T16:00:21Z DEBUG aws_smithy_runtime::client::http::body::minimum_throughput] current throughput: 0 B/s is below minimum: 1 B/s
[2024-08-16T16:00:21Z DEBUG aws_smithy_runtime::client::http::body::minimum_throughput] current throughput: 0 B/s is below minimum: 1 B/s
⠖ Using bucket region us-east-1[2024-08-16T16:00:21Z DEBUG aws_smithy_runtime::client::http::body::minimum_throughput] current throughput: 0 B/s is below minimum: 1 B/s
[2024-08-16T16:00:21Z DEBUG aws_smithy_runtime::client::http::body::minimum_throughput] current throughput: 0 B/s is below minimum: 1 B/s
⠒ Using bucket region us-east-1[2024-08-16T16:00:21Z DEBUG aws_smithy_runtime::client::http::body::minimum_throughput] current throughput: 0 B/s is below minimum: 1 B/s
[2024-08-16T16:00:21Z DEBUG aws_smithy_runtime::client::http::body::minimum_throughput] current throughput: 0 B/s is below minimum: 1 B/s
⠐ Using bucket region us-east-1[2024-08-16T16:00:21Z DEBUG aws_smithy_runtime::client::http::body::minimum_throughput] current throughput: 0 B/s is below minimum: 1 B/s
[2024-08-16T16:00:21Z DEBUG aws_smithy_runtime::client::http::body::minimum_throughput] grace period ended; timing out request
[2024-08-16T16:00:21Z DEBUG aws_smithy_runtime::client::orchestrator] encountered orchestrator error; halting
[2024-08-16T16:00:21Z DEBUG tracing::span] finally_attempt;
[2024-08-16T16:00:21Z DEBUG rustls::common_state] Sending warning alert CloseNotify
[2024-08-16T16:00:21Z DEBUG aws_smithy_runtime::client::retries::strategy::standard] not retrying because we are out of attempts attempts=3 max_attempts=3
[2024-08-16T16:00:21Z DEBUG aws_smithy_runtime::client::orchestrator] a retry is either unnecessary or not possible, exiting attempt loop
[2024-08-16T16:00:21Z DEBUG tracing::span] finally_op;
Error: Failed to upload to S3

Caused by:
    0: dispatch failure
    1: timeout
    2: minimum throughput was specified at 1 B/s, but throughput of 0 B/s was observed

kolupaev pushed a commit to kolupaev/distill-cli that referenced this issue Aug 16, 2024
When uploading large videos to the s3 bucket (~200mb) on a slow uplink
(2 Mb/s) a stall protection for upload kick in.

This protection incorrectly detects stalled upload and panics, even on
active uploads healthy uploads.

See awslabs/aws-sdk-rust#1146

Crate changes:

- add log and env_logger to aid debugging and root causing
- update aws-sdk to latest

Testing:

RUST_LOG=debug cargo run -- -i "~/Downloads/meeting.mp4" -o text
jbnunn pushed a commit to awslabs/distill-cli that referenced this issue Sep 3, 2024
When uploading large videos to the s3 bucket (~200mb) on a slow uplink
(2 Mb/s) a stall protection for upload kick in.

This protection incorrectly detects stalled upload and panics, even on
active uploads healthy uploads.

See awslabs/aws-sdk-rust#1146

Crate changes:

- add log and env_logger to aid debugging and root causing
- update aws-sdk to latest

Testing:

RUST_LOG=debug cargo run -- -i "~/Downloads/meeting.mp4" -o text

Co-authored-by: kolupaev <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug This issue is a bug. closed-for-staleness p1 This is a high priority issue response-requested Waiting on additional info and feedback. Will move to 'closing-soon' in 7 days.
Projects
None yet
Development

No branches or pull requests

6 participants