S3 Upload API fails with ERR_STREAM_WRITE_AFTER_END

[juanfrnz]

Confirm by changing [ ] to [x] below to ensure that it's a bug:

• I've gone through Developer Guide and API reference
• I've checked AWS Forums and StackOverflow for answers
• I've searched for previous similar issues and didn't find any solution

Describe the bug
Uploading large files of a few megabytes of size using the upload aws-sdk s3 API throws UnhandledPromiseRejectionWarning with ERR_STREAM_WRITE_AFTER_END and

Is the issue in the browser/Node.js?
Node.js

Details of the browser/Node.js version
Node.js v14.8.0

SDK version number
[email protected]

To Reproduce (observed behavior)
Upload a file of 2MB passing Body param with type ReadableStream.

const s3 = new aws.S3();
const Body = new PassThrough();

const params = {
      Bucket,
      Key,
      Body
};
s3.upload(params);
Body.pipe(stream);

Exception thrown:

(node:8) UnhandledPromiseRejectionWarning: Error [ERR_STREAM_WRITE_AFTER_END]: write after end
    at writeAfterEnd (_http_outgoing.js:643:15)
    at ClientRequest.end (_http_outgoing.js:763:7)
    at features.constructor.writeBody (/home/node/app/node_modules/aws-sdk/lib/http/node.js:137:14)
    at ClientRequest.<anonymous> (/home/node/app/node_modules/aws-sdk/lib/http/node.js:102:14)
    at ClientRequest.emit (events.js:315:20)
    at ClientRequest.EventEmitter.emit (domain.js:485:12)
    at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:605:11)
    at HTTPParser.parserOnHeadersComplete (_http_common.js:117:17)
    at TLSSocket.socketOnData (_http_client.js:507:22)
    at TLSSocket.emit (events.js:315:20)
    at TLSSocket.EventEmitter.emit (domain.js:485:12)
    at addChunk (_stream_readable.js:302:12)
    at readableAddChunk (_stream_readable.js:278:9)
    at TLSSocket.Readable.push (_stream_readable.js:217:10)
    at TLSWrap.onStreamRead (internal/stream_base_commons.js:186:23)``

Additional context

I think the error is easy to be tracked down to the cases where long files upload triggers 100-Continue what executes https://github.com/aws/aws-sdk-js/blob/master/lib/http/node.js#L100 and https://github.com/aws/aws-sdk-js/blob/master/lib/http/node.js#L137 ending the stream before write ends.

[ajredniwja]

Hey @juanantoniofj thanks for bringing this up, I was not able to reproduce this and I used a fairly large file.

The code I used:

var aws = require('aws-sdk');
var fs = require('fs');

const s3 = new aws.S3();

const params = {
    Bucket: 'xyz',
    Key: 'abc',
    Body: fs.createReadStream('video.mov')
};

s3.upload(params, function (err, data) {
    if (err) console.log(err);
});

I believe this is more related to how the code is written. Please feel free to reach out if you have more inputs.

[github-actions]

This issue has not received a response in 1 week. If you still think there is a problem, please leave a comment to avoid the issue from automatically closing.

[gramakri]

I can consistently reproduce this with node v14.15.4 and Linode Object Storage. Curiously, I can upload fine to other object storages just fine (DO, S3, Minio to name a few). I am using [email protected]. The back trace is similar:

s3 upload error. Error [ERR_STREAM_WRITE_AFTER_END]: write after end
at writeAfterEnd (_http_outgoing.js:668:15)
at ClientRequest.end (_http_outgoing.js:788:7)
at features.constructor.writeBody (/home/yellowtent/box/node_modules/aws-sdk/lib/http/node.js:137:14)
at ClientRequest.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/http/node.js:102:14)
at ClientRequest.emit (events.js:315:20)
at ClientRequest.EventEmitter.emit (domain.js:467:12)
at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:606:11)
at HTTPParser.parserOnHeadersComplete (_http_common.js:126:17)
at TLSSocket.socketOnData (_http_client.js:509:22)
at TLSSocket.emit (events.js:315:20) {
code: 'NetworkingError',
region: 'us-east-1',
hostname: 'xxxx.us-east-1.linodeobjects.com',
retryable: true,
time: 2021-03-14T01:08:00.199Z

[gramakri]

I tried to make a minimal example and it seems I can upload simple files just fine. In my code, it's the equivalent of "tar cvf - | s3_uploader". Looks like something with the source stream causes this issue. I will continue to try to make a minimal example.

[juanfrnz]

Yes, the error is related to node v14. S3 relies on HTTP 100-continue for large files and aws-sdk is ending the pipe before it has finished. Probably not the cleanest solution but I was able to fix this as follow:

diff --git a/lib/http/node.js b/lib/http/node.js
index be69f691..d04fa7cc 100644
--- a/lib/http/node.js
+++ b/lib/http/node.js
@@ -99,7 +99,7 @@ AWS.NodeHttpClient = AWS.util.inherit({
     var expect = httpRequest.headers.Expect || httpRequest.headers.expect;
     if (expect === '100-continue') {
       stream.on('continue', function() {
-        self.writeBody(stream, httpRequest);
+        self.writeBody(stream, httpRequest, true);
       });
     } else {
       this.writeBody(stream, httpRequest);
@@ -108,7 +108,7 @@ AWS.NodeHttpClient = AWS.util.inherit({
     return stream;
   },

-  writeBody: function writeBody(stream, httpRequest) {
+  writeBody: function writeBody(stream, httpRequest, isContinue = false) {
     var body = httpRequest.body;
     var totalBytes = parseInt(httpRequest.headers['Content-Length'], 10);

@@ -134,7 +134,8 @@ AWS.NodeHttpClient = AWS.util.inherit({
           total: totalBytes
         });
       });
-      stream.end(body);
+      if (isContinue) stream.write(body);
+      else stream.end(body);
     } else {
       // no request body
       stream.end();

[gramakri]

@juanantoniofj amazing! I can confirm your patch works.

Can someone from the AWS team re-open this bug and also comment on the patch? Thanks!

[gramakri]

#437 added 100-continue by default.

[gramakri]

I narrowed this problem to Linode Object Storage sending 101 CONTINUE twice. The response from linode looks like this:

HTTP/1.1 100 Continue

HTTP/1.1 100 CONTINUE
Date: Wed, 24 Mar 2021 01:31:03 GMT
Connection: keep-alive

HTTP/1.1 200 OK
Content-Length: 0
ETag: "08db8030b2f1434199d74a4e8189c954"
Accept-Ranges: bytes
x-amz-request-id: tx00000000000000db14129-00605a9657-1f74dbc-default
Date: Wed, 24 Mar 2021 01:31:04 GMT
Connection: close

This causes the code here to pipe the body twice and this results in an error. I have also left a note at https://www.linode.com/community/questions/21180/object-storage-upload-err_stream_write_after_end-write-after-end#answer-75978 . I don't know if the above is correct HTTP or not.

[ajredniwja]

I have reopened the issue, sorry for the delay, but is this still a problem?

[gramakri]

@ajredniwja thanks for re-opening. Yes, this is this a problem and easily reproduced with Linode Object Storage. I have submitted a PR at #3674 . Linode servers respond like below to Expect: 100-continue. I haven't been able to figure out if the response below is valid HTTP since it sends 100 continue twice. But it seems node.js atleast parses it correctly, so it's probably valid.

HTTP/1.1 100 Continue

HTTP/1.1 100 CONTINUE
Date: Wed, 24 Mar 2021 01:31:03 GMT
Connection: keep-alive

HTTP/1.1 200 OK
Content-Length: 0
ETag: "08db8030b2f1434199d74a4e8189c954"
Accept-Ranges: bytes
x-amz-request-id: tx00000000000000db14129-00605a9657-1f74dbc-default
Date: Wed, 24 Mar 2021 01:31:04 GMT
Connection: close

[gramakri]

@ajredniwja this is still a problem, any thoughts on this?

[ajredniwja]

I havent reproduced it with Linode Object Storage as I am not so familiar with that. @gramakri
I will assign someone to look at the PR.

[andresribeiro]

I was having the same problem and, testing this branch from this fork, it worked perfectly

[gramakri]

@ajredniwja #3674 is merged now, so this issue can be closed.