child-process: data loss with piped stdout

[petrosagg]

• Version: v6.2.1
• Platform: Linux rachmaninoff 4.5.3-1-ARCH deps: update openssl to 1.0.1j #1 SMP PREEMPT Sat May 7 20:43:57 CEST 2016 x86_64 GNU/Linux
• Subsystem: child-process

When the stdout of a spawned process is piped into a writable stream that doesn't read fast enough, and the spawned process exits, node will put it into flowing mode and data will be lost.

I have the following minimal testcase that pipes a simple child process to a PassThrough stream, and only after the process has exited the through stream is piped to node's stdout.

• Expected output: All numbers from 1 to 36000
• Actual output: On my machine it stops at around 32000

const spawn = require('child_process').spawn;
const stream = require('stream');

const through = new stream.PassThrough();

const p = spawn('seq', [ '36000' ]);

p.on('exit', function(code) {
    setImmediate(function() {
        through.pipe(process.stdout);
    });
});

p.stdout.pipe(through);

[evanlucas]

Can you try using the close event instead of exit?

[addaleax]

Sounds a lot like a duplicate of #7159, which has a fix pending in #7160

[petrosagg]

@evanlucas the close event is never emitted.

@addaleax I think it's a different issue. I just pushed a PR and testcase that fixes the issue for me

[addaleax]

@petrosagg Yeah, I see that, and I also understand why your PR fixes the issue, but I still think the underlying issue stems from the streams implementation, not from child processes. You can through the description of #7160 if you want to understand why the resume() call is causing problems.

[petrosagg]

@addaleax I just tested #7160 with my testcase and it indeed fixes the issue. Does .resume() respect piped streams? The docs say the readable stream will start emitting data events when .resume() is called so I assumed that data can still be potenialy lost.

[addaleax]

@petrosagg .pipe() essentially works by adding data listeners to the source stream, so I think that should be fine… or am I misunderstanding what you’re saying?

[petrosagg]

@addaleax I'm trying to understand how back-pressure works when one calls resume(). From what you're saying it sounds like that if you have a.pipe(b) and then call a.resume() and b can't have more data written to its buffers then b's data handler will immediately re-pause a. Is this correct?

[addaleax]

@petrosagg Well, I think that’s what supposed to happen, and #7160 would ensure that behaviour.

[addaleax]

Closing as a duplicate but feel free to keep asking if there’s anything unclear :)

[petrosagg]

@addaleax I was studying your testcase. Everything makes sense now. Thanks for the explanation :)