doc: avoid memory leaks when writable.write() return false #10631

mcollina · 2017-01-05T10:37:18Z

The doc specified that writable.write() was advisory only. However,
ignoring that value might lead to memory leaks. This PR specifies that
behavior. Moreover, it adds an example on how to listen for the 'drain'
event correctly.

Checklist

documentation is changed or added
commit message follows commit guidelines

Affected core subsystem(s)

doc, stream

cc @nodejs/streams @binki @Tanuja-Sawant @silverwind

f347dad

evanlucas · 2017-01-05T11:12:53Z

doc/api/stream.md

+In case the `false` return value is ignored, the writable stream will
+unconditionally accept and buffer `chunk` even if it has not not been allowed
+to drain. However, this might lead to memory leaks and garbage
+collection issues, especially if the writable is a [Transform]()


Should it be [Transform][]? for linking?

This may be more readable if like:

especially if the writable stream is a [Transform]() or even especially if the stream is a [Transform]()?

evanlucas · 2017-01-05T11:13:39Z

doc/api/stream.md

+unconditionally accept and buffer `chunk` even if it has not not been allowed
+to drain. However, this might lead to memory leaks and garbage
+collection issues, especially if the writable is a [Transform]()
+(which inherit from [Writable]()).


should this be inherits?

Also, same question about the linking of [Writable]() here

evanlucas · 2017-01-05T11:15:58Z

doc/api/stream.md

+  if (!stream.write(data)) {
+    stream.once('drain', cb)
+  } else {
+    cb()


should we encourage consistency of async vs sync calling of the callback? Maybe wrap in setImmediate()?

I'll wrap in a process.nextTick(), as there is no need to slow things down doing a full I/O cycle.

fair enough :]

mcollina · 2017-01-05T12:15:07Z

@evanlucas thanks, updated!

evanlucas

LGTM with a comment that should be addressed before landing. Thanks!

evanlucas · 2017-01-05T12:16:22Z

doc/api/stream.md

+collection issues, especially if the writable is a [Transform][]
+(which inherits from [Writable][]).
+
+A possible way to handle the [`'drain'``][] event to respect


the drain link has an extra backtick at the end which will break the link.

sam-github · 2017-01-05T14:21:15Z

doc/api/stream.md

+
+In case the `false` return value is ignored, the writable stream will
+unconditionally accept and buffer `chunk` even if it has not not been allowed
+to drain. However, this might lead to memory leaks and garbage


I don't think it should be called a leak, but better to describe that node has limited memory to buffer data that it is not ready to write, and will abort when it runs out of memory. Leak is vague, but more usually applies to memory your application doesn't need, but still refs. In this case, the app needs the write buffers until they get written, they aren't leaking.

Also, why is this especially true for Transform streams? If we are going to say that its more true for particular derived types of Writable, we should say why.

Also, why is this especially true for Transform streams? If we are going to say that its more true for particular derived types of Writable, we should say why.

A Writable in Node core is always writing thing to the underlining resource, e.g. the socket. You can stop that with cork(), but by default it flows the data. Transform things are paused by default until they are piped or an 'data' or 'readable' event handler is added.
So, ignoring when write() returns false will cause the data to be buffered until the stream is flowing. If that does not happen, you have a memory leak.

Should I add the above to the PR?

This is what I was trying to say in the conversation that sparked this PR. If your code leaks when write() returns false, it is still leaking when write() returns true. And it is the stream being written to that is doing the leaking. If write() returns false and never drains itself or you call end() after write() returns true and it never drains its buffer, that’s the same sort of leak (though if the target stream treats end() specially, you might only see the leak happen when the target stream fails to drain after write() returns false).

@binki under your definition, a Transform stream that is not piped is a memory leak, which is the point I am trying to make.

I don't think it should be called a leak, but better to describe that node has limited memory to buffer data that it is not ready to write, and will abort when it runs out of memory. Leak is vague, but more usually applies to memory your application doesn't need, but still refs. In this case, the app needs the write buffers until they get written, they aren't leaking.

It is a memory leak in behavior, but if it's a contested term I'll remove it. As long as the docs is clear that you are likely to get aborted by out of memory if you don't handle drain, then I'm good.

@mcollina I think the info on the Transform stream you put above is a subtle gotcha, well worth documenting. It should be mentioned here, and maybe in the docs for Transform as well? This spreading around is why a guide would be nice.

BTW, another behavior that usually happens in this case is a big RSS with low heap usage. A single stream ballooned the memory usage of the process, but in fact all of that is free. It has been collected, but the process is left big. Most people will flag this behavior as a memory leak, even if it is not in the strict definition of a chunk of memory that is left allocated but it is not referenced by anybody.

How about we define this as a "memory handling issues" or "memory ballooning"?

sam-github · 2017-01-05T14:25:42Z

Its out of scope of this PR, but node needs a guide on "Backpressure", describing the problems it solves (like indefinite buffering causing memory exhaustion and excessive drag on the GC), and what patterns must be respected for an app to utilize it (streams, avoiding use of .write(), and if you are using .write(), respecting its return value). This would be better than mini tutorials spread the through the docs, IMO.

mcollina · 2017-01-05T14:56:52Z

@sam-github I think we should have a topic/guide/whatever on streams. It is probably a big chunk of work on its own.
At the moment that is out of scope, but I will be very happy to discuss.

mcollina · 2017-01-06T11:45:52Z

@sam-github @binki I have updated the PR, let me know.

binki

I think wait for 'drain' instead of listen for 'drain' would be much clearer, my other comments can be ignored.

binki · 2017-01-06T15:10:16Z

doc/api/stream.md

+In case the `false` return value is ignored, the writable stream will
+unconditionally accept and buffer `chunk` even if it has not not been allowed
+to drain. However, Node has a limited amount of memory, and not
+listening to `'drain'` might cause the process to abort by an out of


Much better!

However, I think it would be clearer if you replace listening to `'drain'` with waiting for `'drain'` .

@mcollina I would like to suggest this alternate text to replace from If false is returned to such as high RSS.:

While a stream is not draining, calls to write() will buffer chunk, and return false. Once all currently buffered chunks are drained (accepted for delivery by the operating system), the 'drain' event will be emitted. It is recommended that once write() returns false, no more chunks be written until the 'drain' event is emitted. While calling write() on a stream that is not draining is allowed, Node.js will buffer all written chunks until maximum memory usage occurs, at which point it will abort unconditionally. Even before it aborts, high memory usage will cause poor garbage collector performance and high RSS (which is not typically released back to the system, even after the memory is no longer required). Since TCP sockets may never drain if the remote peer does not read the data, writing a socket that is not draining may lead to a remotely exploitable vulnerability.

binki · 2017-01-06T15:12:30Z

doc/api/stream.md

+because the `Transform` streams are paused by default until
+they are piped or an `'data'` or `'readable'` event handler is added.
+So, ignoring when `write()` returns false will cause the data to be buffered
+until the stream is flowing, causing a memory leak.


I suppose Transform streams cannot flow until they are connected to consumers, but the same issues exist even for writable FS/network streams because you never know when they’ll start draining. Especially if the other end is a TCP server, the TCP server could just never read anything from its end and cause the exact same problem once the OS-level sockets buffers fill up. So I’m not convinced this is actually a Transform-specific thing, but I think I’m alone with that opinion. Also, it’s not a memory leak unless you know for certain that the Transform stream will never be piped to something else in the future. E.g., you could be buffering an entire file in-memory on purpose by using a Transform stream with the identity transform ;-).

I don’t understand the RSS issue, though, does expanding a Buffer and then letting it get GCed at a later point really cause node to allocate memory without ever freeing it? Again, my concerns are probably dismissable.

Also, it’s not a memory leak unless you know for certain that the Transform stream will never be piped to something else in the future. E.g., you could be buffering an entire file in-memory on purpose by using a Transform stream with the identity transform ;-).

Even if it can be done, it does not mean it should be done. That type of use causes a huge lot of problems. I've amended things slightly, clarifying more.

I don’t understand the RSS issue, though, does expanding a Buffer and then letting it get GCed at a later point really cause node to allocate memory without ever freeing it? Again, my concerns are probably dismissable.

Buffers cannot be enlarged, but the heap on which are stored can. No, after it grows it does not go back.

As I understand the heap does shrink when the memory pressure goes down, but it would be delayed as shrinking the heap(usually by compacting the old space) would stop-the-world, so under some kind of workload it might never find a good time to shrink.

EDIT: as I understand Buffer's backing store memory are freed when the weak callbacks are called, and that would be delayed too.

@joyeecheung in my experience in debugging those issues in production, the shrinking does not happen in a reasonable timeframe, i.e. before causing other issues. In those cases memory occupation is stable at around 900MB-1GB.

@mcollina Yes that's what I've observed too. Usually this would not lead to an OOM because there is always the "last resort gc" that would try super hard to clean up everything.

...but that's kinda off topic for this PR. If I am being too talky, just ignore me :D

binki · 2017-01-06T15:22:45Z

doc/api/stream.md

+    process.nextTick(cb)
+  }
+}
+```


So what you’re suggesting is to actually write code as if you should always wait for drain. This function is just a utility to emulate a drain happening even when one won’t happen by providing an asynchronous alternative to write() which completes when stream is no longer trying to exert backpressure. I think a short explanation of how this solution only works if you serialize your write() calls to wait for the cb to be called would be helpful to clarify how this is meant to help. E.g., to respect backpressure is to use the following. You must always wait for cbto be called before calling thiswrite() wrapper again to respect backpressure. Hmm, maybe that just makes it noisier.

This suggestion does simplify the code needed to respect 'drain' by moving the part dealing with write()’s return value behind an “always wait for target to be ready” API. But it would still, depending on where the write() calls are coming from, require significant changes in the code that was originally calling write() to use correctly. But if the code you start with is just a series of calls to write() this would be straightforward enough and is rather nifty.

I think that a small example is worth more than words, but that's me. Is the example clear enough in the context of this document, or is not and it needs some more words nearby?

@mcollina It might be clear enough if people know that they have to wait for cb to be called before calling it again. I’m just not sure if that’s obvious, it wasn’t to me for some reason. Maybe an example of how to call it would have made it clearer to me more quickly.

I've added an example and a comment.

mcollina · 2017-01-09T08:00:11Z

@sam-github are you ok? may I merge?

sam-github · 2017-01-09T13:55:44Z

doc/api/stream.md

+memory error, and other types of memory and garbage collection
+issues, such as high RSS.
+
+The `'drain'` event must be handled if the writable is a [Transform][],


Its not true that it must be handled, can you rephrase as "Writing data while the stream is not draining is particularly problematic for a [Transform][], ..."

sam-github · 2017-01-09T13:58:54Z

doc/api/stream.md

+  } else {
+    process.nextTick(cb)
+  }
+}


Is it worth mentioning that code like the above is a streams anti-pattern? If I saw code like this, I would suggest that the source of the data should be restructured as a "readable stream", and that the readable stream should then be piped to the destination. Even if its not an anti-pattenr, I think its worth at least mentioning that the above code is not necessary when piping streams together, and that restructuring your app as piped streams allows back pressure to be more elegantly handled by node directly.

I disagree that it is an anti-pattern. The major difference is speed, this runs at 2-3x compared to a Readable. However, I agree that applications should built around pipe() rather than write() 100%.

here is the problem: if someone does 3-4 writes without checking 'drain' is never an issue. However, if them wants to write some megabytes of data, then doing this pattern might be better.

Fair enough, and that experience with streams is really valuable. I think you are (quite!) understandably trying to avoid writing a guide on streams and backpressure, while still providing some useful guidance inline in the docs, but I'm concerned that without a little more context people will paste your code into their app, and just move the leak around. No pressure to do a full-scale guide, :-), I'm not -1 on the PR, its an incremental improvement.

sam-github · 2017-01-09T13:59:30Z

doc/api/stream.md

+  }
+}
+
+// wait for cb to be called before doing


Capitalize sentence, end it with a period, and wrap to 80 columns.

sam-github · 2017-01-09T14:06:32Z

doc/api/stream.md

+// wait for cb to be called before doing
+// any other write
+write('hello', () => {
+  write('world', () => {


Now, instead of the string 'world' being buffered in the stream, it is being buffered in a closure, pushing the memory leak out of the stream and closer to whatever process is producing the data to be written. Anybody who understand backpressure will understand that, but they wouldn't need the example in the first place. In the absence of a guide on backpressure, though, I guess this the best we can do, unless @mcollina you can think of some text to give guidance on when this pattern is reasonable to use, as opposed to using streams?

An example of something that could be backpressured safely using this pattern is anything that can be calculated on demand. E.g., a sequence of numbers, consuming a generator, or producing x zeroed out bytes. You could close over the necessary state. But these all would also be better wrapped as Readable streams anyway as suggested above.

I added the example because it was asked for in a review. I am happy in removing it.

mcollina · 2017-01-10T15:39:47Z

@sam-github I have made some changes, please review.

mcollina · 2017-01-11T09:37:31Z

@sam-github I made more edits. It should be good now, and much more clear than before.

The doc specified that writable.write() was advisory only. However, ignoring that value might lead to memory leaks. This PR specifies that behavior. Moreover, it adds an example on how to listen for the 'drain' event correctly. See: nodejs@f347dad

mcollina · 2017-01-11T16:28:52Z

Landed in e9044c8.

The doc specified that writable.write() was advisory only. However, ignoring that value might lead to memory leaks. This PR specifies that behavior. Moreover, it adds an example on how to listen for the 'drain' event correctly. See: f347dad PR-URL: #10631 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Sam Roberts <[email protected]> Reviewed-By: Evan Lucas <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Joyee Cheung <[email protected]>

The doc specified that writable.write() was advisory only. However, ignoring that value might lead to memory leaks. This PR specifies that behavior. Moreover, it adds an example on how to listen for the 'drain' event correctly. See: nodejs@f347dad PR-URL: nodejs#10631 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Sam Roberts <[email protected]> Reviewed-By: Evan Lucas <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Joyee Cheung <[email protected]>

The doc specified that writable.write() was advisory only. However, ignoring that value might lead to memory leaks. This PR specifies that behavior. Moreover, it adds an example on how to listen for the 'drain' event correctly. See: f347dad PR-URL: #10631 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Sam Roberts <[email protected]> Reviewed-By: Evan Lucas <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Joyee Cheung <[email protected]>

MylesBorins · 2017-03-07T01:29:20Z

This did not land cleanly on v4.x. Please feel free to manually backport

mcollina · 2017-03-07T15:41:36Z

Do we really want this on v4? I'm happy to backport if it's needed.
There might be some in-between commits that we might want to merge as well.

jasnell · 2017-03-07T20:54:58Z

I've no problem with it going into v4 unless the other changes are disruptive or it's just too much work

The doc specified that writable.write() was advisory only. However, ignoring that value might lead to memory leaks. This PR specifies that behavior. Moreover, it adds an example on how to listen for the 'drain' event correctly. See: f347dad PR-URL: #10631 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Sam Roberts <[email protected]> Reviewed-By: Evan Lucas <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Joyee Cheung <[email protected]>

The doc specified that writable.write() was advisory only. However, ignoring that value might lead to memory leaks. This PR specifies that behavior. Moreover, it adds an example on how to listen for the 'drain' event correctly. See: nodejs@f347dad PR-URL: nodejs#10631 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Sam Roberts <[email protected]> Reviewed-By: Evan Lucas <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Joyee Cheung <[email protected]>

mcollina · 2017-03-13T10:42:23Z

Backported to v4 in #11824.

mcollina added doc Issues and PRs related to the documentations. stream Issues and PRs related to the stream subsystem. labels Jan 5, 2017

nodejs-github-bot added doc Issues and PRs related to the documentations. stream Issues and PRs related to the stream subsystem. labels Jan 5, 2017

mcollina force-pushed the fix-write-advisory branch from 448fb65 to 9aababd Compare January 5, 2017 10:38

evanlucas reviewed Jan 5, 2017

View reviewed changes

mcollina force-pushed the fix-write-advisory branch from 9aababd to 7f43650 Compare January 5, 2017 12:15

evanlucas approved these changes Jan 5, 2017

View reviewed changes

mcollina force-pushed the fix-write-advisory branch from 7f43650 to f03a56e Compare January 5, 2017 12:19

cjihrig approved these changes Jan 5, 2017

View reviewed changes

sam-github suggested changes Jan 5, 2017

View reviewed changes

jasnell approved these changes Jan 5, 2017

View reviewed changes

mcollina force-pushed the fix-write-advisory branch from f03a56e to 8348219 Compare January 6, 2017 11:43

binki suggested changes Jan 6, 2017

View reviewed changes

mcollina force-pushed the fix-write-advisory branch 2 times, most recently from 906df5f to 8f7ec07 Compare January 7, 2017 10:25

sam-github reviewed Jan 9, 2017

View reviewed changes

sam-github mentioned this pull request Jan 9, 2017

doc: general improvements to net.md copy #7137

Closed

2 tasks

mcollina force-pushed the fix-write-advisory branch 3 times, most recently from 5ca64e1 to e8d4dd6 Compare January 10, 2017 15:39

mcollina force-pushed the fix-write-advisory branch from e8d4dd6 to 46ea4db Compare January 11, 2017 09:36

sam-github approved these changes Jan 11, 2017

View reviewed changes

mcollina force-pushed the fix-write-advisory branch from 46ea4db to e354d5c Compare January 11, 2017 15:55

binki approved these changes Jan 11, 2017

View reviewed changes

mcollina closed this Jan 11, 2017

mcollina mentioned this pull request Jan 12, 2017

doc: guide on backpressure #10766

Closed

italoacasas mentioned this pull request Jan 25, 2017

Proposal to update v7.x-staging #10989

Closed

italoacasas mentioned this pull request Jan 29, 2017

v7.5.0 proposal #11062

Merged

MylesBorins added dont-land-on-v4.x labels Mar 7, 2017

mcollina deleted the fix-write-advisory branch March 8, 2017 18:08

MylesBorins mentioned this pull request Mar 9, 2017

v6.10.1 proposal #11759

Merged

mcollina mentioned this pull request Mar 13, 2017

doc: handle backpressure when write() return false #11824

Closed

doc: avoid memory leaks when writable.write() return false #10631

doc: avoid memory leaks when writable.write() return false #10631

Conversation

mcollina commented Jan 5, 2017 • edited Loading

Checklist

Affected core subsystem(s)

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mcollina commented Jan 5, 2017

evanlucas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sam-github commented Jan 5, 2017

mcollina commented Jan 5, 2017

mcollina commented Jan 6, 2017

binki left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joyeecheung Jan 7, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mcollina commented Jan 9, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mcollina commented Jan 10, 2017

mcollina commented Jan 11, 2017

mcollina commented Jan 11, 2017

MylesBorins commented Mar 7, 2017

mcollina commented Mar 7, 2017

jasnell commented Mar 7, 2017

mcollina commented Mar 13, 2017

mcollina commented Jan 5, 2017 •

edited

Loading

joyeecheung Jan 7, 2017 •

edited

Loading