Node seems slow to reclaim RSS (sometimes partially or never) on slow clients

[zz85]

I've been looking for an explanation to why node's RSS seems slow or never to reclaim itself when after slow clients connected to the ws server gets disconnected.

The observation is that RSS builds up when the server attempts to send data faster than clients can consume. This seems logically, however when clients disconnect, the RSS stays around for a longer period. Sometimes, the memory compacts, but sometimes it doesn't free up every after a long while. v8's heap memory do not seem to be a problem.

My steps to reproduce is this. Run a ws server on a remote server. Connect 200 clients to it locally (to simulate some latency and causing buffering on the server). The server sends 512K of data every 2 seconds. v8 Heap's will remain constant but RSS will grow. Run for 30 seconds and kill clients. I'm testing this with node v4.2.1 and ws 1.0.1.

before killing clients.

after killing clients at 30s (200MB of RSS not cleanup here).

I'm not sure if this is a behaviour coming from ws implementation, nodejs itself or a combination or both. I tested this with pure tcp sockets but the node process cleanup its RSS really quickly when I kill my clients almost immediately, while the behaviour ws doesn't clean up as fast. I can somewhat replicate that with TCP sockets, if I disconnect a portion of the total clients connecting to the server.

One hypothesis is that portion of memory buffers still active is causing chunks of buffers within nodejs not to be freed. If that is true, it seems that this memory fragmentation may cause memory issues on a long lived node process.

I'll try to put up my code in a repo for testing this if anyone's interested to investigate this.

[zz85]

Update: here's the repository to my code which test for this. https://github.com/zz85/node-ws-slow-client-buffer-leak

[Nibbler999]

If you are concerned about memory you should disable compression (perMessageDeflate: false option). That reduces memory usage on your example by 90%. You can also use the send callback to avoid overwhelming slow clients.

[zz85]

That reduces memory usage on your example by 90%

@Nibbler999 that is not true. When perMessageDeflate: false is set, the RSS grows faster in the tests because more memory is being buffered in the sockets' internal queue. In that sense turning on compression can be a good thing.

You can also use the send callback to avoid overwhelming slow clients

This is true, however what we are trying to investigate is why node holds on the memory which should be reclaimed after clients disconnect.

[zz85]

We are also running tests on the windows nodejs servers, and find that the RSS recovers (though after about 1 minute).

Here's a heapsnapshot after clients disconnects a buffer build up on MacOS.

[zz85]

Just an update: there is a fix to address this by @Nibbler999 in nodejs nodejs/node#4773 :)

[lpinca]

@zz85 is the issue solved? Can this be closed?

[zz85]

Yes, this issue was discovered many node versions ago, so I think this can be closed.