Charts Cause Browser Page Crashes

[JohnMacrae]

I've been using a page with either 1 chart with 2 inputs or 2 charts with one input showing MQTT-derived data for an hour at a time (updated every 10 secs). After several hours, the pages 'snap' on both Chrome and Firefox on Win 10.

[Paul-Reed]

Has worked fine here for several days. I have 2 pages, page 1 with six charts with one input each, and page 2 with one chart with two inputs, all being fed via MQTT, and same as you I'm viewing them on Win 10.
When you say 'snap' what exactly are you seeing?
If you refresh the page (F5) or reload it, does it load OK again? and if not, how do you restore the page?
Have you tried viewing the page in your browser's 'Developer's' mode (F12 key on Chrome & Firefox) and looking what errors, if any are, being shown in the console.
Paul

[knolleary]

I have seen this sort or hang myself with the Chart node - so there is definitely something here to investigate. Smells of a memory leak in the chart node.

[durst99]

I`ve got a similar issue. When adding chart nodes my RAM usage for the browser keeps filling up constantly ( Up to 2GB and more) and at some point it crashed.
I already reduced the incoming MQTT data points which makes it a little better, but somehow it still got some memory problems I think.
It does not matter which browers, tryed it with Firefox, Chrome and IE.

[dceejay]

Right now the server end of the chart holds ALL the previous values until they time out (based on the X Range window) - AND worse still sends them all to the client every time new data arrives...
This means a) the array can get very large... and is in memory - so will blow up fairly quick if you have fast moving data, and
b) will send it all on every change to the client for plotting... - which can potentially trash the web socket... (or DOS it anyway)

Options are

1. leave it as is - but change default window to (say) 10 mins, and add to docs - ie so hopefully no so bad - but still down to the user to break.
2. do range reduction on the incoming data - need to pick a sensible algorithm that shows accurate data points and reflects history but reduces resolution. This could then reduce the amount of data points stored and sent to a max of "chart width in pixels", which again could be large - but not horrendous. But would take quite a lot of pre-processing before sending (unless we can find a decent algorithm - all suggestions accepted...)
3. other that I've not yet thought of - feel free to comment

[JohnMacrae]

Good suggestions Dave, thank you. I'm in the testing phase of my project
right now and sampling every 10s - production will be every 5 mins. I'm
sure there will be folks that will want more though :)

Kind Regards
John

On 8 August 2016 at 18:42, Dave Conway-Jones [email protected]
wrote:

Right now the server end of the chart holds ALL the previous values until
they time out (based on the X Range window) - AND worse still sends them
all to the client every time new data arrives...
This means a) the array can get very large... and is in memory - so will
blow up fairly quick if you have fast moving data, and
b) will send it all on every change to the client for plotting... - which
can potentially trash the web socket... (or DOS it anyway)

Options are

1. leave it as is - but change default window to (say) 10 mins, and add to
   docs - ie so hopefully no so bad - but still down to the user to break.
2. do range reduction on the incoming data - need to pick a sensible
   algorithm that shows accurate data points and reflects history but reduces
   resolution. This could then reduce the amount of data points stored and
   sent to a max of "chart width in pixels", which again could be large - but
   not horrendous. But would take quite a lot of pre-processing before sending
   (unless we can find a decent algorithm - all suggestions accepted...)

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#38 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AM80DVCkQXTzRu0k5Mz1M7vPfx3Ke6Zyks5qd1x2gaJpZM4JaKPA
.

[TotallyInformation]

Doesn't seem that sensible to have to resend all the data on each update - wouldn't it be better to let the browser accumulate the data? I've used browser-based charts that can easily handle hundreds of thousands of data points without breaking into a sweat. Indeed, Grafana is a good example, I've thrown some really silly amounts of data at that (multiple lines with 40sec intervals for 12m).

The old UI graph used to grind the browser to a halt if you had more than about 50 data points so things have at least moved in the right direction!

[dceejay]

Won't you want to see data that arrived while the browser wasn't connected ? So it has to be accumulated at the host end as well.

[TotallyInformation]

Hmm, good point, didn't think that through did I! Then there is also the problem of multiple connected clients. How would that work. Things could get complex very quickly. That would seem to indicate that the chart node should be kept simple, maybe having an optional limit added to the config for the node?

There almost certainly isn't a one-size-fits-all solution here since the reason for a chart can vary so wildly. However, it would seem sensible for the standard chart node to be fairly simple with a direction to people to craft their own chart using a Template if they want something more complex.

[jwildeboer]

Similar situation here. I have 2 graphs that collect temperature data (every 30 seconds 2 new values) and the graphs are configured to show data collected fro 24 hours. When I first open the page it sits at some 35M of memory use (Chrome Version 52.0.2743.82 (64-bit) on Red Hat Enterprise Linux 7.2). This grows radically after opening all the tabs a few times to 150MB and when left sitting idle in the background continues to grow to now 260MB, 30 minutes after opening the page.

As it clearly seems to be a memory usage issue - shouldn't we rename the Issue title to "Possible memory leak"?

[patricktokeeffe]

I don't really understand why all data points get sent for each new data point? Why isn't an array of historical/cached data sent during the initial websocket connection, and then only new data points get broadcast?

Is there no mechanism on the browser side for appending new points and discarding those older than X? That kind of logic could live in the chart node, right?

some context: in my lab, we are increasing using digital sensors with Raspberry Pis to conduct field measurements and support student research projects. I'd like to introduce Node-RED as the go-to data acquisition framework for the Pis but near-real time plotting is an essential component for us. Displaying 24 hours of 6 channels updating at 1Hz wouldn't be unreasonable, if we had our way. 😄

You could probably guess this but imposing arbitrary window size limits to 'fix' this issue would make dashboard charts totally worthless for my group and, I suspect, many others as well. Similarly, we would frown upon range reduction (that should really be delegated back to the user to specify an appropriate technique).

[jwildeboer]

Maybe this is related and possibly of help? http://stackoverflow.com/questions/31136762/nvd3-living-chart-memory-leak and http://stackoverflow.com/questions/24689157/nvd3-how-to-refresh-the-data-function-to-product-new-data-on-click/24695324#24695324

[jwildeboer]

After a few days of testing, I can reliably reproduce browser crashes by opening a tab with 2 graphs shown, collecting data at 2 values per minute, displaying 24 hours range. After roundabout 2-3 hours memory consumption on Chrome goes beyond 3GB(!) and the browser page "aw, snaps".

[{"id":"62d68e83.7ff65","type":"ui_chart","z":"8477cc8d.cd87c","group":"24f7f017.23ef6","order":2,"width":"5","height":"3","label":"","name":"DisplaySensorTempOut24h","interpolate":"basis","nodata":"No Data","ymin":"","ymax":"","removeOlder":1,"removeOlderUnit":"86400","x":580,"y":140,"wires":[[],[]]},{"id":"5333513.8a073b","type":"ui_gauge","z":"8477cc8d.cd87c","name":"DisplaySensorTempOutCurrent","group":"24f7f017.23ef6","order":1,"width":"3","height":"3","gtype":"gage","title":"OUT","label":"","format":"{{value | number:1}} °C","min":"-25","max":"60","x":590,"y":100,"wires":[]},{"id":"ca2ccf2a.1aaa1","type":"link in","z":"8477cc8d.cd87c","name":"GetSensorTempOut","links":["a6c52322.eac2a","ee8622c5.44b58","6fbe84f0.bb5f6c"],"x":125,"y":100,"wires":[["5333513.8a073b","62d68e83.7ff65"]]},{"id":"48112a24.b36614","type":"ui_gauge","z":"8477cc8d.cd87c","name":"DisplaySensorTempInCurrent","group":"99e65d86.6e69a","order":1,"width":"3","height":"3","gtype":"gage","title":"IN","label":"","format":"{{value | number:1}} °C","min":"-25","max":"60","x":590,"y":200,"wires":[]},{"id":"72aa610d.b6a55","type":"link in","z":"8477cc8d.cd87c","name":"GetSensorTempIn","links":["26328e18.0ef892"],"x":124,"y":200,"wires":[["48112a24.b36614","68a0c476.90ca0c"]]},{"id":"68a0c476.90ca0c","type":"ui_chart","z":"8477cc8d.cd87c","group":"99e65d86.6e69a","order":2,"width":"5","height":"3","label":"","name":"DisplaySensorTempIn24h","interpolate":"basis","nodata":"","ymin":"","ymax":"","removeOlder":1,"removeOlderUnit":"86400","x":570,"y":240,"wires":[[],[]]},{"id":"24f7f017.23ef6","type":"ui_group","z":"8477cc8d.cd87c","name":"OUT","tab":"e0aab23c.07faa","disp":false,"width":"8"},{"id":"99e65d86.6e69a","type":"ui_group","z":"","name":"IN","tab":"e0aab23c.07faa","order":2,"disp":false,"width":"8"},{"id":"e0aab23c.07faa","type":"ui_tab","z":"","name":"Sensors","icon":"dashboard","order":3}]

[btsimonh]

For a production system, we MUST have a limit on the number of points stored; as well as on the time. I just ran a graph which by mistake swallowed 150000 historical entries, an it crashed both node and the browser; and worse lost my flow changes, and because it did it every startup, i had to physically disconnect the data source to then run up to modify the flow before reconnecting.
I'd like to see a low limit on number of points, able to be raised by configuration.

[btsimonh]

A workaround: with the current chart, you can 'restore' data to it; repeatedly. This actually means you can keep your own store of x/y points, and push them to the chart replacing previous points, so avoiding the issue (any also giving you full control of 'X'!! - and you could 'thin' the points before restoring to save browser mem/transfers). example in this flow: http://flows.nodered.org/flow/af9f48033bbf56a2282f3c7f41d6500b .
Note I found the restore message not well documented; it must have a topic of 'restore', and payload must contain an array of series like
[{"key":"Data","values":[[1478426670102,1.3041],[1478426682091,1.5]]}]
If others have used this kind of workaround, it may be important to retain/add this restore format for the new angular-chart implementation?

[DanCunnington]

Thanks @btsimonh - I'll add an option to limit the number of points and also add some documentation around restoring the chart to #89

[dceejay]

Now pushed to master - 0fe76c2
Going to close this issue - feel free to open a new one against the new inmplementation

[robertatdi]

This still seems to be an issue for me. Anyone else on the same boat? (I am running the latest updates)

[btsimonh]

@robertatdi : check it's not the admin page stealing all your ram - when I have some diagnostics running on an admin page, it tends to use an every increasing amount until it crashes or I refresh.

[tomwimmenhove]

The ram usage seems not to be the only problem. My google Chrome (tested only in linux) also starts slurping tons of CPU when a tab with a chart is visible.