websocket messages are being dropped

[go-oleg]

There have been some reports of outrigger not showing all the points coming out of a juttle program.

A program to reproduce this:

read file -file 'dense.json' -to :2014-01-01T00:59:59:
| keep value1, time
| view timechart -valueField 'value1'

The json file (needs to be renamed to .json): dense.txt

If you run this program, you will get a timechart with the following time range:

According to the data, the time should actually go up to 00:59:59.

In debugging this, i tried adding the following right after this line:

console.log(msg.type);

What I saw was a bit puzzling:

In the screenshot, there are 8 messages with type: "points" but in the console, "points" is only printed 6 times. Additionally, the "sink_end" and "job_end" messages are never printed. So the browser is certainly receiving them according to chrome dev tools, but somehow our WebSocket onmessage event listener isn't seeing them before it gets the onclose event.

Wondering if waiting longer in outrigger to close the websocket would make the problem go away, I tried adding a 1 second setTimeout on this line before actually closing the websocket. Sure enough, I started seeing all the messages:

And as expected the timechart shows the full range:

So...a bit uncertain as to what is going on. Is outrigger closing the websocket too soon? Should we be sending back acknowledgements from the browser? The browser certainly seems to be receiving the messages, just not our WebSocket onmessage handler.

@mnibecker @mstemm, you guys have dealt with some race conditions related to websockets in outrigger/juttle-client-library...any thoughts on what could be going on here?

[mstemm]

We decided that the best + easiest workaround is to actually close the websocket connection from the server side after a short delay. That should give the browser enough time to actually read + process all datapoints before the connection close causes it to lose the final points.

[mstemm]

Filed juttle/juttle-service#9 to track fixing this in juttle-engine.

[mstemm]

Un-assigning this from myself as it's for the old product.