Writing to Influxdb2.0 - attempt at buffering

[pbegg]

This is the first Node App I have ever written so go easy on me. I wanted a plugin that would save data to influx2.0 but buffer the data if the internet connection went down and complete batch writes to hopefully reduce data consumption.
Would love some feedback on the structure of the app and where improvements can be made. It is currently only supporting propulsion and navigation paths as this is all I needed for the time being.

https://github.com/pbegg/signalk-to-influxdb-v2-buffering

[c33howard]

Cool. I had a similar issue. See the other discussion thread about time series in the cloud. My final destination is timestream and I'm writing via S3, but the core of it is getting data off the boat despite a small and unreliable LTE connection. The same approach should work for any webserver and db. Here's my README that outlines the big picture. You've thought about the same problem, so please send feedback.

I quickly skimmed your code. Two larger comments and one small one:

1. I think you should name your files on disk based on the first time they were first submitted, not when you failed to write them to influxdb. The case I'm worried about is that you fail a write to influx, then buffer the item. Later on you read it back in and fail again and write a new file to disk. This could lead to sending duplicates. The other reason to name your files based on the time of submission is that when you recover, the newest data is probably most valuable and the data you want to write to influxdb first. If you have alarms setup, you want to know about alarms ASAP. If you have to work through a backlog before you get to the recent data, you're firing your alarms extra late. If your files are named by time of original submission, that should be easy to do with a descending sort on the results of your directory listing.
2. You really don't want to use the sync file system interface. Especially with a raspberry pi using an SD card. Filesystems on linux can and do block. If you're using the sync interface in a plugin, then all of signalk will block waiting for the filesystem to get back to you. This means you might not see the most recent depth reading off your NMEA bus. Filesystem pauses happen way more than you'd think, even on server hardware with SSDs.
3. Consider abstracting time in your code. This makes things much easier to test. For an example: index.js and test.js

Craig

[pbegg]

Thanks for taking the time to have a look through it! I have actually been reading through your projects the past few days while I was working on this. I have found it very interesting. I am now having a looking in more detail to see how I can improve. I don't know node or javascript at all. so I work out what I want to do then research each function/code that I will need and it is helping reading through other signalk plugins.
I'll keep looking at your solution as that low data usage (only 300mb for a month) is very favourable. I also like the idea of running an instance of signalk in the cloud.

1. I like your idea and to be honest I didn't put much thought into it, it was my first solution to supply a unique file name.
2. I did make a note of this in a to do list. I was having trouble with the async function as I was trying to return a result before the async function had completed. Im still getting my head around callbacks, promises etc..
3. Not 100% sure what you mean by this but i'll read up on it and have a look through your links.

Cheers,
Phil

[c33howard]

Haha. Me too. I've dabbled in javascript, but this is my first work in node. I've got the docs open on one screen and vim on another. I do have a lot of experience with async programming in a few different languages though, which has helped a ton.

1. A file based on date makes a lot of sense. Or at least I think so, I did the same. :-) The challenge is that the files have to get sent to another system, potentially on a remote machine. That's a distributed systems problem, which are about as hard as they get in computer science. There are so many failure cases to deal with.

2. Yeah, it's a bit weird. I've been writing async code for over ten years and I still have to think through it carefully sometimes. One layer of async is relatively easy, it gets harder when you make an async call, which means your API now has to be async too. In an async system, the first function call initiates an action. The return from that function call just means that the system has started work. The callback or promise will be invoked at some arbitrary time later on when the work is done. I find physical analogies can help me. Consider an inbox and outbox (like literal boxes sitting on your desk with pieces of paper that go in them, where each piece of paper describes work or results). When you kick off an async function, you're writing down the work you want to do on the piece of paper and placing it in the inbox. The system comes around at its leisure and pulls work out of the inbox and starts chugging away. As it finishes processing the request on each piece of paper from the inbox, it puts a piece of paper in the outbox with the results written on it. When you notice results in the outbox (ie the callback or promise is invoked), you pick up the piece of paper, read the results, then do something with it. You might kick off more work (make another async call), or write an addendum on the paper and go put the paper in someone else's outbox (ie call another callback or resolve a promise).

3. If you're familiar with the term, I'm "mocking" time. Testing is really hard when you have to rely on real time elapsing. My code does batching over time intervals. I could put sleeps in the test code to see that the batches happen when I expect them to, but then the test might fail if the test is run on a faster or slower computer than mine. Instead, I make a function that returns whatever time I want it to return and manually set the time on the clock in tests. But in the production code, it does use the real clock.

Hope that helps.

Craig

[pbegg]

Yeah big learning curve as I hadn't done any programming until recently when I started to use signalk, lots of options and opportunities with all the data that is available.
Thank you for the analogy - I have been reading up on it quite a bit and watching a few tutorials. I think I understand the concept quite well, its just utilising it in my actual code and making it work is the difficult part.
I understand what you mean by mocking time. I'll have another look through your links to see how you have utilised it.

It helps a lot, thanks, I appreciate you taking the time.