Add high-speed data logging

[PeterJohnson]

Features:

• The data is buffered and written periodically to disk on a background thread
• The file is opened immediately and may be renamed at any time
• The data format is simple and designed to be append only; it performs no rewrites of previously written data and the periodic write approach limits flash wear
• Data format supports named entries, arbitrary data types, and entry metadata
• Data format is binary for maximum speed and every record is timestamped

TODO:

• Writer API
  • C++
  • Java
• Reader API and example
  • C++
  • Java
  • Python
• Download tool
• CSV/JSON conversion tools
• ntcore interface to writer
• cscore interface to writer
• DataLogManager - automated robot side management of data files
  • cleanup of old files when disk space gets too full
  • set filename based on date/time/match number
  • log DS joystick data
  • log NT value changes
  • preferentially save to USB stick if one is mounted
  • change Java to use start() rather than just static construction
  • add option to start() to override directory?
  • add standard "messages" entry
• Unit tests
• Examples
• AsciiDoc standalone documentation of file format
• User-facing documentation

Fixes #2090.

[ajwharton]

Curious as to the rationale of this library as opposed to the myriad of standard logging facilities in Java currently. Is it meant to be smaller? Faster? Less overhead?

[PeterJohnson]

Standard logging facilities are generally designed for string messages, not for binary data. So something custom like this is going to be faster with less overhead for the typical data (e.g. numbers) coming from a robot in real time (at least dozens, if not hundreds, of data points every ~20 ms). String formatting is significantly more expensive than just copying the binary number, and is more difficult to read back into analysis tools. Latency of the actual logging to the main program is also critical in our application. We also need a logging format and implementation that works in both C++ and Java.

[Starlight220]

It looks very similar to the NTEntry API, and I assume that many teams will be using both logging and telemetry for the same value sources. Maybe merge the two, reusing symbols, or at least make it trivial to add logging to a telemetry setup and the other way around?

[PeterJohnson]

My current plan is for the DataLogManager to write all NT value changes to the "main" data log.

[PeterJohnson]

@virtuald for your awareness