Buffer improvements

[binarylogic]

As part of ensuring Vector is enterprise-ready, we need to improve how Vector buffers data. This is core to achieving the level of reliability and durability that enterprises require.

Our disk buffers need a lot of improvement. The initial spike of this work was based on leveldb which has proven problematic for this use case.

This list of items is the high-level work stream related to designing, building, and doing a beta release our buffer improvements for 2022Q1:

• Buffer improvements RFC #9477
• Buffer Improvements: create new buffer sender/receiver #9916
• Buffer Improvements: create new buffer builder #9917
• Buffer Improvements: refactor buffer configuration code to handle existing and chained styles #9918
• Buffer Improvements: write new disk buffer implementation #9920
• Buffer Improvements: write new in-memory buffer implementation #9921
• Buffer Improvements: update topology builder to understand/use new-style buffer implementation #9922
• Buffer Improvements: add adversarial filesystem test variants for disk buffer v2 #10324
• bug(buffers): disk buffer v2 doesn't close when vector triggers shutdown #10428
• bug(buffers): differentiate data directories for disk buffers v1 and v2 #10429
• task(buffers): ensure disk_v1 buffer uses same directory as before #10430
• bug(buffers): disk buffer v2 has inconsistent buffer size after restarting #10436
• task(buffers): consider migration plan from v1 to v2 for disk buffers #10835
• chore(buffers): add unit tests to disk buffer directory naming #10895
• chore(buffers): add unit test for modifying a data file when the buffer is still open (disk v2) #10898
• chore(buffers): fix flaky unit test for disk v2 #10922
• enhancement(buffers): update disk v2 record format to include version field to support schema evolution #10939
• bug(buffers): buffer_events incremented even when event is dropped #11151

[tobz]

Part of eventually releasing all of this buffer work in full is building confidence in the new disk buffer implementation, as well as deprecating older variants, switching to new ones, and so on.

Roughly speaking, here's the gameplan around the release cadence / feature status:

• 0.20 release (02/08/2022):
  • disk v2 reaches beta status (confident in basic behavior, but not yet fully confident that all logic is resilient to corner cases)
  • switch to in memory v2 as impl for memory buffer type
• 0.21 release (03/22/2022):
  • disk v2 reaches stable status (confident in basic behavior, confident in handling corner cases)
  • feature added to transparently migrate any existing disk v1 buffer to a disk v2 buffer
  • switch to disk v2 as impl for disk buffer type
  • deprecate disk v1 + in memory v1
• 0.22 release (05/03/2022):
  • remove disk v1 and in memory v1 entirely
  • remove disk v1->v2 migration code

The biggest hurdle to the 0.20 release is finishing most of the testing-related issues (#10895, #10898, #10922) and the schema evolution issue (#10939) so that we can have more confidence, and hopefully avoid needing to change the on-disk format, once we move disk v2 to beta status.

The biggest hurdle for the 0.21 release would be encouraging users to experiment with the new disk v2 buffers, as well continuing to recreate real-life workloads to stress test it on our own. Additionally, and perhaps tied as the biggest hurdle, would be writing the migration logic to move disk v1 buffers to disk v2 buffers, and doing so correctly and reliably.

[jszwedko]

Closing this since the epic was complete.