Kiss-ICP with a mem limit is permanently GC-ing because _something_ is hogging memory

[teh-cmc]

Via @02alexander:

which basically means that we're GC-ing like crazy in hopes of reducing memory, except the thing that's actually hogging memory is not part of the GC path.

Hence the question is: what is the something in question? 👁️

[teh-cmc]

This feels very much like:

• Ginormous recordings with only a handful of rows obliterate the GC stack #4185

[teh-cmc]

TL;DR: Fixed by removing instance keys:

• Remove instance keys (Archetype-less, PoV-less, join-less queries) #5303

---

So, here's what happening.

The Kiss-ICP pipeline logs a new, bigger point-cloud as its /world/map every frame.
(I assume it does so because it either can't or doesn't want to rely on visible history for whatever reason.)

/world/map number of instances over time:

Because that number is different every frame, we need to generate a new instance key batch matching that number of instances, every frame.
And because that number is quite large, and only getting larger and larger, that's a lot of data.
And because autogenerated keys are cached and unaffected by GC, that's a lot of data hangs around permanently!

That's what you see on the left side of this memgraph:

Now, because that data hangs around indefinitely and is by itself larger than the GC threshold, the GC will now trigger every frame:

Because the GC triggers every frame, the whole viewer slows downs, which means ingestion slows down, which mean raw arrow data starts to accumulate in the different layers of buffers/channels/etc.
That's why the memory permanently grows at a very alarming rate, even though the GC is running like crazy.

That's what you see on the other side of this memgraph:

---

So, one solution to this would be to modify Kiss-ICP to log its own instance keys, that way they would get GC'd as needed.
But that's not possible since we don't expose instance keys to the SDK anymore.

Another solution would be to make instance-key generation smarter by sharing their backing buffers etc.
But that might or might not tricky in and of itself, and then we have those address-based performance hacks around instance keys to take care of... and more importantly we're trying to get rid of instance keys to begin with!

• datastore: implement shared slices for autogenerated keys #1777

So the last is simply to wait for 0.16 to land, which should get rid of (autogenerated) instance keys entirely:

• Remove instance keys (Archetype-less, PoV-less, join-less queries) #5303