SSD Split "cache"

[fulmicoton]

For some slightly high performance/high traffic we want a solution to have splits data available on a local SSDs.

I put "cache" under quotation mark, because it is not really a cache. It should be populated, upon publish (as opposed to "on first read").
I called it "a cache" because it plays no role in durability. If the node hosting this local copy is down, the query can still hit S3 directly and perform as usual.

This ticket will likely involve

• a pre-flight query
• a way to lock splits in the cache for a given grace period, to prevent their eviction
• some logic (rendez vous hashing?) to place splits on nodes?
• some eviction strategy, etc.

[imotov]

What is the role of "a pre-flight query" in this context?

[fulmicoton]

Right now we do assign split's leaf_search to searchers using some logic that tries to spread the work evenly and maximize affinity, the affinity being defined using rendez-vous hashing.

But @trinity-1686a added a "split search result cache" and you are about to add a split cache.
It might be interesting to have one round of RPC to ask leaf nodes if they have data already in their cache before assigning the leaf search request.

[guilload]

Closed via #3857.