Background Eviction strategy for ShardIndexingPressureStore

[getsaurabh02]

This is related to Meta issue #478 (Shard Indexing Pressure feature), and adds a follow up item to it, where we need to have a background eviction strategy for shard indexing pressure trackers, which performs low level operation tracking.

The two phased lifecycle of tracker (hot and cold) has actually helped reduce churns and reduce GC

Currently with #838 we have added two phased lifecycle management for trackers i.e. hot and cold. While hot store can track unlimited number of trackers which are accounting for live requests, cold store has a size limit to evict the trackers residing in cold store once the limit is breached.

Purpose of Cold Store
Similar to memory pools, Cold store acts as an object pool for ShardIndexingPressureTracker objects, to prevent frequent allocation and collection churns. It helps with the reusability of the tracker objects, when a new request arrives in, using a previously used index. Then the older tracker, if present in cold store, can be pulled back to hot store and new values can be updated for tracking. In case not present in cold store, a new tracker object is created, which is functionally indifferent.

Gaps with the current strategy
Size based eviction strategy of cold store do not offer uniform cleanup, as eviction happens for all trackers at once, whenever the store limit is reached. Hence it does not offer any fairness or preference based on the recency of the shards trackers present in the cold store. In such cases, when the entire cold store is purged, we might end up loosing few trackers those were added to the store very recently, and could have been potentially reused again. These extra purges results into new trackers being created if needed again.

Proposal to evaluate
Similar to how ShardsIndicesStatusChecker artefact present in IndexingMemoryController provides a Runnable to check whether a shard is inactive, and sync'd flush can happen, we can to have a Background reaping strategy, which can piggyback upon Tracker's lastSuccessfulRequestTimestamp, to determine the recency of usage. The background thread could evict entries of tracker if their last usage interval is greater than the predefined inactivity timeout.

Few important considerations to make

• Evaluate if performing repetitive scans across all trackers can be sub-optimal, and needs to be supported by an additional LRU based structure, to help background threads in faster cleanups.
• Can periodicity of cleanup if sufficiently large, can end up holding large number of trackers, which needs to be supported by some size based upper cap too.
• To avoid race between eviction and new requests using the same tracker, do we need to have a two phased mark and evict strategy. This is solved by separate cold and hot pools today.