Refresh Policy for items that are expensive to fetch?

[layoutanalysis]

My Java Application needs to cache approx. 1000 JSON objects with an approximate serialized size of 20 KB each. The problem: These objects are very expensive to fetch from the database (response time in the minutes range). Is there a Caffeine refresh policy that allows me to keep the most popular items "hot" after their expiry time?

Imagine a use case where a cached item is accessed until the end of a workday, then expires overnight, which results in a cold cache at the beginning of the next workday. My goal would be to have the most popular items still in the cache the next morning.

[ben-manes]

I think that I've seen this referred to as scheduled refresh in older caching libraries. It would just reload the entire cache every time period. That can be achieved using a ScheduledExecutorService and the Cache.asMap() view. If you wanted the top-k hottest keys to reload, you could use the Cache.policy().eviction().hottest(k) to extract those, and reload. A continuous reload of the data, assuming an unbounded cache, would be best served by an immutable Map and a scheduled reloader thread.

I don't think there is anything intelligent we can or should do. Does any of the above solve your problems?

[cruftex]

There is also an extensive discussion about expiry and refresh in another issue, which had similar requirements, see:
#261 (comment)

[layoutanalysis]

Thank you for your inputs! I will probably go with a scheduled refresh and experiment with different cache policies.

[rrva]

@ben-manes you wrote:

I think that I've seen this referred to as scheduled refresh in older caching libraries. It would just reload the entire cache every time period. That can be achieved using a ScheduledExecutorService and the Cache.asMap() view. If you wanted the top-k hottest keys to reload, you could use the Cache.policy().eviction().hottest(k) to extract those, and reload. A continuous reload of the data, assuming an unbounded cache, would be best served by an immutable Map and a scheduled reloader thread.

If I use this approach with an AsyncLoadingCache, with expireAfterWrite set to 5 secs, and iterate using a scheduled job every second over myCache.synchronous().policy().eviction().hottest(10) and calling myCache.synchronous().refresh(), will this not create a gap where entries might expire before the refresher thread has filled the cache again, so that threads getting values from the cache sometimes have to pay the latency penalty for the load operation? I would like some mechanism which avoids calling threads having to pay this cost, but that instead the old value is served, even if it has expired. Otherwise I have to set the expire time high enough to guarantee that the refresh operation has had time to complete, which I do not want.

[ben-manes]

Currently when an item is marked for refresh it is eligible for expiration and the work is performed asynchronously on the background executor. This resolves your case, though that may change in the future (#373).

There are a few different strategies that you might consider, rather than relying on the above implementation detail.

1. Use variable expiration, via expireAfter, and mark the entries to refresh as ineligible for expiration (faq). This can be done via a computation prior to invoking refresh, so that the duration is recomputed to infinite. When the new value is loaded, it should recompute to the expirable duration.
2. Use a victim cache to capture the expired entries. This can be done using a CacheWriter to synchronously evict expired entries into a fallback cache, which the CacheLoader could inspect on a miss. The victim cache could have a different policy, e.g. just be a Map that you prune periodically, to let you resurrect the expired value.