Core cache implementation can deadlock

[jasontedor]

Here is a stack dump from a failing test:

"Thread-1" #19 prio=5 os_prio=31 tid=0x00007f906eb14800 nid=0x7803 waiting on condition  [0x000070000618c000]
   java.lang.Thread.State: WAITING (parking)
        at jdk.internal.misc.Unsafe.park(java.base@10/Native Method)
        - parking to wait for  <0x00000006cecb5790> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(java.base@10/LockSupport.java:194)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@10/AbstractQueuedSynchronizer.java:883)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.base@10/AbstractQueuedSynchronizer.java:915)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@10/AbstractQueuedSynchronizer.java:1238)
        at java.util.concurrent.locks.ReentrantLock.lock(java.base@10/ReentrantLock.java:267)
        at org.elasticsearch.common.util.concurrent.ReleasableLock.acquire(ReleasableLock.java:55)
        at org.elasticsearch.common.cache.Cache.lambda$computeIfAbsent$5(Cache.java:391)
        at org.elasticsearch.common.cache.Cache$$Lambda$207/1567223321.apply(Unknown Source)
        at java.util.concurrent.CompletableFuture.uniHandle(java.base@10/CompletableFuture.java:930)
        at java.util.concurrent.CompletableFuture$UniHandle.tryFire(java.base@10/CompletableFuture.java:907)
        at java.util.concurrent.CompletableFuture.postComplete(java.base@10/CompletableFuture.java:506)
        at java.util.concurrent.CompletableFuture.complete(java.base@10/CompletableFuture.java:2073)
        at org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:422)
        at org.elasticsearch.common.cache.CacheTests.lambda$testComputeIfAbsentDeadlock$9(CacheTests.java:358)
        at org.elasticsearch.common.cache.CacheTests$$Lambda$200/1841769401.run(Unknown Source)
        at java.lang.Thread.run(java.base@10/Thread.java:844)

"Thread-2" #20 prio=5 os_prio=31 tid=0x00007f906eb17000 nid=0x8d03 waiting on condition  [0x000070000628e000]
   java.lang.Thread.State: WAITING (parking)
        at jdk.internal.misc.Unsafe.park(java.base@10/Native Method)
        - parking to wait for  <0x00000006ce8b7498> (a java.util.concurrent.CompletableFuture$Signaller)
        at java.util.concurrent.locks.LockSupport.park(java.base@10/LockSupport.java:194)
        at java.util.concurrent.CompletableFuture$Signaller.block(java.base@10/CompletableFuture.java:1796)
        at java.util.concurrent.ForkJoinPool.managedBlock(java.base@10/ForkJoinPool.java:3156)
        at java.util.concurrent.CompletableFuture.waitingGet(java.base@10/CompletableFuture.java:1823)
        at java.util.concurrent.CompletableFuture.get(java.base@10/CompletableFuture.java:1998)
        at org.elasticsearch.common.cache.Cache$CacheSegment.remove(Cache.java:290)
        at org.elasticsearch.common.cache.Cache.evictEntry(Cache.java:720)
        at org.elasticsearch.common.cache.Cache.evict(Cache.java:711)
        at org.elasticsearch.common.cache.Cache.promote(Cache.java:701)
        at org.elasticsearch.common.cache.Cache.lambda$computeIfAbsent$5(Cache.java:392)
        at org.elasticsearch.common.cache.Cache$$Lambda$207/1567223321.apply(Unknown Source)
        at java.util.concurrent.CompletableFuture.uniHandle(java.base@10/CompletableFuture.java:930)
        at java.util.concurrent.CompletableFuture$UniHandle.tryFire(java.base@10/CompletableFuture.java:907)
        at java.util.concurrent.CompletableFuture.postComplete(java.base@10/CompletableFuture.java:506)
        at java.util.concurrent.CompletableFuture.complete(java.base@10/CompletableFuture.java:2073)
        at org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:422)
        at org.elasticsearch.common.cache.CacheTests.lambda$testComputeIfAbsentDeadlock$9(CacheTests.java:358)
        at org.elasticsearch.common.cache.CacheTests$$Lambda$200/1841769401.run(Unknown Source)
        at java.lang.Thread.run(java.base@10/Thread.java:844)

I will push the failing test and mark it as awaiting fix by this issue.

[elasticmachine]

Pinging @elastic/es-core-infra