Application gets stuck when Feign client is used with virtual threads

[stsypanov]

Steps to reproduce:

1. Checkout https://github.com/stsypanov/concurrency-demo
2. Run DependencyApplication and then ConcurrencyDemoApplication
3. When both apps are up run StuckApplicationTest
4. It will take about 1-2 minutes for the test to complete
5. Now go to demo-service/application.yml and set spring.threads.virtual.enabled: true (by default it's false).
6. Restart ConcurrencyDemoApplication
7. Run StuckApplicationTest again

If you now attach a profile (e.g. YourKit) you'll see there's potential deadlock with this stacktrace

Potential deadlock: frozen threads found

It seems that the following threads have not changed their stack for more than 10 seconds.
These threads are possibly (but not necessarily!) in a deadlock or hung.

+-----------------------------------------------------------------------------------------------------------------------------+
|                                                            Name                                                             |
+-----------------------------------------------------------------------------------------------------------------------------+
|  +---Read-Updater Frozen for at least 10s <Ignore a false positive>                                                         |
|  | |                                                                                                                        |
|  | +---jdk.internal.misc.Unsafe.park(boolean, long) Unsafe.java (native)                                                    |
|  | |                                                                                                                        |
|  | +---java.util.concurrent.locks.LockSupport.park() LockSupport.java:371                                                   |
|  | |                                                                                                                        |
|  | +---java.util.concurrent.LinkedTransferQueue$DualNode.await(Object, long, Object, boolean) LinkedTransferQueue.java:458  |
|  | |                                                                                                                        |
|  | +---java.util.concurrent.LinkedTransferQueue.xfer(Object, long) LinkedTransferQueue.java:613                             |
|  | |                                                                                                                        |
|  | +---java.util.concurrent.LinkedTransferQueue.take() LinkedTransferQueue.java:1257                                        |
|  | |                                                                                                                        |
|  | +---sun.nio.ch.Poller.updateLoop() Poller.java:286                                                                       |
|  | |                                                                                                                        |
|  | +---sun.nio.ch.Poller$$Lambda.0x0000024081474670.run()                                                                   |
|  | |                                                                                                                        |
|  | +---java.lang.Thread.runWith(Object, Runnable) Thread.java:1596                                                          |
|  | |                                                                                                                        |
|  | +---java.lang.Thread.run() Thread.java:1583                                                                              |
|  | |                                                                                                                        |
|  | +---jdk.internal.misc.InnocuousThread.run() InnocuousThread.java:186                                                     |
|  |                                                                                                                          |
|  +---spring.cloud.inetutils Frozen for at least 10s <Ignore a false positive>                                               |
|  | |                                                                                                                        |
|  | +---java.net.Inet6AddressImpl.getHostByAddr(byte[]) Inet6AddressImpl.java (native)                                       |
|  | |                                                                                                                        |
|  | +---java.net.InetAddress$PlatformResolver.lookupByAddress(byte[]) InetAddress.java:1225                                  |
|  | |                                                                                                                        |
|  | +---java.net.InetAddress.getHostFromNameService(InetAddress, boolean) InetAddress.java:840                               |
|  | |                                                                                                                        |
|  | +---java.net.InetAddress.getHostName(boolean) InetAddress.java:782                                                       |
|  | |                                                                                                                        |
|  | +---java.net.InetAddress.getHostName() InetAddress.java:754                                                              |
|  | |                                                                                                                        |
|  | +---org.springframework.cloud.commons.util.InetUtils$$Lambda.0x0000024081187240.call()                                   |
|  | |                                                                                                                        |
|  | +---java.util.concurrent.FutureTask.run() FutureTask.java:317                                                            |
|  | |                                                                                                                        |
|  | +---java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) ThreadPoolExecutor.java:1144            |
|  | |                                                                                                                        |
|  | +---java.util.concurrent.ThreadPoolExecutor$Worker.run() ThreadPoolExecutor.java:642                                     |
|  | |                                                                                                                        |
|  | +---java.lang.Thread.runWith(Object, Runnable) Thread.java:1596                                                          |
|  | |                                                                                                                        |
|  | +---java.lang.Thread.run() Thread.java:1583                                                                              |
|  |                                                                                                                          |
|  +---Write-Updater Frozen for at least 10s <Ignore a false positive>                                                        |
|    |                                                                                                                        |
|    +---jdk.internal.misc.Unsafe.park(boolean, long) Unsafe.java (native)                                                    |
|    |                                                                                                                        |
|    +---java.util.concurrent.locks.LockSupport.park() LockSupport.java:371                                                   |
|    |                                                                                                                        |
|    +---java.util.concurrent.LinkedTransferQueue$DualNode.await(Object, long, Object, boolean) LinkedTransferQueue.java:458  |
|    |                                                                                                                        |
|    +---java.util.concurrent.LinkedTransferQueue.xfer(Object, long) LinkedTransferQueue.java:613                             |
|    |                                                                                                                        |
|    +---java.util.concurrent.LinkedTransferQueue.take() LinkedTransferQueue.java:1257                                        |
|    |                                                                                                                        |
|    +---sun.nio.ch.Poller.updateLoop() Poller.java:286                                                                       |
|    |                                                                                                                        |
|    +---sun.nio.ch.Poller$$Lambda.0x0000024081474670.run()                                                                   |
|    |                                                                                                                        |
|    +---java.lang.Thread.runWith(Object, Runnable) Thread.java:1596                                                          |
|    |                                                                                                                        |
|    +---java.lang.Thread.run() Thread.java:1583                                                                              |
|    |                                                                                                                        |
|    +---jdk.internal.misc.InnocuousThread.run() InnocuousThread.java:186                                                     |
+-----------------------------------------------------------------------------------------------------------------------------+

This isn't a deadlock, but the application gets stuck at this method

public HostInfo convertAddress(final InetAddress address) {
	HostInfo hostInfo = new HostInfo();
	Future<String> result = this.executorService.submit(address::getHostName);

	String hostname;
	try {
		hostname = result.get(this.properties.getTimeoutSeconds(), TimeUnit.SECONDS);
	}
	catch (Exception e) {
		this.log.info("Cannot determine local hostname");
		hostname = "localhost";
	}
	hostInfo.setHostname(hostname);
	hostInfo.setIpAddress(address.getHostAddress());
	return hostInfo;
}

Apparently, the reason is that executorService is single-thread executor and cannot serve more than 1 request simultaneously:

public InetUtils(final InetUtilsProperties properties) {
	this.properties = properties;
	this.executorService = Executors.newSingleThreadExecutor(r -> {
		Thread thread = new Thread(r);
		thread.setName(InetUtilsProperties.PREFIX);
		thread.setDaemon(true);
		return thread;
	});
}

I suggest to change switch it to Executors.newCachedThreadPool for it's JavaDocs explicitly says

Creates a thread pool that creates new threads as needed, but will reuse previously constructed threads when they are available. These pools will typically improve the performance of programs that execute many short-lived asynchronous tasks. Calls to execute will reuse previously constructed threads if available. If no existing thread is available, a new thread will be created and added to the pool. Threads that have not been used for sixty seconds are terminated and removed from the cache. Thus, a pool that remains idle for long enough will not consume any resources.

[OlgaMaciaszek]

Thanks, @stsypanov. Makes sense.

[stsypanov]

@OlgaMaciaszek hi, it turned out, that the root cause is a JVM issue related virtual threads https://stackoverflow.com/questions/78790376/spring-boot-application-gets-stuck-when-virtual-threads-are-used-on-java-21/78805574#78805574

However, I still think that migration to Executors.newCachedThreadPool() is still reasonable

[stsypanov]

As soon as it's not either Feign, or spring-cloud issue I close this.

[OlgaMaciaszek]

Thanks a lot @stsypanov. Will take a look at your PR today.