Upgrading from 3.5.3 to 3.6.1 ,3.6.3,3.6.4 Quakust test will not start with Thread Thread[vert.x-eventloop-thread-6,5,main] has been blocked for 2460 m #37958

psini · 2023-12-29T09:46:13Z

Describe the bug

Starting a quarkus test it will not start because it will block the following exception. We have at the same time smallrye-jwt and oidc installed. Using quarkus.arc.optimize-contexts=false it starts correctly and works .

at java.base@17.0.9/jdk.internal.misc.Unsafe.park(Native Method)
	at java.base@17.0.9/java.util.concurrent.locks.LockSupport.park(LockSupport.java:211)
	at java.base@17.0.9/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:715)
	at java.base@17.0.9/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:938)
	at java.base@17.0.9/java.util.concurrent.locks.ReentrantLock$Sync.lock(ReentrantLock.java:153)
	at java.base@17.0.9/java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:322)
	at io.quarkus.arc.generator.Default_jakarta_enterprise_context_ApplicationScoped_ContextInstances.c319(Unknown Source)
	at io.quarkus.arc.generator.Default_jakarta_enterprise_context_ApplicationScoped_ContextInstances.computeIfAbsent(Unknown Source)
	at io.quarkus.arc.impl.AbstractSharedContext.get(AbstractSharedContext.java:38)
	at io.quarkus.arc.impl.ClientProxies.getApplicationScopedDelegate(ClientProxies.java:21)
	at io.quarkus.oidc.runtime.OidcAuthenticationMechanism_ClientProxy.arc$delegate(Unknown Source)
	at io.quarkus.oidc.runtime.OidcAuthenticationMechanism_ClientProxy.getCredentialTypes(Unknown Source)
	at io.quarkus.vertx.http.runtime.security.HttpAuthenticator.<init>(HttpAuthenticator.java:43)
	at io.quarkus.vertx.http.runtime.security.HttpAuthenticator_Bean.doCreate(Unknown Source)
	at io.quarkus.vertx.http.runtime.security.HttpAuthenticator_Bean.create(Unknown Source)
	at io.quarkus.vertx.http.runtime.security.HttpAuthenticator_Bean.create(Unknown Source)
	at io.quarkus.arc.impl.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:117)
	at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:41)
	at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:38)
	at io.quarkus.arc.impl.LazyValue.get(LazyValue.java:32)
	at io.quarkus.arc.impl.ComputingCache.computeIfAbsent(ComputingCache.java:69)
	at io.quarkus.arc.impl.ComputingCacheContextInstances.computeIfAbsent(ComputingCacheContextInstances.java:18)
	at io.quarkus.arc.impl.AbstractSharedContext.get(AbstractSharedContext.java:38)
	at io.quarkus.vertx.http.runtime.security.HttpAuthenticator_Bean.get(Unknown Source)
	at io.quarkus.vertx.http.runtime.security.HttpAuthenticator_Bean.get(Unknown Source)
	at io.quarkus.arc.impl.InstanceImpl.getBeanInstance(InstanceImpl.java:325)
	at io.quarkus.arc.impl.InstanceImpl.getInternal(InstanceImpl.java:309)
	at io.quarkus.arc.impl.InstanceImpl.get(InstanceImpl.java:190)
	at io.quarkus.vertx.http.runtime.security.HttpSecurityRecorder$AbstractAuthenticationHandler.handle(HttpSecurityRecorder.java:220)
	at io.quarkus.vertx.http.runtime.security.HttpSecurityRecorder$AbstractAuthenticationHandler.handle(HttpSecurityRecorder.java:208)
	at io.vertx.ext.web.impl.RouteState.handleContext(RouteState.java:1286)
	at io.vertx.ext.web.impl.RoutingContextImplBase.iterateNext(RoutingContextImplBase.java:177)
	at io.vertx.ext.web.impl.RoutingContextImpl.next(RoutingContextImpl.java:144)
	at io.vertx.ext.web.impl.RouterImpl.handle(RouterImpl.java:68)
	at io.vertx.ext.web.impl.RouterImpl.handle(RouterImpl.java:37)
	at io.quarkus.vertx.http.runtime.options.HttpServerCommonHandlers$2.handle(HttpServerCommonHandlers.java:83)
	at io.quarkus.vertx.http.runtime.options.HttpServerCommonHandlers$2.handle(HttpServerCommonHandlers.java:66)
	at io.quarkus.vertx.http.runtime.VertxHttpRecorder$1.handle(VertxHttpRecorder.java:162)
	at io.quarkus.vertx.http.runtime.VertxHttpRecorder$1.handle(VertxHttpRecorder.java:138)
	at io.vertx.core.impl.EventLoopContext.emit(EventLoopContext.java:55)
	at io.vertx.core.impl.DuplicatedContext.emit(DuplicatedContext.java:179)
	at io.vertx.core.http.impl.Http1xServerConnection.handleMessage(Http1xServerConnection.java:174)
	at io.vertx.core.net.impl.ConnectionBase.read(ConnectionBase.java:159)
	at io.vertx.core.net.impl.VertxHandler.channelRead(VertxHandler.java:153)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
	at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:333)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:454)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1471)
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1334)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1383)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base@17.0.9/java.lang.Thread.run(Thread.java:840)

Expected behavior

quarkus should start

Actual behavior

No response

How to Reproduce?

using the following feature Installed features: [cache, cdi, hibernate-validator, kafka-client, micrometer, mongodb-client, oidc, opentelemetry, reactive-routes, rest-client, rest-client-jackson, resteasy, resteasy-jackson, resteasy-multipart, scheduler, security, servlet, smallrye-context-propagation, smallrye-fault-tolerance, smallrye-health, smallrye-jwt, smallrye-openapi, smallrye-reactive-messaging, smallrye-reactive-messaging-kafka, swagger-ui, vertx]

Output of `uname -a` or `ver`

No response

Output of `java -version`

No response

Quarkus version or git rev

No response

Build tool (ie. output of `mvnw --version` or `gradlew --version`)

No response

Additional information

No response

The text was updated successfully, but these errors were encountered:

psini · 2023-12-29T09:47:10Z

@gsmet I suppose is something in [area/arc]but i cannot assign the label

sberyozkin · 2023-12-29T12:24:36Z

@psini Can you please create a simple reproducer ? We have a couple of integration tests combining both smallrye-jwt and oidc, example, https://github.com/quarkusio/quarkus/blob/main/integration-tests/smallrye-jwt-oidc-webapp/pom.xml

Also CC @mkouba @michalvavrik

psini · 2023-12-29T13:42:39Z

Hi i cannot reproduce it with a simple reproducer , it happens only on one project that import different microservices inside one single project. I tried also to debug it but without any success. It seams somthing inside io.quarkus.arc.generator.Default_jakarta_enterprise_context_ApplicationScoped_ContextInstances but i can see the source of it.
Using quarkus.arc.optimize-contexts=false everything works fine so @Ladicek could hava a clue how to check it or why it happens.

psini · 2023-12-29T15:01:31Z

I have done some troubleshooting and found the culprit . I used applicationscoped beans and injectedthem in the test code that is running on main thread.
In one of those beans I had a postconstrunct that do a rest call to the app running on quarkus that has to initialize other beans like OidcAuthenticationMechanism but the lock inside Default_jakarta_enterprise_context_ApplicationScoped_ContextInstances is only one for every bean so it creates a deadlock because the vertx thread that want to init OidcAuthenticationMechanism should wait the lock to be realesed from the creation of the test class on main but it will never happen.
Let me know if you need more info. A possible solution is have a lock inside Default_jakarta_enterprise_context_ApplicationScoped_ContextInstances for each kind of bean and not one for everithing.

sberyozkin · 2023-12-29T15:14:22Z

Thanks @psini, I'll let our CDI experts decide.
My guess is that doing a rest call from a postconstruct method is not correct

psini · 2023-12-29T15:33:03Z

I could agree with you about the rest call but postcontruct could be used also to do heavy work and in the meanwhile other thread could not instantiate other beans that seams a little bit overkilling . In any case without the optimization in place quarkus works great . I'm only worried if there are any other case of some loop that could create a deadlock like this when different thread are intializing the beans.

Ladicek · 2024-01-02T10:38:38Z

CC @mkouba, this is an interesting case :-)

mkouba · 2024-01-02T11:28:34Z

Hm, so if I understand it correctly the problem here is that the rest client call offloads the execution to a different thread and so the shared ReentrantLock does not work correctly.

Although the contract of the logic in a @PostConstruct callback is not defined I would try to avoid this kind of initialization if possible. In general, I recommend users to limit the logic in the @PostConstruct/@PreDestroy callbacks to "local resources" of the bean, i.e. avoid calling the dependencies completely. It may seem too restrictive but it's a safe approach.

A possible solution is have a lock inside Default_jakarta_enterprise_context_ApplicationScoped_ContextInstances for each kind of bean and not one for everything.

Yes, this should work.

- fixes quarkusio#37958

- use a separate lock for each bean in the generated ContextInstances; fixes quarkusio#37958 - remove ContextInstances#forEach() and ContextInstances#clear() - optimize the generated ContextInstances to significantly reduce the size of the generated bytecode

- fixes quarkusio#37958 and quarkusio#38040 - use a separate lock for each bean in the generated ContextInstances - replace ContextInstances#forEach() and ContextInstances#clear() with ContextInstances#removeEach() - optimize the generated ContextInstances to significantly reduce the size of the generated bytecode

teimyBr · 2024-01-05T13:01:52Z

we have the same issue with Thread[vert.x-eventloop-thread-6,5,main] has been blocked

Updating 3 different Service from Quarkus 3.5.3/3.6.0 to 3.6.1/3.6.2/3.6.3/3.6.4

mkouba · 2024-01-05T13:35:35Z

For the record - another solution is to use a @Startup method or a StartupEvent observer method instead of bean constructor/@PostConstruct callback. See #38040 (comment) for an example.

psini added the kind/bug Something isn't working label Dec 29, 2023

quarkus-bot bot added the area/vertx label Dec 29, 2023

sberyozkin added area/security triage/needs-reproducer We are waiting for a reproducer. area/arc Issue related to ARC (dependency injection) and removed area/vertx labels Dec 29, 2023

sberyozkin removed the area/security label Dec 29, 2023

mkouba added a commit to mkouba/quarkus that referenced this issue Jan 2, 2024

ArC: use a separate lock for each bean in generated ContextInstances

728ed08

- fixes quarkusio#37958

mkouba mentioned this issue Jan 2, 2024

ArC: use a separate lock for each bean in generated ContextInstances #37982

Merged

geoand removed the triage/needs-reproducer We are waiting for a reproducer. label Jan 3, 2024

mkouba mentioned this issue Jan 5, 2024

Thread locked with optimize-contexts and @Startup bean #38040

Closed

mkouba closed this as completed in #37982 Jan 5, 2024

quarkus-bot bot added this to the 3.7 - main milestone Jan 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrading from 3.5.3 to 3.6.1 ,3.6.3,3.6.4 Quakust test will not start with Thread Thread[vert.x-eventloop-thread-6,5,main] has been blocked for 2460 m #37958

Upgrading from 3.5.3 to 3.6.1 ,3.6.3,3.6.4 Quakust test will not start with Thread Thread[vert.x-eventloop-thread-6,5,main] has been blocked for 2460 m #37958

psini commented Dec 29, 2023 •

edited by geoand

Loading

psini commented Dec 29, 2023 •

edited

Loading

sberyozkin commented Dec 29, 2023

psini commented Dec 29, 2023

psini commented Dec 29, 2023 •

edited

Loading

sberyozkin commented Dec 29, 2023

psini commented Dec 29, 2023

Ladicek commented Jan 2, 2024

mkouba commented Jan 2, 2024

teimyBr commented Jan 5, 2024

mkouba commented Jan 5, 2024

Upgrading from 3.5.3 to 3.6.1 ,3.6.3,3.6.4 Quakust test will not start with Thread Thread[vert.x-eventloop-thread-6,5,main] has been blocked for 2460 m #37958

Upgrading from 3.5.3 to 3.6.1 ,3.6.3,3.6.4 Quakust test will not start with Thread Thread[vert.x-eventloop-thread-6,5,main] has been blocked for 2460 m #37958

Comments

psini commented Dec 29, 2023 • edited by geoand Loading

Describe the bug

Expected behavior

Actual behavior

How to Reproduce?

Output of uname -a or ver

Output of java -version

Quarkus version or git rev

Build tool (ie. output of mvnw --version or gradlew --version)

Additional information

psini commented Dec 29, 2023 • edited Loading

sberyozkin commented Dec 29, 2023

psini commented Dec 29, 2023

psini commented Dec 29, 2023 • edited Loading

sberyozkin commented Dec 29, 2023

psini commented Dec 29, 2023

Ladicek commented Jan 2, 2024

mkouba commented Jan 2, 2024

teimyBr commented Jan 5, 2024

mkouba commented Jan 5, 2024

psini commented Dec 29, 2023 •

edited by geoand

Loading

Output of `uname -a` or `ver`

Output of `java -version`

Build tool (ie. output of `mvnw --version` or `gradlew --version`)

psini commented Dec 29, 2023 •

edited

Loading

psini commented Dec 29, 2023 •

edited

Loading