Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Mongodb health check fails on Quarkus 3.5.1 #36977

Closed
computerlove opened this issue Nov 9, 2023 · 10 comments
Closed

Mongodb health check fails on Quarkus 3.5.1 #36977

computerlove opened this issue Nov 9, 2023 · 10 comments
Labels
area/health area/mongodb area/smallrye kind/bug Something isn't working

Comments

@computerlove
Copy link
Contributor

computerlove commented Nov 9, 2023 •
edited
Loading

Describe the bug

When we upgraded Quarkus from 3.5.0 to 3.5.1 /q/health fails with

2023-11-09T13:36:51.833+0100  ERROR io.smallrye.health SRHCK01000: Error processing Health Checks: java.lang.IllegalStateException: The current thread cannot be blocked: vert.x-eventloop-thread-1
	at io.smallrye.mutiny.operators.uni.UniBlockingAwait.await(UniBlockingAwait.java:30)
	at io.smallrye.mutiny.groups.UniAwait.atMost(UniAwait.java:65)
	at io.smallrye.mutiny.groups.UniAwait.indefinitely(UniAwait.java:46)
	at io.quarkus.mongodb.health.MongoHealthCheck.call(MongoHealthCheck.java:152)
	at io.quarkus.mongodb.health.MongoHealthCheck_ClientProxy.call(Unknown Source)
	at io.smallrye.context.impl.wrappers.SlowContextualSupplier.get(SlowContextualSupplier.java:21)
	at io.smallrye.mutiny.operators.uni.builders.UniCreateFromItemSupplier.subscribe(UniCreateFromItemSupplier.java:28)
	at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
	at io.smallrye.mutiny.operators.uni.UniOnFailureFlatMap.subscribe(UniOnFailureFlatMap.java:31)
	at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
	at io.smallrye.mutiny.operators.uni.UniOnItemTransform.subscribe(UniOnItemTransform.java:22)
	at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
	at io.smallrye.mutiny.operators.uni.UniAndCombination$UniHandler.subscribe(UniAndCombination.java:233)
	at io.smallrye.mutiny.operators.uni.UniAndCombination$AndSupervisor.run(UniAndCombination.java:87)
	at io.smallrye.mutiny.operators.uni.UniAndCombination.subscribe(UniAndCombination.java:53)
	at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
	at io.smallrye.mutiny.operators.uni.UniEmitOn.subscribe(UniEmitOn.java:22)
	at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
	at io.smallrye.mutiny.groups.UniSubscribe.withSubscriber(UniSubscribe.java:51)
	at io.smallrye.mutiny.groups.UniSubscribe.with(UniSubscribe.java:133)
	at io.smallrye.mutiny.groups.UniSubscribe.with(UniSubscribe.java:151)
	at io.quarkus.smallrye.health.runtime.SmallRyeHealthHandlerBase.doHandle(SmallRyeHealthHandlerBase.java:50)
	at io.quarkus.smallrye.health.runtime.SmallRyeHealthHandlerBase.handle(SmallRyeHealthHandlerBase.java:35)
	at io.quarkus.smallrye.health.runtime.SmallRyeHealthHandler.handle(SmallRyeHealthHandler.java:8)
	at io.quarkus.smallrye.health.runtime.SmallRyeHealthHandlerBase.handle(SmallRyeHealthHandlerBase.java:23)
	at io.vertx.ext.web.impl.RouteState.handleContext(RouteState.java:1286)
	at io.vertx.ext.web.impl.RoutingContextImplBase.iterateNext(RoutingContextImplBase.java:177)
	at io.vertx.ext.web.impl.RoutingContextWrapper.next(RoutingContextWrapper.java:200)
	at io.vertx.ext.web.impl.RouterImpl.handleContext(RouterImpl.java:250)
	at io.vertx.ext.web.impl.RouteState.handleContext(RouteState.java:1286)
	at io.vertx.ext.web.impl.RoutingContextImplBase.iterateNext(RoutingContextImplBase.java:177)
	at io.vertx.ext.web.impl.RoutingContextImpl.next(RoutingContextImpl.java:144)
	at io.quarkus.vertx.http.runtime.options.HttpServerCommonHandlers$1.handle(HttpServerCommonHandlers.java:59)
	at io.quarkus.vertx.http.runtime.options.HttpServerCommonHandlers$1.handle(HttpServerCommonHandlers.java:37)
	at io.vertx.ext.web.impl.RouteState.handleContext(RouteState.java:1286)
	at io.vertx.ext.web.impl.RoutingContextImplBase.iterateNext(RoutingContextImplBase.java:177)
	at io.vertx.ext.web.impl.RoutingContextImpl.next(RoutingContextImpl.java:144)
	at io.quarkus.vertx.http.runtime.security.AbstractHttpAuthorizer.doPermissionCheck(AbstractHttpAuthorizer.java:96)
	at io.quarkus.vertx.http.runtime.security.AbstractHttpAuthorizer.checkPermission(AbstractHttpAuthorizer.java:79)
	at io.quarkus.vertx.http.runtime.security.HttpAuthorizer.checkPermission(HttpAuthorizer.java:16)
	at io.quarkus.vertx.http.runtime.security.HttpSecurityRecorder$2.handle(HttpSecurityRecorder.java:86)
	at io.quarkus.vertx.http.runtime.security.HttpSecurityRecorder$2.handle(HttpSecurityRecorder.java:78)
	at io.vertx.ext.web.impl.RouteState.handleContext(RouteState.java:1286)
	at io.vertx.ext.web.impl.RoutingContextImplBase.iterateNext(RoutingContextImplBase.java:177)
	at io.vertx.ext.web.impl.RoutingContextImpl.next(RoutingContextImpl.java:144)
	at io.quarkus.resteasy.reactive.server.runtime.ResteasyReactiveRecorder$13.handle(ResteasyReactiveRecorder.java:339)
	at io.quarkus.resteasy.reactive.server.runtime.ResteasyReactiveRecorder$13.handle(ResteasyReactiveRecorder.java:332)
	at io.vertx.ext.web.impl.RouteState.handleContext(RouteState.java:1286)
	at io.vertx.ext.web.impl.RoutingContextImplBase.iterateNext(RoutingContextImplBase.java:177)
	at io.vertx.ext.web.impl.RoutingContextImpl.next(RoutingContextImpl.java:144)
	at io.quarkus.vertx.http.runtime.security.HttpSecurityRecorder$AbstractAuthenticationHandler$3$1.onItem(HttpSecurityRecorder.java:455)
	at io.quarkus.vertx.http.runtime.security.HttpSecurityRecorder$AbstractAuthenticationHandler$3$1.onItem(HttpSecurityRecorder.java:445)
	at io.smallrye.mutiny.operators.uni.UniOnItemTransformToUni$UniOnItemTransformToUniProcessor.onItem(UniOnItemTransformToUni.java:60)
	at io.smallrye.mutiny.operators.uni.builders.DefaultUniEmitter.complete(DefaultUniEmitter.java:37)
	at io.smallrye.mutiny.groups.UniOnNull.lambda$failWith$1(UniOnNull.java:46)
	at io.smallrye.context.impl.wrappers.SlowContextualBiConsumer.accept(SlowContextualBiConsumer.java:21)
	at io.smallrye.mutiny.groups.UniOnItem.lambda$transformToUni$4(UniOnItem.java:177)
	at io.smallrye.context.impl.wrappers.SlowContextualConsumer.accept(SlowContextualConsumer.java:21)
	at io.smallrye.mutiny.operators.uni.builders.UniCreateWithEmitter.subscribe(UniCreateWithEmitter.java:22)
	at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
	at io.smallrye.mutiny.operators.uni.UniOnItemTransformToUni$UniOnItemTransformToUniProcessor.performInnerSubscription(UniOnItemTransformToUni.java:81)
	at io.smallrye.mutiny.operators.uni.UniOnItemTransformToUni$UniOnItemTransformToUniProcessor.onItem(UniOnItemTransformToUni.java:57)
	at io.smallrye.mutiny.operators.uni.builders.UniCreateFromKnownItem$KnownItemSubscription.forward(UniCreateFromKnownItem.java:38)
	at io.smallrye.mutiny.operators.uni.builders.UniCreateFromKnownItem.subscribe(UniCreateFromKnownItem.java:23)
	at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
	at io.smallrye.mutiny.operators.uni.UniOnItemTransformToUni.subscribe(UniOnItemTransformToUni.java:25)
	at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
	at io.smallrye.mutiny.groups.UniSubscribe.withSubscriber(UniSubscribe.java:51)
	at io.quarkus.vertx.http.runtime.security.HttpSecurityRecorder$AbstractAuthenticationHandler$3.onItem(HttpSecurityRecorder.java:445)
	at io.quarkus.vertx.http.runtime.security.HttpSecurityRecorder$AbstractAuthenticationHandler$3.onItem(HttpSecurityRecorder.java:431)
	at io.smallrye.mutiny.operators.uni.UniMemoizeOp.forwardTo(UniMemoizeOp.java:123)
	at io.smallrye.mutiny.operators.uni.UniMemoizeOp.notifyAwaiters(UniMemoizeOp.java:135)
	at io.smallrye.mutiny.operators.uni.UniMemoizeOp.onItem(UniMemoizeOp.java:97)
	at io.smallrye.mutiny.operators.uni.UniOnItemTransformToUni$UniOnItemTransformToUniProcessor.onItem(UniOnItemTransformToUni.java:60)
	at io.smallrye.mutiny.operators.uni.builders.UniCreateFromItemSupplier.subscribe(UniCreateFromItemSupplier.java:29)
	at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
	at io.smallrye.mutiny.operators.uni.UniOnItemTransformToUni$UniOnItemTransformToUniProcessor.performInnerSubscription(UniOnItemTransformToUni.java:81)
	at io.smallrye.mutiny.operators.uni.UniOnItemTransformToUni$UniOnItemTransformToUniProcessor.onItem(UniOnItemTransformToUni.java:57)
	at io.smallrye.mutiny.operators.uni.UniOnItemTransform$UniOnItemTransformProcessor.onItem(UniOnItemTransform.java:43)
	at io.smallrye.mutiny.operators.uni.UniOnItemTransform$UniOnItemTransformProcessor.onItem(UniOnItemTransform.java:43)
	at io.smallrye.mutiny.operators.uni.builders.UniCreateFromKnownItem$KnownItemSubscription.forward(UniCreateFromKnownItem.java:38)
	at io.smallrye.mutiny.operators.uni.builders.UniCreateFromKnownItem.subscribe(UniCreateFromKnownItem.java:23)
	at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
	at io.smallrye.mutiny.operators.uni.UniOnItemTransform.subscribe(UniOnItemTransform.java:22)
	at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
	at io.smallrye.mutiny.operators.uni.UniOnItemTransform.subscribe(UniOnItemTransform.java:22)
	at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
	at io.smallrye.mutiny.operators.uni.UniOnItemTransformToUni.subscribe(UniOnItemTransformToUni.java:25)
	at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
	at io.smallrye.mutiny.groups.UniSubscribe.withSubscriber(UniSubscribe.java:51)
	at io.smallrye.mutiny.operators.uni.UniMemoizeOp.subscribe(UniMemoizeOp.java:59)
	at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
	at io.smallrye.mutiny.groups.UniSubscribe.withSubscriber(UniSubscribe.java:51)
	at io.quarkus.vertx.http.runtime.security.HttpSecurityRecorder$AbstractAuthenticationHandler.handle(HttpSecurityRecorder.java:431)
	at io.quarkus.vertx.http.runtime.security.HttpSecurityRecorder$AbstractAuthenticationHandler.handle(HttpSecurityRecorder.java:385)
	at io.vertx.ext.web.impl.RouteState.handleContext(RouteState.java:1286)
	at io.vertx.ext.web.impl.RoutingContextImplBase.iterateNext(RoutingContextImplBase.java:177)
	at io.vertx.ext.web.impl.RoutingContextImpl.next(RoutingContextImpl.java:144)
	at io.vertx.ext.web.impl.RouterImpl.handle(RouterImpl.java:68)
	at io.vertx.ext.web.impl.RouterImpl.handle(RouterImpl.java:37)
	at io.quarkus.vertx.http.runtime.options.HttpServerCommonHandlers$3.handle(HttpServerCommonHandlers.java:100)
	at io.quarkus.vertx.http.runtime.options.HttpServerCommonHandlers$3.handle(HttpServerCommonHandlers.java:97)
	at io.quarkus.vertx.http.runtime.filters.GracefulShutdownFilter.handle(GracefulShutdownFilter.java:51)
	at io.quarkus.vertx.http.runtime.filters.GracefulShutdownFilter.handle(GracefulShutdownFilter.java:14)
	at io.quarkus.vertx.http.runtime.VertxHttpRecorder$7.handle(VertxHttpRecorder.java:500)
	at io.quarkus.vertx.http.runtime.VertxHttpRecorder$7.handle(VertxHttpRecorder.java:497)
	at io.quarkus.vertx.http.runtime.options.HttpServerCommonHandlers$2.handle(HttpServerCommonHandlers.java:83)
	at io.quarkus.vertx.http.runtime.options.HttpServerCommonHandlers$2.handle(HttpServerCommonHandlers.java:66)
	at io.quarkus.vertx.http.runtime.VertxHttpRecorder$1.handle(VertxHttpRecorder.java:162)
	at io.quarkus.vertx.http.runtime.VertxHttpRecorder$1.handle(VertxHttpRecorder.java:138)
	at io.vertx.core.impl.EventLoopContext.emit(EventLoopContext.java:55)
	at io.vertx.core.impl.DuplicatedContext.emit(DuplicatedContext.java:179)
	at io.vertx.core.http.impl.Http2ServerRequest.dispatch(Http2ServerRequest.java:108)
	at io.vertx.core.http.impl.Http2ServerStream.onHeaders(Http2ServerStream.java:116)
	at io.vertx.core.http.impl.Http2ServerConnection.onHeadersRead(Http2ServerConnection.java:184)
	at io.vertx.core.http.impl.Http2ConnectionBase.onHeadersRead(Http2ConnectionBase.java:208)
	at io.vertx.core.http.impl.Http2ServerConnection.onHeadersRead(Http2ServerConnection.java:45)
	at io.netty.handler.codec.http2.Http2FrameListenerDecorator.onHeadersRead(Http2FrameListenerDecorator.java:48)
	at io.netty.handler.codec.http2.Http2FrameListenerDecorator.onHeadersRead(Http2FrameListenerDecorator.java:48)
	at io.netty.handler.codec.http2.Http2EmptyDataFrameListener.onHeadersRead(Http2EmptyDataFrameListener.java:63)
	at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onHeadersRead(DefaultHttp2ConnectionDecoder.java:409)
	at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onHeadersRead(DefaultHttp2ConnectionDecoder.java:337)
	at io.netty.handler.codec.http2.DefaultHttp2FrameReader$2.processFragment(DefaultHttp2FrameReader.java:476)
	at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readHeadersFrame(DefaultHttp2FrameReader.java:484)
	at io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:253)
	at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:159)
	at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:173)
	at io.netty.handler.codec.http2.DecoratingHttp2ConnectionDecoder.decodeFrame(DecoratingHttp2ConnectionDecoder.java:63)
	at io.netty.handler.codec.http2.DecoratingHttp2ConnectionDecoder.decodeFrame(DecoratingHttp2ConnectionDecoder.java:63)
	at io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:393)
	at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:453)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
	at io.vertx.core.http.impl.VertxHttp2ConnectionHandler.channelRead(VertxHttp2ConnectionHandler.java:415)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1471)
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1334)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1383)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:1583)

the behavior is not present in quarkus:dev

Expected behavior

Health check does not fail because of IllegalStateException: The current thread cannot be blocked

Actual behavior

Health check does fail because of IllegalStateException: The current thread cannot be blocked

How to Reproduce?

Have not managed to create a reproducer.
In the environment this is running the proxy in front of the application seems to connect with http2 and keeping the connection alive.

Current attempts produce the following stack: (forcing return false in Infrastructure.canCallerThreadBeBlocked())

        2023-11-09 14:56:31,581 ERROR [io.sma.health] (executor-thread-1) SRHCK01000: Error processing Health Checks: java.lang.IllegalStateException: The current thread cannot be blocked: executor-thread-1
        at io.smallrye.mutiny.operators.uni.UniBlockingAwait.await(UniBlockingAwait.java:30)
        at io.smallrye.mutiny.groups.UniAwait.atMost(UniAwait.java:65)
        at io.smallrye.mutiny.groups.UniAwait.indefinitely(UniAwait.java:46)
        at io.quarkus.mongodb.health.MongoHealthCheck.call(MongoHealthCheck.java:152)
        at io.quarkus.mongodb.health.MongoHealthCheck_ClientProxy.call(Unknown Source)
        at io.smallrye.context.impl.wrappers.SlowContextualSupplier.get(SlowContextualSupplier.java:21)
        at io.smallrye.mutiny.operators.uni.builders.UniCreateFromItemSupplier.subscribe(UniCreateFromItemSupplier.java:28)
        at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
        at io.smallrye.mutiny.operators.uni.UniOnFailureFlatMap.subscribe(UniOnFailureFlatMap.java:31)
        at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
        at io.smallrye.mutiny.operators.uni.UniOnItemTransform.subscribe(UniOnItemTransform.java:22)
        at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
        at io.smallrye.mutiny.operators.uni.UniRunSubscribeOn.lambda$subscribe$0(UniRunSubscribeOn.java:27)
        at io.smallrye.mutiny.vertx.MutinyHelper.lambda$blockingExecutor$6(MutinyHelper.java:62)
        at io.vertx.core.impl.ContextBase.lambda$executeBlocking$1(ContextBase.java:180)
        at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:277)
        at io.vertx.core.impl.ContextBase.lambda$internalExecuteBlocking$2(ContextBase.java:199)
        at io.quarkus.vertx.core.runtime.VertxCoreRecorder$14.runWith(VertxCoreRecorder.java:582)
        at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2513)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1538)
        at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:29)
        at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:29)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:1583)

Output of uname -a or ver

No response

Output of java -version

OpenJDK Runtime Environment (build 21+35-2513)

Quarkus version or git rev

3.5.1

Build tool (ie. output of mvnw --version or gradlew --version)

mvn

Additional information

No response
@computerlove computerlove added the kind/bug Something isn't working label Nov 9, 2023
@quarkus-bot quarkus-bot
Copy link

quarkus-bot bot commented Nov 9, 2023

/cc @evanchooly (mongodb), @jmartisk (health), @loicmathieu (mongodb), @xstefank (health)

@loicmathieu
Copy link
Contributor

@geoand @jmartisk as far as I know health check are runs in blocking threads by default so I have no idea why this errors pops up on 3.5.1. Do you know if something changed?

@geoand
Copy link
Contributor

geoand commented Nov 10, 2023

No idea, I have not been in the loop that past few weeks.

@xstefank any ideas?

@gsmet
Copy link
Member

gsmet commented Nov 10, 2023

My guess is that it's related to #36434 that was backported in 3.5.1.

That is very problematic for sure. @cescoffier does it ring a bell?

@cescoffier
Copy link
Member

Hum, that's exactly what I explained. We cannot rush such a change, it has consequences.

We would need to check the mongo health check and see why it's blocking (why using an async signature).

@cescoffier
Copy link
Member

The signature is synchronous, so that check should not be called on an event loop.

@gsmet
Copy link
Member

gsmet commented Nov 10, 2023

Okay, I will revert the change then.

This was referenced Nov 10, 2023
Merged
Merged
@xstefank
Copy link
Member

xstefank commented Nov 10, 2023 •
edited
Loading

OK, I agree reverting is the best way now, I need to investigate why this happens.

@gsmet
Copy link
Member

gsmet commented Nov 10, 2023

I reverted the culprit: #37001 .

I will release 3.5.2 probably on Tuesday next week.

@gsmet gsmet closed this as completed Nov 10, 2023
@ahus1 ahus1 mentioned this issue Nov 24, 2023
Closed
@xstefank xstefank mentioned this issue Nov 28, 2023
Merged
@xstefank
Copy link
Member

Just to complete the full picture here. After investigation with @computerlove, we found out that the issue was caused by initialization of health check Unis on the blocking thread (through scheduler) which created MongoHealthCheck Uni wrapper wrongly to run on eventloop - https://github.com/quarkusio/quarkus/pull/37352/files#diff-52ee975c6ce19b708828df815121d4e32ac83ed2c6a2c2651753f01d4999bec6L31-L32. This was fixed in the last PR and this issue is no longer reproducible.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/health area/mongodb area/smallrye kind/bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@cescoffier @computerlove @gsmet @loicmathieu @geoand @xstefank