Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[JDK 23+15] org.apache.kafka.common.security.oauthbearer.internals.OAuthBearerSaslClientCallbackHandler.handleCallback fails with UnsupportedOperationException #39634

Open
jerboaa opened this issue Mar 22, 2024 · 34 comments

Comments

@jerboaa
Copy link
Contributor

jerboaa commented Mar 22, 2024

Describe the bug

In our Mandrel native integration tests we see new failures since the JDK 23+15 ea build in quarkus-integration-test-kafka-oauth-keycloak-999-SNAPSHOT-runner tests:

2024-03-22 02:18:58,664 WARN  [org.apa.kaf.com.net.Selector] (kafka-producer-network-thread | kafka-producer-out) [Producer clientId=kafka-producer-out] Unexpected error from localhost/127.0.0.1 (channelId=-1); closing connection: java.lang.UnsupportedOperationException: getSubject is supported only if a security manager is allowed
	at java.base@23-beta/javax.security.auth.Subject.getSubject(Subject.java:347)
	at org.apache.kafka.common.security.oauthbearer.internals.OAuthBearerSaslClientCallbackHandler.handleCallback(OAuthBearerSaslClientCallbackHandler.java:99)
	at org.apache.kafka.common.security.oauthbearer.internals.OAuthBearerSaslClientCallbackHandler.handle(OAuthBearerSaslClientCallbackHandler.java:83)
	at org.apache.kafka.common.security.oauthbearer.internals.OAuthBearerSaslClient.evaluateChallenge(OAuthBearerSaslClient.java:92)
	at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.lambda$createSaslToken$1(SaslClientAuthenticator.java:535)
	at java.base@23-beta/jdk.internal.vm.ScopedValueContainer.callWithoutScope(ScopedValueContainer.java:162)
	at java.base@23-beta/jdk.internal.vm.ScopedValueContainer.call(ScopedValueContainer.java:147)
	at java.base@23-beta/java.lang.ScopedValue$Carrier.runWith(ScopedValue.java:74)
	at java.base@23-beta/java.lang.ScopedValue$Carrier.call(ScopedValue.java:419)
	at java.base@23-beta/java.lang.ScopedValue.callWhere(ScopedValue.java:588)
	at java.base@23-beta/javax.security.auth.Subject.callAs(Subject.java:439)
	at java.base@23-beta/javax.security.auth.Subject.doAs(Subject.java:614)
	at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.createSaslToken(SaslClientAuthenticator.java:535)
	at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.sendSaslClientToken(SaslClientAuthenticator.java:434)
	at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.sendInitialToken(SaslClientAuthenticator.java:333)
	at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.authenticate(SaslClientAuthenticator.java:274)
	at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:181)
	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543)
	at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
	at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:585)
	at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:349)
	at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:252)
	at java.base@23-beta/java.lang.Thread.runWith(Thread.java:1588)
	at java.base@23-beta/java.lang.Thread.run(Thread.java:1575)
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine(PlatformThreads.java:836)
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine(PlatformThreads.java:812)

See: https://github.com/graalvm/mandrel/actions/runs/8384081617/job/22961614228#step:12:968

I'm pretty sure, though, this isn't a native-only issue but can happen in JVM mode on JDK 23+15 as well.

Expected behavior

No java.lang.UnsupportedOperationException: getSubject is supported only if a security manager is allowed is being thrown.

Actual behavior

java.lang.UnsupportedOperationException is being thrown.

How to Reproduce?

Build Integration Tests - Kafka OAUTH with Keycloak native test and run test with a 23-beta+15-ea, vendor version: Mandrel-24.1.0-dev13054560 mandrel build. For example from here.

Additional information

This seems to be caused by https://bugs.openjdk.org/browse/JDK-8296244 freshly part of JDK 23+15. See the CSR on how to possibly fix it: https://bugs.openjdk.org/browse/JDK-8327134

Copy link

quarkus-bot bot commented Mar 22, 2024

/cc @Karm (mandrel), @alesj (kafka), @cescoffier (kafka), @galderz (mandrel), @ozangunalp (kafka), @pedroigor (bearer-token), @sberyozkin (bearer-token,security), @zakkak (mandrel)

@cescoffier
Copy link
Member

Unfortunately there is not much we can do. It's an issue to report in the Apache Kafka Client.

@jerboaa
Copy link
Contributor Author

jerboaa commented Mar 22, 2024

OK. Feel free to bring this to the Apache Kafka Client team.

@gsmet
Copy link
Member

gsmet commented Mar 25, 2024

Isn't the security manager supposed to go away? I'm surprised they made it mandatory for some operations?

@jerboaa
Copy link
Contributor Author

jerboaa commented Mar 25, 2024

Isn't the security manager supposed to go away? I'm surprised they made it mandatory for some operations?

Yes the security manager is going away. The reason this change was done was supporting this move. Please see the CSR of this change which should have all the info needed: https://bugs.openjdk.org/browse/JDK-8327134

It appears a possible work-around is to add -Djava.security.manager=allow. But this comes with the following caveat:

If for some reason, user cannot modify their code in time, they can add
`-Djava.security.manager=allow` on the command line as a workaround.
This allows the old implementation to be used even if a Security Manager is never set.
Please note that in a future version when the Security Manager is completely
removed this workaround would fail.

Hope that helps.

@jerboaa
Copy link
Contributor Author

jerboaa commented Apr 4, 2024

@cescoffier Is there an upstream kafka client issue for this? Who would we need to talk to to get it created? I'm hoping to get this on the radar for kafka client folks.

@cescoffier
Copy link
Member

Yes, it's an issue for Apache Kafka (the upstream project).

@jerboaa
Copy link
Contributor Author

jerboaa commented Apr 4, 2024

Understood. If possible, could somebody create an upstream issue? That tracker seems invite only and I'm not keen on creating one just for a single bug. Could somebody with an (existing) account there file one? Thanks!

https://issues.apache.org/jira/projects/KAFKA/issues

@cescoffier
Copy link
Member

@ozangunalp do you have an account?

@jerboaa let me ping Mickael Maison (working in the AMQ Streams team)

@jerboaa
Copy link
Contributor Author

jerboaa commented Apr 5, 2024

Thanks! Please let me know once somebody has created an issue.

@cescoffier
Copy link
Member

@jerboaa There is an existing ticket: https://issues.apache.org/jira/browse/KAFKA-15862.

However, it's not planned before the Q3.

@jerboaa
Copy link
Contributor Author

jerboaa commented Apr 8, 2024

@cescoffier OK, thanks. Whether or not that is a problem for quarkus is something you know better. GraalVM for JDK 23 will get out in September 2024, which is when a fix/known issue for this needs to be in place.

@cescoffier
Copy link
Member

Yeah, the timing might be problematic. Basically, if it's not fixed before September 2024, it would not be possible to run Quarkus apps using Kafka authentication in native.

Note that it's not only Quarkus, it's anything compiling Java to native using GraalVM.

@jerboaa
Copy link
Contributor Author

jerboaa commented Apr 8, 2024

Yeah, the timing might be problematic. Basically, if it's not fixed before September 2024, it would not be possible to run Quarkus apps using Kafka authentication in native.

That's my understanding. At least with GraalVM for JDK 23 (the older GraalVM for JDK 21 would still work).

@geoand
Copy link
Contributor

geoand commented Jun 6, 2024

Is this still an issue?

@jerboaa
Copy link
Contributor Author

jerboaa commented Jun 6, 2024

Yes. See graalvm/mandrel#742 (comment)

@cescoffier
Copy link
Member

The Kafka team has an issue for it too. It should land 'just in time' (September-ish)

@jerboaa
Copy link
Contributor Author

jerboaa commented Aug 22, 2024

The Kafka team has an issue for it too. It should land 'just in time' (September-ish)

@cescoffier Do you know any update on this? JDK 23 is in RC 2 and there will be a Mandrel for JDK 23 release shortly after JDK 23 GA. Should we start planning on drafting a "known issue" or the like for this, or do you think there will be a fix for it in time still.

@cescoffier
Copy link
Member

@jerboaa no news so far - there is a keep and a jira (https://issues.apache.org/jira/browse/KAFKA-15862), but nothing in motion.

@jerboaa
Copy link
Contributor Author

jerboaa commented Aug 22, 2024

Thanks. Sounds like it's going to be a known issue for the Mandrel 24.1 for JDK 23 release.

@jerboaa
Copy link
Contributor Author

jerboaa commented Sep 6, 2024

@cescoffier Is there a way to advise quarkus users about the issue with the kafka-oauth-keycloak extension when they run on JDK 23+? Could we tell them that this extension isn't working on JDK 23 perhaps pointing here?

@cescoffier
Copy link
Member

Except in the documentation, i don't think we can do much.

@ozangunalp do you think we can intercept the exception and log a more meaningful message?

@ozangunalp
Copy link
Contributor

ozangunalp commented Sep 6, 2024

Last time I checked I couldn't reproduce this on JVM mode with 23.ea.29-open (by simply running the kafka-oauth-keycloak test)

@jerboaa is there anywhere I can get an early access release Mandrel for JDK 23?

cc @cescoffier

@zakkak
Copy link
Contributor

zakkak commented Sep 6, 2024

@jerboaa is there anywhere I can get an early access release Mandrel for JDK 23?

@ozangunalp you can get the latest build from https://github.com/graalvm/mandrel/actions/runs/10722382634/artifacts/1896697043

Built in this CI run https://github.com/graalvm/mandrel/actions/runs/10722382634

@jerboaa
Copy link
Contributor Author

jerboaa commented Sep 6, 2024

Last time I checked I couldn't reproduce this on JVM mode with 23.ea.29-open (by simply running the kafka-oauth-keycloak test)

You probably need to exercise this code path:

2024-03-22 02:18:58,664 WARN  [org.apa.kaf.com.net.Selector] (kafka-producer-network-thread | kafka-producer-out) [Producer clientId=kafka-producer-out] Unexpected error from localhost/127.0.0.1 (channelId=-1); closing connection: java.lang.UnsupportedOperationException: getSubject is supported only if a security manager is allowed
	at java.base@23-beta/javax.security.auth.Subject.getSubject(Subject.java:347)
	at org.apache.kafka.common.security.oauthbearer.internals.OAuthBearerSaslClientCallbackHandler.handleCallback(OAuthBearerSaslClientCallbackHandler.java:99)
	at org.apache.kafka.common.security.oauthbearer.internals.OAuthBearerSaslClientCallbackHandler.handle(OAuthBearerSaslClientCallbackHandler.java:83)
	at org.apache.kafka.common.security.oauthbearer.internals.OAuthBearerSaslClient.evaluateChallenge(OAuthBearerSaslClient.java:92)
	at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.lambda$createSaslToken$1(SaslClientAuthenticator.java:535)

Not sure if the test does that in all cases.

@ozangunalp
Copy link
Contributor

ozangunalp commented Sep 6, 2024

That's not about the test code path. I had a -Djava.security.manager=allow on that test.
I can certainly reproduce it on jdk 23.

@jerboaa
Copy link
Contributor Author

jerboaa commented Sep 6, 2024

I had a -Djava.security.manager=allow on that test.

That would explain it yes.

zakkak added a commit to zakkak/quarkus that referenced this issue Sep 26, 2024
The test is known to fail for quite some time generating a lot of CI
noise. Let's skip it till
quarkusio#39634 is resolved.
zakkak added a commit to zakkak/quarkus that referenced this issue Sep 26, 2024
The test is known to fail for quite some time generating a lot of CI
noise. Let's skip it till
quarkusio#39634 is resolved.
zakkak added a commit to zakkak/quarkus that referenced this issue Sep 26, 2024
The test is known to fail for quite some time generating a lot of CI
noise. Let's skip it till
quarkusio#39634 is resolved.
@cescoffier
Copy link
Member

The root cause should be fixed in Kafka 4. Until that, we would need the flag (as shown by Ozan)

@jerboaa
Copy link
Contributor Author

jerboaa commented Nov 6, 2024

The root cause should be fixed in Kafka 4. Until that, we would need the flag (as shown by Ozan)

Note that -Djava.security.manager=allow will unconditional throw an exception once JEP 486 integrates for JDK 24. Is there a plan to move to Kafka 4 in time for JDK 25, say?

@cescoffier
Copy link
Member

Hard to say, it will be tricky.
Kafka 4 is not released yet, hopefully this year. But even with that we would need to do some integration work to continue supporting both Kafka 3 and 4 for a bit.

@drakgoku
Copy link

drakgoku commented Nov 10, 2024

This happens because from JDK 17 onwards, the Security Manager is deprecated and in JDK 23 there are significant changes in how security is handled.

It seems that all of us who are on JDK 23 or updated projects have been stuck because of Kafka and have posted it in the wrong place. It's not Quarkus' fault but Kafka's. But since this problem doesn't exist on the Internet, then we come to post it here.
So this post should be redirected to Kafka or Java or even spring.

This is going to be forgotten. You have to link to Java, Kafka and Spring boot.

This mainly has to be linked to Kafka/Java to be fixed.

@zakkak
Copy link
Contributor

zakkak commented Nov 15, 2024

So this post should be redirected to Kafka or Java or even spring.
...
This mainly has to be linked to Kafka/Java to be fixed.

@drakgoku this is already tracked in Kafka as https://issues.apache.org/jira/browse/KAFKA-15862 and https://cwiki.apache.org/confluence/display/KAFKA/KIP-1006%3A+Remove+SecurityManager+Support

As mentioned in spring-projects/spring-kafka#3619 (comment) it would be best if people affected by it could somehow reach the Kafka team.

I don't have an account on their issue tracker and I can't really see the comments but it looks like the issue was last updated on 30/Sep/24, and according to @cescoffier it will be fixed in version 4.0.0

Update: You can also try reaching out the Kafka team through their mailing lists https://kafka.apache.org/contact.html

@anshupitlia
Copy link

Isn't the security manager supposed to go away? I'm surprised they made it mandatory for some operations?

Yes the security manager is going away. The reason this change was done was supporting this move. Please see the CSR of this change which should have all the info needed: https://bugs.openjdk.org/browse/JDK-8327134

It appears a possible work-around is to add -Djava.security.manager=allow. But this comes with the following caveat:

If for some reason, user cannot modify their code in time, they can add
`-Djava.security.manager=allow` on the command line as a workaround.
This allows the old implementation to be used even if a Security Manager is never set.
Please note that in a future version when the Security Manager is completely
removed this workaround would fail.

Hope that helps.

Tried this config, still not working. :(

anshupitlia added a commit to anshupitlia/product-information-system that referenced this issue Nov 16, 2024
…s not have security manager which is needed by Kafka. More details here - quarkusio/quarkus#39634.
bschuhmann pushed a commit to bschuhmann/quarkus that referenced this issue Nov 16, 2024
The test is known to fail for quite some time generating a lot of CI
noise. Let's skip it till
quarkusio#39634 is resolved.
@vsevel
Copy link
Contributor

vsevel commented Dec 16, 2024

we have been hit by this issue as well trying to upgrade to 23 to solve a graalvm issue fixed in 21, as suggested in a jobrunr issue discussed in this zulip thread.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

10 participants