Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Second re-authentication fails with OAUTHBEARER when default credentials are used #176

Closed
jamielwhite opened this issue Jun 6, 2024 · 5 comments

Comments

@jamielwhite
Copy link

jamielwhite commented Jun 6, 2024

This issue is similar to #143. The resolution for that issue did fix the case where the first re-authentication fails if awsRoleArn is not specified, but I've noticed our application fails to re-authenticate on the second attempt.

This occurs in a Java application, but I was also able to reproduce it with a console consumer. The consumer ran successfully for a while, but it failed to authenticate due to a SaslAuthenticationException: Session too short error once it hit the 2 hour mark. Since our default role credential would expire after one hour, this seems to happen during the second re-authentication.

kafka-console-consumer.sh --topic $TOPIC --bootstrap-server $BOOTSTRAP_SERVER --consumer.config $COMMAND_CONFIG 
...
[2024-06-06 18:09:43,453] ERROR [Consumer clientId=console-consumer, groupId=console-consumer-51258] Connection to node 1 failed authentication due to: [fe6e184c-87f2-4837-b41a-7a044a16ae5e]: Session too short (org.apache.kafka.clients.NetworkClient)
[2024-06-06 18:09:43,454] ERROR Error processing message, terminating consumer process:  (kafka.tools.ConsoleConsumer$)
org.apache.kafka.common.errors.SaslAuthenticationException: [fe6e184c-87f2-4837-b41a-7a044a16ae5e]: Session too short
[2024-06-06 18:09:43,469] ERROR [Consumer clientId=console-consumer, groupId=console-consumer-51258] Connection to node 2 failed authentication due to: [164e180f-95b2-4bec-9be6-bf39f1f77759]: Session too short (org.apache.kafka.clients.NetworkClient)
[2024-06-06 18:09:43,469] ERROR [Consumer clientId=console-consumer, groupId=console-consumer-51258] Failed to close coordinator with a timeout(ms)=30000 (org.apache.kafka.clients.consumer.internals.LegacyKafkaConsumer)
org.apache.kafka.common.errors.SaslAuthenticationException: [164e180f-95b2-4bec-9be6-bf39f1f77759]: Session too short
[2024-06-06 18:09:43,484] ERROR Unknown error when running consumer:  (kafka.tools.ConsoleConsumer$)
org.apache.kafka.common.KafkaException: Failed to close kafka consumer
	at org.apache.kafka.clients.consumer.internals.LegacyKafkaConsumer.close(LegacyKafkaConsumer.java:1163)
	at org.apache.kafka.clients.consumer.internals.LegacyKafkaConsumer.close(LegacyKafkaConsumer.java:1103)
	at org.apache.kafka.clients.consumer.internals.LegacyKafkaConsumer.close(LegacyKafkaConsumer.java:1091)
	at org.apache.kafka.clients.consumer.KafkaConsumer.close(KafkaConsumer.java:1757)
	at kafka.tools.ConsoleConsumer$ConsumerWrapper.cleanup(ConsoleConsumer.scala:488)
	at kafka.tools.ConsoleConsumer$.run(ConsoleConsumer.scala:79)
	at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:54)
	at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala)
Caused by: org.apache.kafka.common.errors.SaslAuthenticationException: [164e180f-95b2-4bec-9be6-bf39f1f77759]: Session too short

Here's the config used for this:

security.protocol=SASL_SSL
sasl.mechanism=OAUTHBEARER
sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required;
sasl.login.callback.handler.class=software.amazon.msk.auth.iam.IAMOAuthBearerLoginCallbackHandler

We have applications which specify an awsRoleArn in their sasl.jaas.config and are able to run indefinitely without this re-authentication error, so I suspect this only impacts configurations without awsRoleArn.

Kafka version: 3.7.0
aws-msk-iam-auth version: 2.0.3

@jamielwhite
Copy link
Author

jamielwhite commented Jun 20, 2024

I have some applications which provide an awsRoleArn, and these are now encountering the same SaslAuthenticationException: Session too short error after upgrading from 2.0.3 to 2.1.1. There seems to have been a regression in behavior between these versions. However, the error occurs after 1 hour instead of 15 minutes as it did in #143.

(I ended up splitting this issue out into #180 since it came up in a different version)

@sidyag
Copy link
Contributor

sidyag commented Jul 8, 2024

I think this PR should fix it: #182 Can you test with this?

@jamielwhite
Copy link
Author

Similar to this comment on #180, I don't think this will work because we aren't using awsProfileName. This issue happens specifically when there are no additional properties in our sasl.jaas.config. We're running on pods in a Kubernetes cluster with credentials provided by our service account.

@sidyag
Copy link
Contributor

sidyag commented Jul 17, 2024

Fixed in release 2.2.0

@sidyag sidyag closed this as completed Jul 17, 2024
Copy link

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants