IAMClientCallbackHandler could not be found #96

tzookb · 2023-01-06T19:20:50Z

keep getting this error:

[2023-01-06 14:13:05,494] ERROR Unknown error when running consumer:  (kafka.tools.ConsoleConsumer$:43)
org.apache.kafka.common.config.ConfigException: Invalid value software.amazon.msk.auth.iam.IAMClientCallbackHandler for configuration sasl.client.callback.handler.class: Class software.amazon.msk.auth.iam.IAMClientCallbackHandler could not be found.
	at org.apache.kafka.common.config.ConfigDef.parseType(ConfigDef.java:758)
	at org.apache.kafka.common.config.ConfigDef.parseValue(ConfigDef.java:504)
	at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:497)
	at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:113)
	at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:133)
	at org.apache.kafka.clients.consumer.ConsumerConfig.<init>(ConsumerConfig.java:630)
	at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:664)
	at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:645)
	at kafka.tools.ConsoleConsumer$.run(ConsoleConsumer.scala:65)
	at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:52)
	at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala)

I saw it was noted in the troubleshooting
and it says to:

That means that this aws-msk-iam-auth library is not on the classpath of the Kafka client. Please add the aws-msk-iam-auth library to the classpath and try again.

I tried to do it with this command below:
my command

env CLASSPATH=/Users/tzookb/kafka-connect/aws-msk-iam-auth-1.1.1-all.jar bash kafka-avro-console-consumer \
   --topic mytopic \
   --bootstrap-server "myserver.com:9098,myserver2.com:9098,myserver3.com:9098" \
   --from-beginning \
   --consumer.config /Users/tzookb/kafka-connect/service-desk-gamma-client.properties \
   --property schema.registry.url=http://localhost:8080/schemaRegistry \
   --property print.key=true \
   --property print.value=true \
   --value-deserializer io.confluent.kafka.serializers.KafkaAvroDeserializer \
   --key-deserializer org.apache.kafka.common.serialization.StringDeserializer

but I still get the same error.
anything Im missing?

The text was updated successfully, but these errors were encountered:

Pablo-platypus · 2023-01-19T18:09:49Z

Depending on how you are running your consumer, you need to add the aws-msk-iam-auth library to the classpath differently. Some options are:

(1) add it to the project dependencies using for example the maven repo in the ReadMe then build the project.
(2) add it to the java env classpath like you are doing in your example.
(3) add it to external jar files manifest files.

About (3), running an already existing jar file will probably override the env classpath with the internals META-INF/MANIFEST.mf file class-path, so you need to edit it to let it know where to find this external jar library.

From https://javarevisited.blogspot.com/2011/01/how-classpath-work-in-java.html#axzz7qlxl0T5c
In this case, you can set your Java classpath in the META-INF/MANIFEST.MF file by using the Class-Path attribute. In short Class-path attribute in manifest file overrides classpath specified by -cp, -classpath or CLASSPATH environment variable.

How to edit an existing jar file's MANIFEST:
https://stackoverflow.com/questions/22682809/how-can-i-change-manifest-file-in-java

Extract the manifest:
jar xvf MyProject.jar META-INF/MANIFEST.MF
Edit the manifest
Reinsert the edited manifest:
jar uvf MyProject.jar META-INF/MANIFEST.MF

For 3. I actually use (add the M option)
jar uMvf MyProject.jar META-INF/MANIFEST.MF

jbarrow6 · 2023-04-17T22:35:17Z

I was able to solve this issue by copying aws-msk-iam-auth-1.1.1-all.jar into kafka_2.12-2.8.1/libs/

jyoti-c · 2023-05-28T06:22:19Z

I was able to solve this issue by copying aws-msk-iam-auth-1.1.1-all.jar into kafka_2.12-2.8.1/libs/

thanks, worked for me too

singhbaljit · 2023-06-30T16:17:45Z

It doesn't seem to be working using kafka_2.13-2.8.2; works with kafka_2.12-2.8.2.

itconsultor · 2023-11-09T21:41:43Z

Thanks, it worked for me too

Devarsh23 · 2024-02-21T17:35:53Z

We are facing the same issue when we are trying to use the IAM authentication from kafka connect.
We are getting Invalid value software.amazon.msk.auth.iam.IAMClientCallbackHandler for configuration sasl.client.callback.handler.class: Class software.amazon.msk.auth.iam.IAMClientCallbackHandler could not be found.

We are using the confluentinc/cp-kafka-connect-base image with version 7.5.0.
So, while creating our image, which will use the IAM auth, we are using the latest jar of version 2.0.3.
In this image, we are copying this jar into the following path

/usr/share/java/
/etc-kafka-connect/jars
/usr/share/java/cp-base-new

We are running our kafka-connect in the distributed mode.
We have also set the CLASSPATH in the docker file as an environment variable to this path /usr/share/java/.
Also, we have set the plugin path for our kafka-connect to /usr/share/java,/usr/share/confluent-hub-components
But after all this, we still get the same error mentioned above.
Do you guys have any idea on how to resolve this issue?

idanna · 2024-03-13T13:54:45Z

@Devarsh23
Doing the following while building the connect image worked for me:

FROM confluentinc/cp-kafka-connect-base:latest
COPY aws-msk-iam-auth-2.0.3-all.jar /usr/share/java/kafka

sabper · 2024-08-06T05:03:48Z

we are using the confluentinc/cp-kafka-connect-base image with version 7.7.0.

following work for me:

CLASSPATH: /usr/share/java/cp-base-new/aws-msk-iam-auth-2.2.0-all.jar
aws-msk-iam-auth-2.2.0-all.jar path : /usr/share/java/cp-base-new/
- /usr/share/java/kafka/ : IAMClientCallbackHandler could not be found.
- /usr/share/java/cp-base-new/ : work

DGondhiya · 2024-08-08T10:03:57Z

we have tried this path (/usr/share/java/cp-base-new/) in distroless and scratch image but getting the same error. Any lead on classpath, where can we put this explicitly?

yanivkrol · 2024-08-12T11:24:40Z

Hi, I'm trying to use kafka cli with AWS_MSK_IAM mechanism and I get the same error.

I put the jar aws-msk-iam-auth-2.2.0-all.jar in /opt/homebrew/Cellar/kafka/3.7.1/libexec/libsz.

Any ideas?

command:
kafka-topics --bootstrap-server server --describe --topic topic --command-config client.properties

Also doesn't work If install kafka directly :

curl https://archive.apache.org/dist/kafka/2.8.1/kafka_2.12-2.8.1.tgz -o kafka_2.12-2.8.1.tgz

curl https://github.com/aws/aws-msk-iam-auth/releases/download/v1.1.1/aws-msk-iam-auth-1.1.1-all.jar -o aws-msk-iam-auth-1.1.1-all.jar

cp aws-msk-iam-auth-1.1.1-all.jar kafka_2.12-2.8.1/libs/

kafka_2.12-2.8.1/bin/kafka-topics.sh --list

Exception in thread "main" org.apache.kafka.common.config.ConfigException: Invalid value
software.amazon.msk.auth.iam.IAMClientCallbackHandler for configuration sasl.client.callback.handler.class: 
Class software.amazon.msk.auth.iam.IAMClientCallbackHandler could not be found.

kvanbrabant · 2024-08-22T14:39:31Z

We faced the same error when converting our kafka-connect cluster from TLS authentication to AWS IAM authentication.
In the end, we got it working by copying the jar file in these locations:

/usr/share/java/kafka
/usr/share/java/cp-base-new

AFAIK, the first location is where by default the classpath gets loaded for creating the admin client, serving the connector list, status, tasks, etc. through the REST endpoint. This admin client starts by reading the deployed connectors from a kafka topic and needs read access to the topic storing these connectors, so it fails when it can't authenticate.
The second location is probably added to the classpath of each connector plugin. This is needed when a sink connector needs to read from a topic or a source connector needs to write to it. Also, sink connectors need to be allowed to use the group called "connector-group" (by default).
I can't find any documentation on confluent that acknowledges this, so if anyone knows where it is, please post the link.

thranasingha · 2024-08-28T06:10:32Z

We faced the same issue when we were trying to create kafka consumer from spring boot application with plain java. We needed it to be a async task. So we used a following code snippet.

CompletableFuture.runAsync( () -> { ... });

With this asynchronus task, it will try to instantiate software.amazon.msk.auth.iam.IAMLoginModule with the class loader. Class loader which will be used for CompletableFuture.runAsync() is ForkJoinPool which is created when JVM is initialized. So it will not able to access the spring class loader which leads to have class not found exception.

Solution we found is to provide a separate executor pool which suggested in https://stackoverflow.com/questions/49113207/completablefuture-forkjoinpool-set-class-loader answer

ExecutorService pool = Executors.newFixedThreadPool(10); final CompletableFuture<String> future = CompletableFuture.supplyAsync(() -> { /* ... */ }, pool);

This executor pool solved my class loader issue.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

IAMClientCallbackHandler could not be found #96

IAMClientCallbackHandler could not be found #96

tzookb commented Jan 6, 2023

Pablo-platypus commented Jan 19, 2023 •

edited

Loading

jbarrow6 commented Apr 17, 2023

jyoti-c commented May 28, 2023

singhbaljit commented Jun 30, 2023 •

edited

Loading

itconsultor commented Nov 9, 2023

Devarsh23 commented Feb 21, 2024

idanna commented Mar 13, 2024

sabper commented Aug 6, 2024 •

edited

Loading

DGondhiya commented Aug 8, 2024 •

edited

Loading

yanivkrol commented Aug 12, 2024 •

edited

Loading

kvanbrabant commented Aug 22, 2024 •

edited

Loading

thranasingha commented Aug 28, 2024

IAMClientCallbackHandler could not be found #96

IAMClientCallbackHandler could not be found #96

Comments

tzookb commented Jan 6, 2023

Pablo-platypus commented Jan 19, 2023 • edited Loading

jbarrow6 commented Apr 17, 2023

jyoti-c commented May 28, 2023

singhbaljit commented Jun 30, 2023 • edited Loading

itconsultor commented Nov 9, 2023

Devarsh23 commented Feb 21, 2024

idanna commented Mar 13, 2024

sabper commented Aug 6, 2024 • edited Loading

DGondhiya commented Aug 8, 2024 • edited Loading

yanivkrol commented Aug 12, 2024 • edited Loading

kvanbrabant commented Aug 22, 2024 • edited Loading

thranasingha commented Aug 28, 2024

Pablo-platypus commented Jan 19, 2023 •

edited

Loading

singhbaljit commented Jun 30, 2023 •

edited

Loading

sabper commented Aug 6, 2024 •

edited

Loading

DGondhiya commented Aug 8, 2024 •

edited

Loading

yanivkrol commented Aug 12, 2024 •

edited

Loading

kvanbrabant commented Aug 22, 2024 •

edited

Loading