Databricks - StreamingQueryException

[nph4rd]

Hi - I'm trying to read from Kafka from Databricks, as outlined here.

At my organisation we can only connect via IAM, so I added the following options:

  .option("kafka.security.protocol", "SASL_SSL")
  .option("kafka.sasl.mechanism", "AWS_MSK_IAM")
  .option("kafka.sasl.jaas.config", "software.amazon.msk.auth.iam.IAMLoginModule required;")
  .option("kafka.sasl.client.callback.handler.class", "software.amazon.msk.auth.iam.IAMClientCallbackHandler")

and uploaded the JAR file for the 1.1.6 version of aws-msk-iam-auth to Databricks.

This resulted in the following error message:

StreamingQueryException: Query[...] terminated with exception: org/apache/kafka/common/security/auth/AuthenticateCallbackHandler 

I found a potential patch here.

However, it would be far from ideal to work with a patched forked of this library. Is there anything I can do on my side to make the connection work?