Feature request: define/override AWS region with sasl.jaas.config (Error: Host … does not belong to a valid region)

[yermulnik]

Our use case is that we want to use custom MSK broker endpoint hostnames by aliasing them to original endpoints provided by MSK (sensitive info in the below code snippet is replaces with Xs):

> host dev-use1-msk-cluster-b-01.our.custom.domain.name
dev-use1-msk-cluster-b-01.our.custom.domain.name is an alias for b-1.devuse1mskXXXX.XXXX.c16.kafka.us-east-1.amazonaws.com.
b-1.devuse1mskXXXX.XXXX.c16.kafka.us-east-1.amazonaws.com has address 10.XX.XX.XX

This helps us template MSK broker endpoint hostnames for different environments to eliminate the need for an app to query client connection info from AWS API.

This works w/o issues within AWS, though this fails when using AWS MSK IAM Auth library to connect from outside AWS as 1) EC2 metadata endpoint doesn't exist there and 2) library cannot derive region from the endpoint DNS hostname.

Ref: https://github.com/aws/aws-msk-iam-auth/blob/main/src/main/java/software/amazon/msk/auth/iam/internals/AuthenticationRequestParams.java#L62-L71

I wasn't able to figure out how to override (explicitly set) AWS region with sasl.jaas.config and I lean to think there's no option for that at the moment.

Is it possible to implement a feature to provide AWS region name (awsRegion) and to disable querying AWS EC2 Metadata Endpoint (awsDisableMetadata) just like other AWS parameters (awsProfileName, awsRoleArn, so on) at https://github.com/aws/aws-msk-iam-auth/blob/main/src/main/java/software/amazon/msk/auth/iam/internals/MSKCredentialProvider.java#L83-L96 ?

Example properties file content:

security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required awsProfileName=dev-msk-cluster;
sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler
# Disable SSL server host name verification
ssl.endpoint.identification.algorithm=

Example error output using aws-msk-iam-auth-2.1.0-all.jar and kafka_2.13-3.6.1 client (slightly trimmed for the sake of brevity):

./${KAFKA_VERSION}/bin/kafka-topics.sh --list --command-config msk-iam-auth-client.properties --bootstrap-server $KAFKA_BROKERS
[2024-05-31 16:21:16,612] WARN Unable to retrieve the requested metadata (/latest/dynamic/instance-identity/document). Failed to connect to service endpoint:  (aws_msk_iam_auth_shadow.com.amazonaws.util.EC2MetadataUtils)
aws_msk_iam_auth_shadow.com.amazonaws.SdkClientException: Failed to connect to service endpoint:
        at aws_msk_iam_auth_shadow.com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:119)
[…]
Caused by: java.net.SocketTimeoutException: connect timed out
        at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
[…]
[2024-05-31 16:21:16,631] ERROR [AdminClient clientId=adminclient-1] Connection to node -3 (dev-use1-msk-cluster-b-01.our.custom.domain.name/10.XX.XX.XX:9098) failed authentication due to: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: Exception while evaluating challenge [Caused by java.lang.IllegalArgumentException: Host dev-use1-msk-cluster-b-01.our.custom.domain.name does not belong to a valid region.]) occurred when evaluating SASL token received from the Kafka Broker. Kafka Client will go to AUTHENTICATION_FAILED state. (org.apache.kafka.clients.NetworkClient)
[2024-05-31 16:21:16,632] WARN [AdminClient clientId=adminclient-1] Metadata update failed due to authentication error (org.apache.kafka.clients.admin.internals.AdminMetadataManager)
org.apache.kafka.common.errors.SaslAuthenticationException: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: Exception while evaluating challenge [Caused by java.lang.IllegalArgumentException: Host dev-use1-msk-cluster-b-01.our.custom.domain.name does not belong to a valid region.]) occurred when evaluating SASL token received from the Kafka Broker. Kafka Client will go to AUTHENTICATION_FAILED state.
Caused by: javax.security.sasl.SaslException: Exception while evaluating challenge [Caused by java.lang.IllegalArgumentException: Host dev-use1-msk-cluster-b-01.our.custom.domain.name does not belong to a valid region.]
        at software.amazon.msk.auth.iam.internals.IAMSaslClient.evaluateChallenge(IAMSaslClient.java:114)
[…]
aused by: java.lang.IllegalArgumentException: Host dev-use1-msk-cluster-b-01.our.custom.domain.name does not belong to a valid region.
        at software.amazon.msk.auth.iam.internals.AuthenticationRequestParams.create(AuthenticationRequestParams.java:66)
[…]
Error while executing topic command : An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: Exception while evaluating challenge [Caused by java.lang.IllegalArgumentException: Host dev-use1-msk-cluster-b-01.our.custom.domain.name does not belong to a valid region.]) occurred when evaluating SASL token received from the Kafka Broker. Kafka Client will go to AUTHENTICATION_FAILED state.
[2024-05-31 16:21:16,637] ERROR org.apache.kafka.common.errors.SaslAuthenticationException: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: Exception while evaluating challenge [Caused by java.lang.IllegalArgumentException: Host dev-use1-msk-cluster-b-01.our.custom.domain.name does not belong to a valid region.]) occurred when evaluating SASL token received from the Kafka Broker. Kafka Client will go to AUTHENTICATION_FAILED state.
Caused by: javax.security.sasl.SaslException: Exception while evaluating challenge [Caused by java.lang.IllegalArgumentException: Host dev-use1-msk-cluster-b-01.our.custom.domain.name does not belong to a valid region.]
        at software.amazon.msk.auth.iam.internals.IAMSaslClient.evaluateChallenge(IAMSaslClient.java:114)
[…]
Caused by: java.lang.IllegalArgumentException: Host dev-use1-msk-cluster-b-01.our.custom.domain.name does not belong to a valid region.
        at software.amazon.msk.auth.iam.internals.AuthenticationRequestParams.create(AuthenticationRequestParams.java:66)
        at software.amazon.msk.auth.iam.internals.IAMSaslClient.generateClientMessage(IAMSaslClient.java:144)
        at software.amazon.msk.auth.iam.internals.IAMSaslClient.evaluateChallenge(IAMSaslClient.java:96)
        ... 14 more
 (kafka.admin.TopicCommand$)

Thank you.

[y-usuf]

I am also having the same issue as @yermulnik. Is there any update as to whether this request will be implemented? Or a possible work around it in the meantime?

[henry-ahn0]

I am experiencing the same issue. I am trying to access the AWS MSK broker through a proxy rather than using the broker host directly, and this results in the same problem..

UPDATE)

I upgraded to version 2.2.0 and the error disappeared in version 2.2.0 🤔