-
Notifications
You must be signed in to change notification settings - Fork 68
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Unable to use regional sts endpoint when running on KDA app #118
Comments
I am running into very same issue though the documentation states that IAM java code should use the regional STS endpoint with MirrorMaker2 and MSK Connect to replicate Kafka data from one cluster to another. Thanks |
The first things to try is to follow https://docs.aws.amazon.com/sdkref/latest/guide/feature-sts-regionalized-endpoints.html
If you can't (it was my case with KDA), I created a PR with a fix: #119 we need to wait for it to be merged. |
Thank you @menuetb for the response. I am using MSK Connect and I don't know if there is an option to set sts_regional_endpoints = regional in the properties though I have set the region and rolearn as below.
|
- Revert "Remove formatting" This reverts commit 45be4a5. - Revert "use API to get ServiceEndpoint" This reverts commit e68c800. - Revert "add tests & remove unnesseray import" This reverts commit 8f735c0. - Revert "Update MSKCredentialProvider.java" This reverts commit 34d68b2. - Revert "Add support for STS Regional Endpoint (aws#118)" This reverts commit ed0027f.
We are reverting the commits in #136 as we are seeing the following issue where passing only
|
We also have this PR, are they related? #37 |
No, this PR #37 is useful if you can set the environment variable AWS_STS_REGIONAL_ENDPOINTS. In services such as Managed Service for Apache Flink we cant set this environment variable |
FWIW this also affects Scala Spark jobs on Glue. Also, although only slightly related, the recently released MSK IAM library for Python has exactly the same problem. So pyflink and pyspark on Glue do not work with IAM and VPC endpoints for the same reason. @hhkkxxx133 I think I can see why the error you mentioned was happening. aws-msk-iam-auth/src/main/java/software/amazon/msk/auth/iam/internals/MSKCredentialProvider.java Line 280 in 34d68b2
If the region parameter is not set, it defaults to "aws-global". But:
is not valid, hence the error message: "Credential should be scoped to a valid region" I think the logic is in the wrong place, the decision should happen when the STS client is constructed, something like:
Alternatively, a separate parameter in the jaas config could be used, eg awsStsEndpoint. in some ways I think I prefer this second approach, because it makes very clear the intent of the user is to override the endpoint configuration. I can submit a PR, if either of these approaches are OK? Andy |
Thank you for the comment. This indeed fixed the issue we were having. I have merged the change already. Closing the issue. |
|
I'm writing a KDA application that needs to read data from an MSK cluster in another account.
The VPC does not have internet access.
I set up a regional endpoint for STS in my VPC.
The MSK cluster uses IAM authentication and the set up of the cross account trust policy is done.
Here is the conf for my aws-msk-iam-auth client:
I see in the logs that the client is not trying to use the regional endpoint that I configured but that he is still trying to use the global endpoint.
So I can't use STS with an MSK app in a VPC without an internet connection.
The text was updated successfully, but these errors were encountered: