Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(out_kafka2): adds support for AWS IAM authentication to MSK usin… #481

Merged
merged 1 commit into from
Jan 23, 2023
Merged

feat(out_kafka2): adds support for AWS IAM authentication to MSK usin… #481

merged 1 commit into from
Jan 23, 2023

Conversation

raytung
Copy link
Member

@raytung raytung commented Jan 21, 2023 •
edited
Loading

…g long lived credentials

Addresses #472

This PR adds support for AWS IAM authentication using long lived credentials (access key id and secret access keys). To support AWS assume role and STS, we will need to wait for upstream's ruby-kafka library support.

We will need to bump ruby-kafka to 1.5.0 in order to support this feature.

@raytung
feat(out_kafka2): adds support for AWS IAM authentication to MSK usin…
c7fce94 
…g long lived credentials

This commit adds support for AWS IAM authentication using long lived
credentials (access key id and secret access keys). To support AWS
assume role and STS, we will need to wait for upstream's `ruby-kafka`
library support.

We will need to bump `ruby-kafka` to 1.5.0 in order to support this
feature.

Signed-off-by: Ray Tung <[email protected]>
@raytung raytung marked this pull request as ready for review January 21, 2023 04:21
@ashie ashie self-requested a review January 23, 2023 04:15
ashie
ashie approved these changes Jan 23, 2023
View reviewed changes
Copy link
Member

@ashie ashie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
Thanks!

@ashie
Copy link
Member

ashie commented Jan 23, 2023 •
edited
Loading

It's not this PR's problem but I think it would be better to unify indent & setting mandatory parameters.

@ashie
Copy link
Member

ashie commented Jan 23, 2023

we will need to wait for upstream's ruby-kafka library support.

BTW according to https://github.com/zendesk/ruby-kafka ruby-kafka is no longer maintained.
We might need to consider deprecating also this plugin...

@ashie ashie merged commit 2253efd into fluent:master Jan 23, 2023
@raytung
Copy link
Member Author

raytung commented Jan 23, 2023 •
edited
Loading

@ashie I saw that too 😢 I spent a bit of time digging up the story for librdkafka but it looks like there's a heavy (perhaps overlooked) push back from the author / Confluent to support pluggable authentication call back in librdkafka which means there's no way to support AWS IAM auth 😢 (Basically they are not implementing this KIP https://cwiki.apache.org/confluence/display/KAFKA/KIP-86%3A+Configurable+SASL+callback+handlers)

@raytung raytung deleted the feat/aws-iam-auth branch January 23, 2023 04:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ashie ashie ashie approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@raytung @ashie