Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for MSK with IAM authentication #521

Open
wants to merge 12 commits into
base: master
Choose a base branch
from

Conversation

madebydna
Copy link

This adds support for using MSK IAM authentication with the rdkafka2 output type. Authentication and authorization with an MSK cluster are facilitated through a base64-encoded signed URL, which is generated by the aws-msk-iam-sasl-signer-ruby library.

@madebydna madebydna closed this Dec 2, 2024
Support for AWS MSK with IAM authentication
@madebydna
Copy link
Author

Reopening it based on conversation here.

@madebydna madebydna reopened this Dec 4, 2024
@madebydna
Copy link
Author

I fixed the dependencies that require Ruby > 3.0, so that the CI tests using Ruby 2.7 now also pass. I also added rdkafka to the gemspec file instead of the Gemfile, but leaving the USE_RDKAFKA conditional intact. Was there a particular reason that dependency was only in the Gemfile?

If there is interest at all in merging it in, I'd be happy to continue making improvements to the code with some input. Please let me know if this feature is not something that is considered useful and I can close the PR.

In terms of possible improvements, I was wondering if it would be better to have a config param such as use_aws_msk_iam_auth and not require the two rdkafka_options of security.protocol and sasl.mechanisms to be set explicitly. So instead of

<match *>
    @type rdkafka2
    # other options
    
    # AWS Region (required)
    aws_msk_region us-east-1
    
    # MSK IAM authentication settings (required)
    rdkafka_options {
      "security.protocol": "sasl_ssl",
      "sasl.mechanisms": "OAUTHBEARER"
    }
</match>

the config could look like this:

<match *>
    @type rdkafka2
    # ... other options ....

    use_aws_msk_iam_auth true
    aws_msk_region us-east-1
</match>

Also, I was wondering if there is a way to not use the @shared_producer but use the per-thread producers instead, since that is the default. Currently, there is a deadlock when calling get_producer from the oauthbearer_token_refresh_callback and the the share_producer setting is not set to true.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant