Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] custreamz fails to connect to kafka with oauth login #9410

Closed
pthalasta opened this issue Oct 9, 2021 · 2 comments · Fixed by #9486
Closed

[BUG] custreamz fails to connect to kafka with oauth login #9410

pthalasta opened this issue Oct 9, 2021 · 2 comments · Fixed by #9486
Assignees
@jdye64
Labels
bug Something isn't working libcudf Affects libcudf (C++/CUDA) code. Python Affects Python cuDF API.

Comments

@pthalasta
Copy link

Describe the bug
custreamz does not work with kafka oauth configs due to Kafka data source object has a parameter kafka_config of type map[string string]. However, confluent kafka can take 'oauth_cb': functools.partial(function) to get token from the server to connect to Kafka broker.

https://github.com/rapidsai/cudf/blob/branch-21.10/python/custreamz/custreamz/kafka.py#L30
https://github.com/rapidsai/cudf/blob/branch-21.10/python/cudf_kafka/cudf_kafka/_lib/kafka.pxd#L21

librdkafka's example link: https://github.com/confluentinc/confluent-kafka-python/blob/master/examples/oauth_producer.py

Steps/Code to reproduce bug
Can be replicated by trying to connect to Kafka broker with oauth

Expected behavior
https://github.com/rapidsai/cudf/blob/branch-21.10/python/custreamz/custreamz/kafka.py#L30 should be changed to accept a config of type map[string,object]

Environment overview (please complete the following information)

  • Environment location: all
  • Method of cuDF install: conda but shouldn't matter
@pthalasta pthalasta added Needs Triage Need team to review and classify bug Something isn't working labels Oct 9, 2021
@jdye64 jdye64 mentioned this issue Oct 21, 2021
Merged
@beckernick beckernick added Python Affects Python cuDF API. libcudf Affects libcudf (C++/CUDA) code. and removed Needs Triage Need team to review and classify labels Oct 25, 2021
@github-actions
Copy link

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.

@jdye64
Copy link
Contributor

jdye64 commented Dec 22, 2021

PR #9486 should be merged in the coming days to resolve this issue.

@rapids-bot rapids-bot bot closed this as completed in #9486 Jan 6, 2022
rapids-bot bot pushed a commit that referenced this issue Jan 6, 2022
@jdye64
custreamz oauth callback for kafka (librdkafka) (#9486)
23603d1 
Previously it was impossible to use custreamz with oauth enabled Kafka brokers. This PR adds a feature so that the user can supply a Python function which is invoked to get the oauth token, from a http endpoint for example, and then supply that token to librdkafka to be used in both the initial connection to kafka and also subsequently as the token becomes stale.

This closes #9410

Authors:
  - Jeremy Dyer (https://github.com/jdye64)
  - AJ Schmidt (https://github.com/ajschmidt8)

Approvers:
  - Robert Maynard (https://github.com/robertmaynard)
  - Vyas Ramasubramani (https://github.com/vyasr)
  - Ram (Ramakrishna Prabhu) (https://github.com/rgsl888prabhu)
  - AJ Schmidt (https://github.com/ajschmidt8)

URL: #9486
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jdye64 jdye64

Labels
bug Something isn't working libcudf Affects libcudf (C++/CUDA) code. Python Affects Python cuDF API.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

custreamz oauth callback for kafka (librdkafka) jdye64/cudf
3 participants
@jdye64 @beckernick @pthalasta