Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature Request]: Access PipelineOptions in BeamSqlSeekableTable #28145

Closed
1 of 15 tasks
gabrywu opened this issue Aug 24, 2023 · 3 comments · Fixed by #28253
Closed
1 of 15 tasks

[Feature Request]: Access PipelineOptions in BeamSqlSeekableTable #28145

gabrywu opened this issue Aug 24, 2023 · 3 comments · Fixed by #28253
Labels
awaiting triage java new feature P3

Comments

@gabrywu
Copy link
Member

gabrywu commented Aug 24, 2023 •
edited
Loading

What would you like to happen?

I'm creating a BeamSqlSeekableTable class, and want to access PipelineOptions.
For example, a HadoopDataSeekableTable needs loading data from Hadoop, and our Hadoop cluster needs keberos auth.
So, my question is that is it possible to access pipeline options to complete the auth?

I know that I can pass auth parameters to table properties, however, it will be convenient to get parameters from PipelineOptions.

Issue Priority

Priority: 3 (nice-to-have improvement)

Issue Components

  • Component: Python SDK
  • Component: Java SDK
  • Component: Go SDK
  • Component: Typescript SDK
  • Component: IO connector
  • Component: Beam examples
  • Component: Beam playground
  • Component: Beam katas
  • Component: Website
  • Component: Spark Runner
  • Component: Flink Runner
  • Component: Samza Runner
  • Component: Twister2 Runner
  • Component: Hazelcast Jet Runner
  • Component: Google Cloud Dataflow Runner
@gabrywu
Copy link
Member Author

gabrywu commented Aug 30, 2023

any one need this feature? I can create a PR to this feature (already released our production environment)

@liferoad
Copy link
Collaborator

any one need this feature? I can create a PR to this feature (already released our production environment)

Please do a PR. Thanks for contributing to Beam!

gabrywu added a commit to gabrywu/beam that referenced this issue Aug 31, 2023
@gabrywu
closing apache#28145
8c4a4ff
@gabrywu gabrywu mentioned this issue Aug 31, 2023
Merged
@gabrywu
Copy link
Member Author

gabrywu commented Aug 31, 2023 •
edited
Loading

any one need this feature? I can create a PR to this feature (already released our production environment)

Please do a PR. Thanks for contributing to Beam!

PR here, #28253, @liferoad

@gabrywu gabrywu changed the title [Feature Request]: access PipelineOptions in BeamSqlSeekableTable [Feature Request]: Access PipelineOptions in BeamSqlSeekableTable Sep 1, 2023
Abacn pushed a commit that referenced this issue Sep 5, 2023
@gabrywu
Pass runtime configs & variables to BeamSqlSeekableTable (#28253)
3364271 
* closing #28145

* remove unsupported parameter & add parameter type to context

* remove unnecessary semicolon
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
awaiting triage java new feature P3
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Pass runtime configs & variables to BeamSqlSeekableTable gabrywu/beam
2 participants
@liferoad @gabrywu