Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Correctly parse labels if they are passed as a single string instead of a list #34183

Merged
merged 4 commits into from
Mar 5, 2025

Conversation

damccorm
Copy link
Contributor

@damccorm damccorm commented Mar 5, 2025

Right now, if you submit a job like:

options = PipelineOptions()
options.view_as(StandardOptions).runner = "DataflowRunner"
<other options>
options.view_as(GoogleCloudOptions).labels = {'env': 'bluecore-qa-extensibility', 'team': 'cdp-vertical'}

with beam.Pipeline(options=options) as p:
  <do stuff>

The job does not correctly recognize the labels. We could validate and fail because labels is expected to be a list, but that might break existing (mostly working) pipelines. This is an in between where we can still do a best effort to correctly identify/use the user's labels.


Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests
Go tests

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

@@ -447,6 +447,8 @@ def __init__(self, options, proto_pipeline):
if self.google_cloud_options.labels:
self.proto.labels = dataflow.Job.LabelsValue()
labels = self.google_cloud_options.labels
if isinstance(labels, str):
labels = [labels]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you mean options.view_as(GoogleCloudOptions).labels = {'env': 'bluecore-qa-extensibility', 'team': 'cdp-vertical'} is interpreted as a string although it is a Python dict?

Copy link
Contributor Author

@damccorm damccorm Mar 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whoops, meant to make this a draft until I full tested it. But you're right, this won't fully work yet. I think I'll also want something like:

if not isinstance(labels, Iterable):
   labels = [str(labels)]

But I'll fix/test this with the string and dict cases

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should now handle both cases, and I added some representative test cases, PTAL

@damccorm damccorm marked this pull request as draft March 5, 2025 18:04
@damccorm damccorm marked this pull request as ready for review March 5, 2025 18:12
@damccorm
Copy link
Contributor Author

damccorm commented Mar 5, 2025

R: @liferoad

Copy link
Contributor

github-actions bot commented Mar 5, 2025

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers

@damccorm damccorm merged commit 9da7006 into master Mar 5, 2025
87 checks passed
@damccorm damccorm deleted the users/damccorm/parseLabels branch March 5, 2025 19:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants