Update dataflow python to use external containers #26383
Conversation
…damccorm/externalContainers
Codecov Report
@@ Coverage Diff @@
## master #26383 +/- ##
==========================================
- Coverage 81.12% 81.08% -0.04%
==========================================
Files 469 469
Lines 67185 67199 +14
==========================================
- Hits 54501 54486 -15
- Misses 12684 12713 +29
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 14 files with indirect coverage changes 📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
|
Run Python Dataflow ValidatesRunner |
|
Run Python Dataflow V2 ValidatesRunner |
|
Run Python Dataflow ValidatesRunner |
|
Run Python Dataflow V2 ValidatesRunner |
|
Run Python Dataflow ValidatesContainer |
|
R: @tvalentyn |
|
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control |
| repository=container_repo, version_suffix=version_suffix) | ||
|
|
||
| if is_runner_v2: | ||
| version_suffix = '%s.%s' % (sys.version_info[0:2]) |
There was a problem hiding this comment.
We actually need this, it inserts a . while the last one does not.
With that said, I don't think this is helpful (the difference is too subtle), so I removed version_suffix entirely and replaced it with major/minor
|
Run Python Dataflow ValidatesRunner |
|
Run Python Dataflow V2 ValidatesRunner |
|
Run Python Dataflow ValidatesContainer |
1 similar comment
|
Run Python Dataflow ValidatesContainer |
This updates python to use containers created as part of the Beam snapshot process instead of containers built by Dataflow.
These containers are initially published to https://pantheon.corp.google.com/gcr/images/apache-beam-testing/global/beam_portability/ and then mirrored to https://pantheon.corp.google.com/gcr/images/cloud-dataflow/global/v1beta3/ (for example, for 3.8 containers are mirrored to - https://pantheon.corp.google.com/gcr/images/cloud-dataflow/global/v1beta3/beam_python3.8_sdk)
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI.