Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow local packages in requirements.txt dependency list. #23684

Merged
merged 5 commits into from
Oct 21, 2022

Conversation

robertwb
Copy link
Contributor

We pull these out and install them with the extra local packages infrastructure.


Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Choose reviewer(s) and mention them in a comment (R: @username).
  • Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests
Go tests

See CI.md for more information about GitHub Actions CI.

We pull these out and install them with the extra local packages infrastructure.
@robertwb
Copy link
Contributor Author

R: @chamikaramj

@robertwb robertwb requested a review from chamikaramj October 18, 2022 00:26
@github-actions
Copy link
Contributor

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

@codecov
Copy link

codecov bot commented Oct 18, 2022

Codecov Report

Merging #23684 (77dd157) into master (a138a4f) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master   #23684   +/-   ##
=======================================
  Coverage   73.17%   73.17%           
=======================================
  Files         720      720           
  Lines       96044    96076   +32     
=======================================
+ Hits        70277    70305   +28     
- Misses      24455    24459    +4     
  Partials     1312     1312           
Flag Coverage Δ
python 82.94% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...s/python/apache_beam/runners/portability/stager.py 86.27% <100.00%> (+0.72%) ⬆️
sdks/python/apache_beam/internal/gcp/auth.py 79.16% <0.00%> (-3.70%) ⬇️
.../python/apache_beam/transforms/periodicsequence.py 98.38% <0.00%> (-1.62%) ⬇️
sdks/python/apache_beam/runners/direct/executor.py 96.46% <0.00%> (-0.55%) ⬇️
...hon/apache_beam/runners/worker/bundle_processor.py 93.30% <0.00%> (-0.38%) ⬇️
sdks/python/apache_beam/io/gcp/gcsio.py 92.12% <0.00%> (-0.14%) ⬇️
sdks/python/apache_beam/runners/common.py 88.59% <0.00%> (-0.13%) ⬇️
sdks/python/apache_beam/io/gcp/bigquery.py 71.34% <0.00%> (+0.03%) ⬆️
...on/apache_beam/runners/dataflow/dataflow_runner.py 80.80% <0.00%> (+0.13%) ⬆️
...ks/python/apache_beam/runners/worker/sdk_worker.py 89.24% <0.00%> (+0.16%) ⬆️
... and 4 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@AnandInguva
Copy link
Contributor

How is this different from using --extra_package?

@robertwb
Copy link
Contributor Author

Because the user can place these directly inside their requirements file rather than listing them individually. This is particularly valuable for cross-language, or where one wants to keep the construction-time and execution-time environments in sync via a single file.

@AnandInguva
Copy link
Contributor

Because the user can place these directly inside their requirements file rather than listing them individually. This is particularly valuable for cross-language, or where one wants to keep the construction-time and execution-time environments in sync via a single file.

Do we want to update the documentation to reflect this change? https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/

@robertwb
Copy link
Contributor Author

Failure due to #21104

@robertwb
Copy link
Contributor Author

Run Python PreCommit

@robertwb
Copy link
Contributor Author

Run Python_PVR_Flink PreCommit

@robertwb
Copy link
Contributor Author

Good call. Updated the docs.

@@ -50,6 +50,15 @@
* ([#X](https://github.com/apache/beam/issues/X)).
-->

# [2.44.0] - Unreleased

## New Features / Improvements
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we also add a unit test for this feature ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

@AnandInguva
Copy link
Contributor

cc : @tvalentyn

@chamikaramj
Copy link
Contributor

LGTM. Thanks.

@robertwb robertwb merged commit 8f300b0 into apache:master Oct 21, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants