[BEAM-13624] Run generatePythonRequirementsAll for pyarrow 6

[TheNeuralBit]

ValidatesRunner compliance status (on master branch)

Lang	ULR	Dataflow	Flink	Samza	Spark	Twister2
Go	---					---
Java
Python	---					---
XLang						---

Examples testing status on various runners

Lang	ULR	Dataflow	Flink	Samza	Spark	Twister2
Go	---	---	---	---	---	---	---
Java	---		---	---	---	---	---
Python	---	---	---	---	---	---	---
XLang	---	---	---	---	---	---	---

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Go	Java	Python

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website	Whitespace	Typescript
Non-portable
Portable	---			---	---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

[TheNeuralBit]

This went very smoothly once I realized I just needed to install python-3.x-venv in the start-build-env.sh container. It looks like this is already handled in #15819

[codecov]

Codecov Report

Merging #16000 (7be74ef) into master (6dc1b6f) will decrease coverage by 0.00%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #16000      +/-   ##
==========================================
- Coverage   83.62%   83.62%   -0.01%     
==========================================
  Files         445      445              
  Lines       61371    61371              
==========================================
- Hits        51324    51322       -2     
- Misses      10047    10049       +2     

Impacted Files	Coverage Δ
...hon/apache_beam/runners/direct/test_stream_impl.py	94.02% <0.00%> (-2.24%)	⬇️
sdks/python/apache_beam/internal/metrics/metric.py	90.00% <0.00%> (-1.00%)	⬇️
sdks/python/apache_beam/transforms/util.py	95.86% <0.00%> (-0.17%)	⬇️
...hon/apache_beam/runners/worker/bundle_processor.py	93.51% <0.00%> (-0.13%)	⬇️
...eam/runners/interactive/interactive_environment.py	91.72% <0.00%> (+0.34%)	⬆️
...runners/interactive/display/pcoll_visualization.py	86.36% <0.00%> (+0.50%)	⬆️
...pache_beam/runners/interactive/interactive_beam.py	76.58% <0.00%> (+0.97%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5656e26...7be74ef. Read the comment docs.

[tvalentyn]

this looks like a bug..
see:
https://stackoverflow.com/questions/39577984/what-is-pkg-resources-0-0-0-in-output-of-pip-freeze-command

sounds like we might need to grep it out from the output for the script to work from the local env? or perhaps this will also go away with virtualenv->venv switch, although I don't fully understand when virtualenv was involved here. FWIW I don't get this line when I run this script.

[TheNeuralBit]

Good catch. I can add a patch to grep out pkg_resources and remove it from this PR.

As far as why this happened.. I don't think virtualenv was ever involved. It's notable that this only happened for py38, and not the other python versions. I'm not sure what that means though.

[tvalentyn]

are you running it on mac?

[TheNeuralBit]

No I'm running on Debian (gLinux). But ./gradlew was actually executed inside the ./start-build-env.sh (ubuntu) container.

Added 99f489f to filter out pkg_resources

[tvalentyn]

PR number LGTM.