Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BEAM-10702, BEAM-10757] Cherrypick #12571: Do not implicitly decompress artifacts #12619

Merged
merged 2 commits into from
Aug 19, 2020

Conversation

jkff
Copy link
Contributor

@jkff jkff commented Aug 18, 2020

Also adds a Julia set test on portable local runner, which uses a
setup.py and hence exercises the artifact staging codepath.

This is a squashed cherrypick of #12571 + a change to CHANGES.md.

I also added the LICENSE for enum34 to manual_licenses as the original repo is down. I took the license from somebody's backup: https://github.com/jamespharaoh/python-enum34

Post-Commit Tests Status (on master branch)

Lang SDK Dataflow Flink Samza Spark Twister2
Go Build Status --- Build Status --- Build Status ---
Java Build Status Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status Build Status
Build Status
Build Status
Build Status
Python Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
--- Build Status ---
XLang Build Status --- Build Status --- Build Status ---

Pre-Commit Tests Status (on master branch)

--- Java Python Go Website
Non-portable Build Status Build Status
Build Status
Build Status
Build Status
Build Status Build Status
Portable --- Build Status --- ---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels

See CI.md for more information about GitHub Actions CI.

Also adds a Julia set test on portable local runner, which uses a
setup.py and hence exercises the artifact staging codepath.

This is a squashed cherrypick of #12571
@jkff
Copy link
Contributor Author

jkff commented Aug 18, 2020

Hmm all the builds are pending apparently cause something's going on with Jenkins - eg https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Commit/ says "Jenkins is going to shut down". Haven't seen this before. Do I need to do anything?

@tvalentyn
Copy link
Contributor

not sure, perhaps related to https://issues.apache.org/jira/browse/BEAM-10751.

@tvalentyn
Copy link
Contributor

retest this please

Copy link
Contributor

@youngoli youngoli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This all looks good, thanks Eugene! I'll merge it in after the tests pass, assuming the Python Precommit issue is fixed.

@tvalentyn
Copy link
Contributor

tvalentyn commented Aug 18, 2020

@youngoli note that precommit IT tests will fail until Dataflow containers for the release branch are built. This change does not affect Dataflow codepath.

@jkff
Copy link
Contributor Author

jkff commented Aug 18, 2020

retest this please

@jkff
Copy link
Contributor Author

jkff commented Aug 18, 2020

Retesting cause https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Commit/12849/consoleFull shows an error in :sdks:python:container:py2:docker but it passes when I run this target locally.

@tvalentyn
Copy link
Contributor

Run PythonDocker PreCommit

@tvalentyn
Copy link
Contributor

ERROR:root:Error occurred when pull license for enum34 from {'license': 'https://bitbucket.org/stoneleaf/enum34/raw/c208549a93b71b948ff7bbdfd29dce8f85527916/enum/LICENSE'}.
16:06:02 Traceback (most recent call last):
16:06:02   File "/tmp/license_scripts/pull_licenses_py.py", line 94, in pull_from_url
16:06:02     url_read = urlopen(config['license'])
16:06:02   File "/usr/local/lib/python2.7/urllib2.py", line 154, in urlopen
16:06:02 �[0m�[91m    return opener.open(url, data, timeout)
16:06:02   File "/usr/local/lib/python2.7/urllib2.py", line 435, in open
16:06:02     response = meth(req, response)
16:06:02   File "/usr/local/lib/python2.7/urllib2.py", line 548, in http_response
16:06:02     'http', request, response, code, msg, hdrs)
16:06:02   File "/usr/local/lib/python2.7/urllib2.py", line 473, in error
16:06:02     return self._call_chain(*args)
16:06:02 �[0m�[91m  File "/usr/local/lib/python2.7/urllib2.py", line 407, in _call_chain
16:06:02     result = func(*args)
16:06:02   File "/usr/local/lib/python2.7/urllib2.py", line 556, in http_error_default

We would need to change how we fetch the license for enum34. This must be blocking precommits now on master as well.

@tvalentyn
Copy link
Contributor

@ethanfurman FYI, the project homepage link (https://bitbucket.org/stoneleaf/enum34) at https://pypi.org/project/enum34/ is now broken.

@jkff
Copy link
Contributor Author

jkff commented Aug 18, 2020

Yeah it'll have to be added to manual_licenses. I'll do that in this PR in a moment.

@tvalentyn
Copy link
Contributor

@tvalentyn
Copy link
Contributor

thanks, @jkff, will assign that to you then.

@jkff jkff changed the title [BEAM-10702] Cherrypick #12571: Do not implicitly decompress artifacts [BEAM-10702, BEAM-10757] Cherrypick #12571: Do not implicitly decompress artifacts Aug 18, 2020
license: "https://bitbucket.org/stoneleaf/enum34/raw/c208549a93b71b948ff7bbdfd29dce8f85527916/enum/LICENSE"
# The original repo is down. This license taken from somebody's clone:
# https://github.com/jamespharaoh/python-enum34/blob/master/enum/LICENSE
license: "file:///tmp/license_scripts/manual_licenses/enum34/LICENSE"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

did you git add the LICENSE file? Also, it would be great if you send a PR for master with these commit(s). Thanks!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops, added now, thanks! Will send a separate PR to master. Do you prefer if I remove it from here?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Feel free to keep it here as well, we can merge these into 2.24.0 as two separate commits. Thanks!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jkff jkff force-pushed the jkff-cherry-12571 branch from d85ba7d to 9ea783d Compare August 18, 2020 23:55
@jkff
Copy link
Contributor Author

jkff commented Aug 19, 2020

The PythonDocs precommit is failing because it was only added at master in #12575 and isn't present in the 2.24 branch.

@youngoli
Copy link
Contributor

Ack. The Python Precommit is also only failing because of BEAM-10697, so that's not an issue.

I'm wondering about the PVR_Flink PreCommit though, I haven't seen that error below. Flake maybe? (Link)

@youngoli
Copy link
Contributor

Run Python2_PVR_Flink PreCommit

@jkff
Copy link
Contributor Author

jkff commented Aug 19, 2020

Alright, we now have 2 failing precommits both due to known issues: Python with BEAM-10697, and PythonDocs because it doesn't exist at 2.24. Looks like BEAM-10697 is resolved, but not cherrypicked into 2.24. @youngoli Do you want to ignore it for this PR or should we wait for someone to cherrypick a fix and rebase on that?

@youngoli
Copy link
Contributor

Yeah, I think it's safe to ignore the Python precommit error here. I'll merge it in.

@youngoli youngoli merged commit 33d2437 into release-2.24.0 Aug 19, 2020
@kennknowles kennknowles deleted the jkff-cherry-12571 branch February 4, 2022 16:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants