Invoke Pants via module in ITs. #8265
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This enables uniform execution of Pants in a subprocess whether running in pantsbuild/pants from our custom venv, from a pants PEX or from a pantsbuild/setup venv. In order to support this mode, apply the `PYTHONPATH` scrubbing hack in `PytestRun` to `PythonRun` and `PythonRepl` as well pending a fix for: pex-tool/pex#707
|
NB: There are 5 more uses of PEX.run I found, but none of these appear to disrupt ITs so I'd prefer to leave as-is and get them fixed by upgrading to pex 1.6.11 pex-tool/pex#766:
|
|
Thanks John! This didn't have any reviewers on it, and I half-acted on it anyway by adding @Eric-Arellano ... is it reviewable? |
|
Thanks @stuhood, it is indeed reviewable. |
Eric-Arellano
approved these changes
Sep 11, 2019
src/python/pants/backend/python/tasks/python_execution_task_base.py
Outdated
Show resolved
Hide resolved
stuhood
reviewed
Sep 13, 2019
| @@ -383,6 +356,7 @@ def run_pants_with_workdir_without_waiting(self, command, workdir, config=None, | |||
| env = os.environ.copy() | |||
| if extra_env: | |||
| env.update(extra_env) | |||
| env.update(PYTHONPATH=os.pathsep.join(sys.path)) | |||
There was a problem hiding this comment.
Hey gang: I believe that this code might have caused an issue with "loose-source" plugins not ending up "all the way" on the python path when running integration tests.
The loose-source plugins are loaded with [GLOBAL] pythonpath, but the subprocessed pants runs fail with:
Exception caught: (pants.base.exceptions.BackendConfigurationError)
File "/opt/twitter_mde/package/eepython36/68cc93925092a6fc31d254392558fa8c840f7bd5407db9435ca48ff1b8f6c07f/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/opt/twitter_mde/package/eepython36/68cc93925092a6fc31d254392558fa8c840f7bd5407db9435ca48ff1b8f6c07f/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/Users/stuhood/src/source/.pants.d/pyprep/sources/cf3f7a2a3d17140e6302e01caf2b23a165031273/pants/__main__.py", line 8, in <module>
pants_loader.main()
File "/Users/stuhood/src/source/.pants.d/pyprep/sources/cf3f7a2a3d17140e6302e01caf2b23a165031273/pants/bin/pants_loader.py", line 78, in main
PantsLoader.run()
File "/Users/stuhood/src/source/.pants.d/pyprep/sources/cf3f7a2a3d17140e6302e01caf2b23a165031273/pants/bin/pants_loader.py", line 74, in run
cls.load_and_execute(entrypoint)
File "/Users/stuhood/src/source/.pants.d/pyprep/sources/cf3f7a2a3d17140e6302e01caf2b23a165031273/pants/bin/pants_loader.py", line 67, in load_and_execute
entrypoint_main()
File "/Users/stuhood/src/source/.pants.d/pyprep/sources/cf3f7a2a3d17140e6302e01caf2b23a165031273/pants/bin/pants_exe.py", line 32, in main
PantsRunner(start_time=start_time).run()
File "/Users/stuhood/src/source/.pants.d/pyprep/sources/cf3f7a2a3d17140e6302e01caf2b23a165031273/pants/bin/pants_runner.py", line 85, in run
options_bootstrapper=options_bootstrapper
File "/Users/stuhood/src/source/.pants.d/pyprep/sources/cf3f7a2a3d17140e6302e01caf2b23a165031273/pants/bin/local_pants_runner.py", line 142, in create
options_bootstrapper=options_bootstrapper,
File "/Users/stuhood/src/source/.pants.d/pyprep/sources/cf3f7a2a3d17140e6302e01caf2b23a165031273/pants/bin/local_pants_runner.py", line 91, in parse_options
build_config = BuildConfigInitializer.get(options_bootstrapper)
File "/Users/stuhood/src/source/.pants.d/pyprep/sources/cf3f7a2a3d17140e6302e01caf2b23a165031273/pants/init/options_initializer.py", line 38, in get
cls._cached_build_config = cls(options_bootstrapper).setup()
File "/Users/stuhood/src/source/.pants.d/pyprep/sources/cf3f7a2a3d17140e6302e01caf2b23a165031273/pants/init/options_initializer.py", line 69, in setup
self._bootstrap_options.backend_packages
File "/Users/stuhood/src/source/.pants.d/pyprep/sources/cf3f7a2a3d17140e6302e01caf2b23a165031273/pants/init/options_initializer.py", line 58, in _load_plugins
return load_backends_and_plugins(plugins, working_set, backend_packages)
File "/Users/stuhood/src/source/.pants.d/pyprep/sources/cf3f7a2a3d17140e6302e01caf2b23a165031273/pants/init/extension_loader.py", line 32, in load_backends_and_plugins
load_build_configuration_from_source(build_configuration, backends)
File "/Users/stuhood/src/source/.pants.d/pyprep/sources/cf3f7a2a3d17140e6302e01caf2b23a165031273/pants/init/extension_loader.py", line 113, in load_build_configuration_from_source
load_backend(build_configuration, backend_package)
File "/Users/stuhood/src/source/.pants.d/pyprep/sources/cf3f7a2a3d17140e6302e01caf2b23a165031273/pants/init/extension_loader.py", line 129, in load_backend
raise BackendConfigurationError(f'Failed to load the {backend_module} backend: {e!r}')
Exception message: Failed to load the twitter.pants_internal.register backend: ModuleNotFoundError("No module named 'twitter.pants_internal.tasks.some_twitter_task'",)
Which is odd, because it sounds like maybe the register.py is located, but not some of the other code...
There was a problem hiding this comment.
Nothing comes to mind. Perhaps you can create a repro against pants own loose source plugins in a local IT? Easier, just hand-edit /Users/stuhood/src/source/.pants.d/pyprep/sources/cf3f7a2a3d17140e6302e01caf2b23a165031273/pants/init/extension_loader.py to print out sys.path and dump sys.modules to start.
There was a problem hiding this comment.
Holy mackerel. This was undeclared BUILD level dependencies in these tests. That is very unexpected, and very welcome... I have to go change our README to indicate that integration tests do need valid dependency lists. Exciting =)
stuhood
added a commit
that referenced
this pull request
Jul 7, 2020
### Problem Since #8265, we have been running `PantsRunIntegrationTest`s from loose sources in the repository, which are included via `src/python/pants/testutil:int-test`'s dependency on `src/python/pants/bin:pants_local_binary`. That change thus removed the need for the `pants.pex` from our wrapper scripts. ### Solution Remove the "`pants.pex` for integration tests" mechanism. Post #8625, pants is run from the PYTHONPATH of the test target, which will automatically include either loose sources or a `pants_requirement` target, depending on whether the test is run in or out of the pantsbuild/pants repo. Additionally, remove one set of tests that was attempting to test that nested runs of pants do not deadlock. #7654 will eventually allow that issue to be resolved by allowing the concurrent run rather than via any sort of automatic disabling of `pantsd`, and that is much easier to test via concurrent runs _without_ nesting. [ci skip-rust-tests]
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This enables uniform execution of Pants in a subprocess whether running
in pantsbuild/pants from our custom venv, from a pants PEX or from a
pantsbuild/setup venv.
In order to support this mode, apply the
PYTHONPATHscrubbing hack inPytestRuntoPythonRunandPythonReplas well pending a fix for:pex-tool/pex#707