unhandled AttributeError during pex bootstrapping with PEX_PATH #598
Comments
|
This is not a setuptools symbol, its a python 3 symbol via https://www.python.org/dev/peps/pep-0420/ These things become the So... I think you must be running:
|
|
Alright - this seems to be right and works with both classic @@ -53,15 +53,19 @@ class PEXEnvironment(Environment):
return explode_dir
@classmethod
- def update_module_paths(cls, new_code_path):
+ def update_module_paths(cls, pex_file, explode_dir):
# Force subsequent imports to come from the .pex directory rather than the .pex file.
- TRACER.log('Adding to the head of sys.path: %s' % new_code_path)
- sys.path.insert(0, new_code_path)
- for name, module in sys.modules.items():
- if hasattr(module, "__path__"):
- module_dir = os.path.join(new_code_path, *name.split("."))
- TRACER.log('Adding to the head of %s.__path__: %s' % (module.__name__, module_dir))
- module.__path__.insert(0, module_dir)
+ TRACER.log('Adding exploded pex dir to the head of sys.path: %s' % explode_dir)
+ sys.path.insert(0, explode_dir)
+ pex_path = os.path.realpath(pex_file)
+ for name, module in list(sys.modules.items()):
+ if hasattr(module, '__path__'):
+ for item in module.__path__:
+ if os.path.realpath(item).startswith(pex_path):
+ TRACER.log('Un-importing module %s initially imported from within zipped pex %s'
+ % (name, pex_path), V=2)
+ sys.modules.pop(name)
+ break
@classmethod
def write_zipped_internal_cache(cls, pex, pex_info):
@@ -203,7 +207,8 @@ class PEXEnvironment(Environment):
self.update_candidate_distributions(self.load_internal_cache(self._pex, self._pex_info))
if not self._pex_info.zip_safe and os.path.isfile(self._pex):
- self.update_module_paths(self.force_local(self._pex, self._pex_info))
+ explode_dir = self.force_local(pex_file=self._pex, pex_info=self._pex_info)
+ self.update_module_paths(pex_file=self._pex, explode_dir=explode_dir)
all_reqs = [Requirement.parse(req) for req in self._pex_info.requirements]
I'm working on a unit test... |
|
As a small piece of the test puzzle, I have confirmed that a wheel built from:
Generates a |
|
sorry for the late response - been swamped this week and just noticed your reply.
indeed we're running under python3 and for the latter definitely inadvertently - I saw this choke on both here's a snippit w/ debug logging (sorry, I meant to include this when circling back w/ a repro): the only reference to |
|
here's a minimal repro (using an older pex36 release, but should still be helpful): the odd bit to me here is the exact positional requirement of the repro - passing only a singular, seemingly problematic pex on nor does reordering: nor does unspecifying (or doing that to any 1-2 of the 3..all 3 must be marked zip_safe=False fwict to repro) |
|
Thanks Kris! Yeah, so all that is needed:
Item 1 puts implicit namespace packages in play We have: As such, the example can be shrunk to: |
|
Super compact version of all this that puts the key factors outlined above in play: |
|
Noting some problems that make it hard to test the failing code, whose intent is to ensure any namespace package with components internal to a zipped pex are subsequently imported from the exploded zip contents:
I do have a fix for 3 though and this allows a test for the case the broken code was attempting to handle. That case is legit! Without a fix to the broken code in this issue's backtrace (and a fix to 3) otherwise legitimate pex loose source code with namespaces that is not zip safe is in fact broken by pex. Fixes coming... |
|
A further issue here is varying behavior in setuptools (pkg_resources) handling PEP-420 namespace packages. At the low end of our current compatibility range for setuptools, pkg_resources blows up for similar reasons, expecting |
Previously, pex would blow up trying to adjust import paths if a PEP-420 implicit namespace package was encountered. Add a test of the path adjustment functionality both to confirm it was needed (it was) and that the fix technique works with all forms of namespace packages by only assuming they support `append`. Fixes pex-tool#598
### Problem See pex-tool/pex#623 for the pex release ticket. This release incorporates two changes, one of which (pex-tool/pex#599) was necessary to fix the issue in pex-tool/pex#598. ### Solution Bump the pex version to `1.5.3`.
See pex-tool/pex#623 for the pex release ticket. This release incorporates two changes, one of which (pex-tool/pex#599) was necessary to fix the issue in pex-tool/pex#598. Bump the pex version to `1.5.3`.
Previously, pex would blow up trying to adjust import paths if a PEP-420 implicit namespace package was encountered. Add a test of the path adjustment functionality both to confirm it was needed (it was) and that the fix technique works with all forms of namespace packages by only assuming they support `append`. Fixes pex-tool#598
Previously, pex would blow up trying to adjust import paths if a PEP-420 implicit namespace package was encountered. Add a test of the path adjustment functionality both to confirm it was needed (it was) and that the fix technique works with all forms of namespace packages by only assuming they support `append`. Fixes pex-tool#598
Previously, pex would blow up trying to adjust import paths if a PEP-420 implicit namespace package was encountered. Add a test of the path adjustment functionality both to confirm it was needed (it was) and that the fix technique works with all forms of namespace packages by only assuming they support `append`. To support all this, work around a pypy zipimporter bug: https://bitbucket.org/pypy/pypy/issues/1686 and, as a side-effect, eliminate our CI workaround for the pypy unit test shard. Fixes pex-tool#598 Fixes pex-tool#497
Previously, pex would blow up trying to adjust import paths if a PEP-420 implicit namespace package was encountered. Add a test of the path adjustment functionality both to confirm it was needed (it was) and that the fix technique works with all forms of namespace packages by only assuming they support `append`. To support all this, work around a pypy zipimporter bug: https://bitbucket.org/pypy/pypy/issues/1686 and, as a side-effect, eliminate our CI workaround for the pypy unit test shard. Fixes #598 Fixes #497
I don't have a minimal repro yet, but in some internal code that uses PEX_PATH with two other pex files that contain namespace packages we seem to be seeing the following unhandled exception on startup:
I think this points to a setuptools version problem, but I've ensured uniform alignment across
setuptools==33.1.1in all 3 involved pex files + build-time interpreter with no luck.The text was updated successfully, but these errors were encountered: