Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

--no-pre-install-wheels --requirements-pex .. --layout packed gives [Errno 21] Is a directory: '.../.deps/....whl' errors #2391

Open
huonw opened this issue Mar 24, 2024 · 3 comments
Open

--no-pre-install-wheels --requirements-pex .. --layout packed gives [Errno 21] Is a directory: '.../.deps/....whl' errors #2391

huonw opened this issue Mar 24, 2024 · 3 comments
Labels
bug

Comments

@huonw
Copy link
Collaborator

huonw commented Mar 24, 2024

It seems pex emits "Is a directory" errors referring to wheels for invocations that use all three of --requirements-pex ... --no-pre-install-wheels --layout packed.

For instance:

curl -OLs https://github.com/pex-tool/pex/releases/download/v2.2.2/pex
chmod +x ./pex

./pex cowsay==6.1 -o reqs.pex
./pex --layout packed --no-pre-install-wheels --requirements-pex reqs.pex -o output.pex

Output:

[Errno 21] Is a directory: '/private/var/folders/sv/vd266m4d4lvctgs2wpnhjs9w0000gn/T/tmp4032w10a/.deps/cowsay-6.1-py3-none-any.whl'
Output with PEX_VERBOSE=1 on second invocation 
pex: Laying out PEX zipfile /Users/huon/tmp/pex-loose-no-pre-install/pex: 0.1ms
pex: Executing installed PEX for /Users/huon/tmp/pex-loose-no-pre-install/./pex at /Users/huon/.pex/unzipped_pexes/ad5d02c39f095829e9578f565f825d0835bfcd73
pex:   Testing /Users/huon/.pyenv/versions/3.10.4/bin/python3.10 can resolve PEX at /Users/huon/.pex/unzipped_pexes/ad5d02c39f095829e9578f565f825d0835bfcd73: 2.8ms
pex: Using the current interpreter /Users/huon/.pyenv/versions/3.10.4/bin/python since it matches constraints and PYTHONPATH is not set.
pex: Discarding site packages path: /Users/huon/.pyenv/versions/3.10.4/lib/python3.10/site-packages
pex: Tainted path element: /Users/huon/.pyenv/versions/3.10.4/lib/python3.10/site-packages
pex: Scrubbing from user site: /Users/huon/.local/lib/python3.10/site-packages
pex: Scrubbing from site-packages: /Users/huon/.pyenv/versions/3.10.4/lib/python3.10/site-packages
pex: New sys.path: ['/Users/huon/.pex/unzipped_pexes/ad5d02c39f095829e9578f565f825d0835bfcd73/.bootstrap/pex/vendor/_vendored/attrs', '/Users/huon/.pex/unzipped_pexes/ad5d02c39f095829e9578f565f825d0835bfcd73/.bootstrap', '/Users/huon/.pex/unzipped_pexes/ad5d02c39f095829e9578f565f825d0835bfcd73', '/Users/huon/.pyenv/versions/3.10.4/lib/python310.zip', '/Users/huon/.pyenv/versions/3.10.4/lib/python3.10', '/Users/huon/.pyenv/versions/3.10.4/lib/python3.10/lib-dynload']
pex: Activating PEX virtual environment from /Users/huon/.pex/unzipped_pexes/ad5d02c39f095829e9578f565f825d0835bfcd73: 0.2ms
pex: Bootstrap complete, performing final sys.path modifications...
pex: PYTHONPATH contains:
pex:     /Users/huon/.pex/unzipped_pexes/ad5d02c39f095829e9578f565f825d0835bfcd73
pex:   * /Users/huon/.pyenv/versions/3.10.4/lib/python310.zip
pex:     /Users/huon/.pyenv/versions/3.10.4/lib/python3.10
pex:     /Users/huon/.pyenv/versions/3.10.4/lib/python3.10/lib-dynload
pex:     /Users/huon/.pex/installed_wheels/5f30587359f5364bfbab6d6f902d7418fc7ef9e81e0d6329781a775d3a189b97/pex-2.2.2-py2.py3-none-any.whl
pex:     /Users/huon/.pex/unzipped_pexes/ad5d02c39f095829e9578f565f825d0835bfcd73/.bootstrap
pex:   * - paths that do not exist or will be imported via zipimport
pex: Building pex: 3.4ms                         
pex:   Adding distributions from pexes: reqs.pex: 2.5ms
pex:     Laying out PEX zipfile reqs.pex: 0.1ms
pex:   Resolving distributions for requirements: : 0.1ms
pex:     Resolving requirements.: 0.1ms
pex:   Configuring PEX dependencies: 0.2ms
Previous binary unexpectedly exists, cleaning: output.pex
pex: Zipping PEX .bootstrap/ code.: 0.0ms
Traceback (most recent call last):
  File "/Users/huon/.pex/installed_wheels/5f30587359f5364bfbab6d6f902d7418fc7ef9e81e0d6329781a775d3a189b97/pex-2.2.2-py2.py3-none-any.whl/pex/common.py", line 111, in safe_copy
    os.link(source, dest)
PermissionError: [Errno 1] Operation not permitted: '/private/var/folders/sv/vd266m4d4lvctgs2wpnhjs9w0000gn/T/tmp970ao6ze/.deps/cowsay-6.1-py3-none-any.whl' -> 'output.pex~/.deps/cowsay-6.1-py3-none-any.whl'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/huon/.pex/installed_wheels/5f30587359f5364bfbab6d6f902d7418fc7ef9e81e0d6329781a775d3a189b97/pex-2.2.2-py2.py3-none-any.whl/pex/result.py", line 105, in catch
    return func(*args, **kwargs)
  File "/Users/huon/.pex/installed_wheels/5f30587359f5364bfbab6d6f902d7418fc7ef9e81e0d6329781a775d3a189b97/pex-2.2.2-py2.py3-none-any.whl/pex/bin/pex.py", line 1072, in do_main
    pex_builder.build(
  File "/Users/huon/.pex/installed_wheels/5f30587359f5364bfbab6d6f902d7418fc7ef9e81e0d6329781a775d3a189b97/pex-2.2.2-py2.py3-none-any.whl/pex/pex_builder.py", line 739, in build
    self._build_packedapp(
  File "/Users/huon/.pex/installed_wheels/5f30587359f5364bfbab6d6f902d7418fc7ef9e81e0d6329781a775d3a189b97/pex-2.2.2-py2.py3-none-any.whl/pex/pex_builder.py", line 844, in _build_packedapp
    safe_copy(os.path.join(self._chroot.chroot, path), dest)
  File "/Users/huon/.pex/installed_wheels/5f30587359f5364bfbab6d6f902d7418fc7ef9e81e0d6329781a775d3a189b97/pex-2.2.2-py2.py3-none-any.whl/pex/common.py", line 129, in safe_copy
    do_copy()
  File "/Users/huon/.pex/installed_wheels/5f30587359f5364bfbab6d6f902d7418fc7ef9e81e0d6329781a775d3a189b97/pex-2.2.2-py2.py3-none-any.whl/pex/common.py", line 104, in do_copy
    shutil.copy(source, temp_dest)
  File "/Users/huon/.pyenv/versions/3.10.4/lib/python3.10/shutil.py", line 417, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/Users/huon/.pyenv/versions/3.10.4/lib/python3.10/shutil.py", line 254, in copyfile
    with open(src, 'rb') as fsrc:
IsADirectoryError: [Errno 21] Is a directory: '/private/var/folders/sv/vd266m4d4lvctgs2wpnhjs9w0000gn/T/tmp970ao6ze/.deps/cowsay-6.1-py3-none-any.whl'
[Errno 21] Is a directory: '/private/var/folders/sv/vd266m4d4lvctgs2wpnhjs9w0000gn/T/tmp970ao6ze/.deps/cowsay-6.1-py3-none-any.whl'
Details of the full 3×3 grid of requirements/outputs layouts

As part of my investigation, I wondered whether the layout of the --requirements-pex mattered. It doesn't seem to, but here's what I did. Script that does all of them:

cd $(mktemp -d)

curl -OL https://github.com/pex-tool/pex/releases/download/v2.2.2/pex
chmod +x ./pex

for reqs in loose packed zipapp; do
    rm -rf reqs.pex
    ./pex --layout $reqs cowsay==6.1 -o reqs-$reqs.pex
    for output in loose packed zipapp; do
        echo
        echo "*** requirements=$reqs, output=$output"
        rm -rf target.pex
        ./pex --layout $output --no-pre-install-wheels --requirements-pex reqs-$reqs.pex -o output-$reqs-$output.pex
    done
done
requirements \ output loose packed zipapp
loose ✅ ❌ ✅
packed ✅ ❌ ✅
zipapp ✅ ❌ ✅
@huonw huonw added the bug label Mar 24, 2024
@huonw huonw mentioned this issue Mar 24, 2024
Merged
@jsirois jsirois self-assigned this Mar 24, 2024
@jsirois
Copy link
Member

jsirois commented Mar 24, 2024

You won't like the answer here - although a bug, the fix will be merely to make the error message a controlled one. This is the ~same case as here, which I did catch when adding this feature:

pex/pex/environment.py

Lines 284 to 289 in 27c2db2

if result_type_wheel_file:
if not self._pex_info.deps_are_wheel_files:
raise ResolveError(
"Cannot resolve .whl files from PEX at {pex}; its dependencies are in the "
"form of pre-installed wheel chroots.".format(pex=self.source_pex)
)

Fixing this (and the code above) will only be possible with the completion of #2299. I have #2299 all but done, but I've been side-tracked working on the lock update support Pants needs; so won't be circling back to that until #2373 ships.

jsirois added a commit to jsirois/pex that referenced this issue Mar 24, 2024
@jsirois
Guard against mismatched --requirements-pex.
a6bdb31 
Pex cannot yet handle turning installed wheel chroots back into wheel
files (see pex-tool#2299). As such, until that time, it must reject attempts
to combine PEXes with different wheel packaging.

Clarifies pex-tool#2391
@jsirois jsirois mentioned this issue Mar 24, 2024
Merged
@huonw
Copy link
Collaborator Author

huonw commented Mar 25, 2024

You won't like the answer here

Perfectly happy with "unsupported, but we can make the error message better".

jsirois added a commit that referenced this issue Mar 28, 2024
@jsirois
Guard against mismatched --requirements-pex. (#2392)
70b6d47 
Pex cannot yet handle turning installed wheel chroots back into wheel
files (see #2299). As such, until that time, it must reject attempts
to combine PEXes with different wheel packaging.

Clarifies #2391
@jsirois
Copy link
Member

jsirois commented Mar 28, 2024

The OP is not solved since it relies on functionality forthcoming in #2299; so I'll leave this issue open but stop progress / un-assign for now.

@jsirois jsirois removed their assignment Mar 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jsirois @huonw