Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

A directory under both purelib and data is incorrectly removed by --target #7548

Closed
godlygeek opened this issue Jan 3, 2020 · 3 comments
Closed
Labels
C: target pip install's --target option's behaviour handling resolution: duplicate Duplicate of an existing issue/PR state: awaiting PR Feature discussed, PR is needed type: bug A confirmed bug or unintended behavior

Comments

@godlygeek
Copy link
Contributor

Environment

  • pip version: 19.3.1
  • Python version: 3.8.0
  • OS: Linux

Description
If a directory is duplicated between purelib and data, --target blows it away in the middle of an install, so it winds up containing only the data files.

Expected behavior
The set of directories to be written should be computed only once and any removals should be done before creating any directories.

How to Reproduce
Given this reproducer as reproducer.sh:

mkdir issue7548_reproducer
cd issue7548_reproducer

echo data >data.txt
mkdir issue7548
touch issue7548/__init__.py

cat >setup.py <<EOF
from setuptools import setup

setup(
    name='issue7548',
    version='0.0.1',
    packages=['issue7548'],
    data_files=[('issue7548', ['data.txt'])],
)
EOF

pip install --upgrade pip

pip install --upgrade --prefix /tmp/issue7548_prefix .
find /tmp/issue7548_prefix

pip install --upgrade --target /tmp/issue7548_target .
find /tmp/issue7548_target

Run it as sh reproducer.sh inside an activated virtualenv.

Output

Expect to see among the output:

Successfully installed issue7548-0.0.1
/tmp/issue7548_prefix
/tmp/issue7548_prefix/issue7548
/tmp/issue7548_prefix/issue7548/data.txt
/tmp/issue7548_prefix/lib
/tmp/issue7548_prefix/lib/python3.8
/tmp/issue7548_prefix/lib/python3.8/site-packages
/tmp/issue7548_prefix/lib/python3.8/site-packages/issue7548
/tmp/issue7548_prefix/lib/python3.8/site-packages/issue7548/__init__.py
/tmp/issue7548_prefix/lib/python3.8/site-packages/issue7548/__pycache__
/tmp/issue7548_prefix/lib/python3.8/site-packages/issue7548/__pycache__/__init__.cpython-38.pyc
/tmp/issue7548_prefix/lib/python3.8/site-packages/issue7548-0.0.1-py3.8.egg-info
/tmp/issue7548_prefix/lib/python3.8/site-packages/issue7548-0.0.1-py3.8.egg-info/PKG-INFO
/tmp/issue7548_prefix/lib/python3.8/site-packages/issue7548-0.0.1-py3.8.egg-info/SOURCES.txt
/tmp/issue7548_prefix/lib/python3.8/site-packages/issue7548-0.0.1-py3.8.egg-info/dependency_links.txt
/tmp/issue7548_prefix/lib/python3.8/site-packages/issue7548-0.0.1-py3.8.egg-info/installed-files.txt
/tmp/issue7548_prefix/lib/python3.8/site-packages/issue7548-0.0.1-py3.8.egg-info/top_level.txt

Successfully installed issue7548-0.0.1
/tmp/issue7548_target
/tmp/issue7548_target/issue7548
/tmp/issue7548_target/issue7548/data.txt
/tmp/issue7548_target/issue7548-0.0.1-py3.8.egg-info
/tmp/issue7548_target/issue7548-0.0.1-py3.8.egg-info/PKG-INFO
/tmp/issue7548_target/issue7548-0.0.1-py3.8.egg-info/SOURCES.txt
/tmp/issue7548_target/issue7548-0.0.1-py3.8.egg-info/dependency_links.txt
/tmp/issue7548_target/issue7548-0.0.1-py3.8.egg-info/installed-files.txt
/tmp/issue7548_target/issue7548-0.0.1-py3.8.egg-info/top_level.txt

That is, when --prefix was used it created a separate issue7548 directory in lib/python3.8/site-packages and in the root directory, as expected, and the purelib one contains an __init__.py and the data one contains data.txt, as expected.

When --target was used only data.txt exists, not __init__.py, because in between installing the purelib files and installing the data files the _handle_target_dir function removed the issue7548 directory.

@triage-new-issues triage-new-issues bot added the S: needs triage Issues/PRs that need to be triaged label Jan 3, 2020
@chrahunt chrahunt added C: target pip install's --target option's behaviour handling type: bug A confirmed bug or unintended behavior labels Jan 3, 2020
@triage-new-issues triage-new-issues bot removed the S: needs triage Issues/PRs that need to be triaged label Jan 3, 2020
@chrahunt chrahunt added the state: awaiting PR Feature discussed, PR is needed label Jan 3, 2020
@chrahunt
Copy link
Member

chrahunt commented Jan 3, 2020

Thank you for such a nice report! I was able to confirm this with your script and I think you're right about the cause.

In general I think the desired approach for data files is actually to put them within packages themselves. See importlib.resources for the standard library module that implements accessing such files. Doing this should avoid the issue.

@gutsytechster
Copy link
Contributor

What can be done to resolve this issue?

@uranusjr
Copy link
Member

uranusjr commented Feb 6, 2021

Closing in favour of #8799 since more discussions happend there.

@uranusjr uranusjr closed this as completed Feb 6, 2021
@uranusjr uranusjr added the resolution: duplicate Duplicate of an existing issue/PR label Feb 6, 2021
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Oct 3, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
C: target pip install's --target option's behaviour handling resolution: duplicate Duplicate of an existing issue/PR state: awaiting PR Feature discussed, PR is needed type: bug A confirmed bug or unintended behavior
Projects
None yet
Development

No branches or pull requests

4 participants