Lacking parity between requirements.txt and install_requires syntax

[vitaly-krugl]

Environment

• pip version: 18.1
• Python version: Python 2.7.15
• OS: OS X 10.13.6 (High Sierra)

• setuptools version: 40.6.3

Description
Per PEP-508, I can now add 'selogging_python@https://host.company.com/content/groups/public/com/akamai/se/selogging-python/1.0.4/selogging-python-1.0.4.tgz' to the install_requires arg of setuptools.setup() and selogging_python dependency gets installed when the app containing that install_requires is installed. However, if I add selogging_python@https://host.company.com/content/groups/public/com/akamai/se/selogging-python/1.0.4/selogging-python-1.0.4.tgz
to my requirements.txt, then pip install -r requirements.txt fails with:

Processing ./selogging_python@https:/host.company.com/content/groups/public/com/akamai/se/selogging-python/1.0.4/selogging-python-1.0.4.tgz
Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: '/Users/vkruglik/git/vitaly_s_place/https-orchestration/selogging_python@https:/host.company.com/content/groups/public/com/akamai/se/selogging-python/1.0.4/selogging-python-1.0.4.tgz'

Expected behavior
I expected the same syntax to work in requirements.txt as in install_requires. Doesn't it make sense to support the same syntax in both?

How to Reproduce

1. add selogging_python@https://host.company.com/content/groups/public/com/akamai/se/selogging-python/1.0.4/selogging-python-1.0.4.tgz
   to my requirements.txt
   run pip install -r requirements.txt
   Output

Processing ./selogging_python@https:/host.company.com/content/groups/public/com/akamai/se/selogging-python/1.0.4/selogging-python-1.0.4.tgz
Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: '/Users/vkruglik/git/vitaly_s_place/https-orchestration/selogging_python@https:/host.company.com/content/groups/public/com/akamai/se/selogging-python/1.0.4/selogging-python-1.0.4.tgz'

Additional Info: If I add "#sha1=98286070ea0d7b1c6e9a0899717d8330c6d538a9" to the end of the URL in requirements.txt, then pip install -r requirements.txt works okay. However, since install_requires works without this "#sha1=98286070ea0d7b1c6e9a0899717d8330c6d538a9", I would expect pip install -r ... to work without it as well. It appears that pip install -r ... gets confused without the "#sha1=98286070ea0d7b1c6e9a0899717d8330c6d538a9" suffix into thinking that the URL is a file path inside the current working directory.

[RonnyPfannschmidt]

whats your setuptools version?

[vitaly-krugl]

@RonnyPfannschmidt , I updated the Environment info to include my setuptools version. I also added Additional Info at the end of the How to Reproduce section that shines additional light on the problem.

[uranusjr]

(Previous comment deleted; I read the spec wrong.)

I wonder if this is actually a problem in the spec. requirements.txt allows specifying a path as a requirement, and as such, there is no way to uniquely dinstinguish a URL based lookup to a local path, since @ is a valid character in path (at least on some systems). This is probably also why requirements.txt parsing fails for you; the <name>@<url> case is eaten by the path branch since it does look like one (programmatically).

[cjerdonek]

See also #6202 where PR #6203 has been submitted to resolve this.

[ghost]

For what it's worth, I mentioned this in the other ticket, but install_requires kinda wants PEP 508 URLs when you add github package sources but don't want to use dependency_links (which are really undesirable because they don't install automatically anymore). However, this means if you also want a requirements.txt you cannot just read that in the setup.py and slap that into install_requires automatically, so you end up with:

Either 1. all dependencies maintained twice in two differing formats that are bound to get out of sync

or 2. really ugly manually hacked transformation code: https://github.com/wobblui/wobblui/blob/2795a8f6cd2ffa7d97e84033f69afecbdf82607e/setup.py#L104

So if someone with an interest in solving this ticket wants to save the world from really ugly setup.py files, solving this bug would make quite a difference in these corner cases

Edit: I got the transform code though so I'm definitely good personally, so this was just some extra info on the impact. It's not like this is a showstopper, it'd just be nice to eventually be able to throw the ugly transform code out

[javabrett]

What is the status here - is it an accepted bug and looking-for-help, or work-underway/under-review, or issue-likely-rejected/won't-fix? This is biting me so just wondering whether it is time to pitch-in.

[pradyunsg]

There's #6203 for this.

[chrahunt]

Closing, since this was fixed by #6203 as demonstrated by:

t.sh

#!/bin/sh
cd "$(mktemp -d)"
python -m venv venv
echo "requests @ https://files.pythonhosted.org/packages/01/62/ddcf76d1d19885e8579acb1b1df26a852b03472c0e46d2b959a714c90608/requests-2.22.0.tar.gz" > requirements.txt
venv/bin/pip install pip==19.1

venv/bin/pip install -r requirements.txt

venv/bin/pip install --upgrade pip

venv/bin/pip install -r requirements.txt

Output

Collecting pip==19.1
  Using cached https://files.pythonhosted.org/packages/f9/fb/863012b13912709c13cf5cfdbfb304fa6c727659d6290438e1a88df9d848/pip-19.1-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 19.2.3
    Uninstalling pip-19.2.3:
      Successfully uninstalled pip-19.2.3
Successfully installed pip-19.1
WARNING: Requirement 'requests @ https://files.pythonhosted.org/packages/01/62/ddcf76d1d19885e8579acb1b1df26a852b03472c0e46d2b959a714c90608/requests-2.22.0.tar.gz' looks like a filename, but the file does not exist
Processing ./requests @ https:/files.pythonhosted.org/packages/01/62/ddcf76d1d19885e8579acb1b1df26a852b03472c0e46d2b959a714c90608/requests-2.22.0.tar.gz
ERROR: Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: '/tmp/user/1000/tmp.uD8Och6CiY/requests @ https:/files.pythonhosted.org/packages/01/62/ddcf76d1d19885e8579acb1b1df26a852b03472c0e46d2b959a714c90608/requests-2.22.0.tar.gz'

WARNING: You are using pip version 19.1, however version 19.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting pip
  Using cached https://files.pythonhosted.org/packages/00/b6/9cfa56b4081ad13874b0c6f96af8ce16cfbc1cb06bedf8e9164ce5551ec1/pip-19.3.1-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 19.1
    Uninstalling pip-19.1:
      Successfully uninstalled pip-19.1
Successfully installed pip-19.3.1
Collecting requests@ https://files.pythonhosted.org/packages/01/62/ddcf76d1d19885e8579acb1b1df26a852b03472c0e46d2b959a714c90608/requests-2.22.0.tar.gz
  Using cached https://files.pythonhosted.org/packages/01/62/ddcf76d1d19885e8579acb1b1df26a852b03472c0e46d2b959a714c90608/requests-2.22.0.tar.gz
Collecting chardet<3.1.0,>=3.0.2
  Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting idna<2.9,>=2.5
  Using cached https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
  Using cached https://files.pythonhosted.org/packages/e0/da/55f51ea951e1b7c63a579c09dd7db825bb730ec1fe9c0180fc77bfb31448/urllib3-1.25.6-py2.py3-none-any.whl
Collecting certifi>=2017.4.17
  Using cached https://files.pythonhosted.org/packages/18/b0/8146a4f8dd402f60744fa380bc73ca47303cccf8b9190fd16a827281eac2/certifi-2019.9.11-py2.py3-none-any.whl
Installing collected packages: chardet, idna, urllib3, certifi, requests
    Running setup.py install for requests ... done
Successfully installed certifi-2019.9.11 chardet-3.0.4 idna-2.8 requests-2.22.0 urllib3-1.25.6