Installion Command with --pre and Pinning is Inconsistent

[jambonrose]

This is not an exact duplicate of #2019, but the two are certainly related.

Help Output:

$ python -m pipenv.help output

Pipenv version: '11.10.0'

Pipenv location: '/Users/andrew/.virtualenvs/static-site-test/lib/python3.6/site-packages/pipenv'

Python location: '/Users/andrew/.virtualenvs/static-site-test/bin/python'

Other Python installations in PATH:

• 2.6: /usr/bin/python2.6

• 2.6: /usr/bin/python2.6

• 2.7: /opt/local/bin/python2.7

• 2.7: /opt/local/bin/python2.7

• 2.7: /usr/bin/python2.7

• 3.4: /opt/local/bin/python3.4m

• 3.4: /opt/local/bin/python3.4

• 3.5: /opt/local/bin/python3.5m

• 3.5: /opt/local/bin/python3.5

• 3.6: /opt/local/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6m

• 3.6: /Users/andrew/.virtualenvs/static-site-test/bin/python3.6

• 3.6: /opt/local/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6

• 3.6: /opt/local/bin/python3.6

• 3.6.4: /Users/andrew/.virtualenvs/static-site-test/bin/python

• 3.6.4: /opt/local/bin/python

• 2.7.10: /usr/bin/python

• 3.6.4: /Users/andrew/.virtualenvs/static-site-test/bin/python3

• 3.6.4: /opt/local/Library/Frameworks/Python.framework/Versions/3.6/bin/python3

• 3.6.4: /opt/local/bin/python3

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.6.4',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '15.6.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 15.6.0: Tue Jan 30 11:45:51 PST '
                     '2018; root:xnu-3248.73.8~1/RELEASE_X86_64',
 'python_full_version': '3.6.4',
 'python_version': '3.6',
 'sys_platform': 'darwin'}

System environment variables:

• GIT_PS1_SHOWUNTRACKEDFILES
• VIRTUALENVWRAPPER_WORKON_CD
• VIRTUALENVWRAPPER_HOOK_DIR
• LANG
• XPC_FLAGS
• OLDPWD
• TERM_SESSION_ID
• WORKON_HOME
• GEM_HOME
• __CF_USER_TEXT_ENCODING
• CFLAGS
• SSH_AUTH_SOCK
• VIRTUAL_ENV
• PREFIX
• USER
• ENV
• LSCOLORS
• PWD
• HOME
• GIT_PS1_SHOWDIRTYSTATE
• TERM_PROGRAM
• CLICOLOR
• TERM_PROGRAM_VERSION
• GIT_PS1_SHOWCOLORHINTS
• Apple_PubSub_Socket_Render
• TMPDIR
• PROJECT_HOME
• XPC_SERVICE_NAME
• VIRTUALENVWRAPPER_SCRIPT
• TERM
• SHELL
• SHLVL
• PROMPT_COMMAND
• MANPATH
• LOGNAME
• LDFLAGS
• VIRTUALENVWRAPPER_PROJECT_FILENAME
• PATH
• PS1
• _
• PYTHONDONTWRITEBYTECODE
• PIP_PYTHON_PATH
• PYTHONUNBUFFERED

Pipenv–specific environment variables:

Debug–specific environment variables:

• PATH: /Users/andrew/.virtualenvs/static-site-test/bin:/opt/local/Library/Frameworks/Python.framework/Versions/3.6/bin:/Applications/Postgres.app/Contents/Versions/9.4/bin:/opt/local/bin:/opt/local/sbin:/Users/andrew/.config/yarn/global/bin:/Users/andrew/.rubygems/2.3/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Frameworks/Mono.framework/Versions/Current/Commands
• SHELL: /opt/local/bin/bash
• LANG: en_US.UTF-8
• PWD: /Users/andrew/Development/static-site-test
• VIRTUAL_ENV: /Users/andrew/.virtualenvs/static-site-test

Expected result

I expect to be able to install a pre-release like I would a full-release package if there are no full-releases for a package, or else to receive a note about the --pre flag.

Actual result

When I attempt to install a dependency currently in pre-release (black in the following instances) with pipenv and a >= pin, I'm told it's unable to resolve the dependency, and that the error exists in the subgraph. However, if I use a == pin, then everything works. This behavior is inconsistent and confusing, and the error message is misleading. I note that I can use the >= pin if I specify the --pre tag.

$ # the following work
$ pipenv install black --pre
$ pipenv install 'black>=18.4a0' --pre # installs 18.4a2
$ pipenv install 'black==18.4a0'

$ # the following do not; error output below
$ pipenv install black
$ pipenv install 'black>=18.4a0'

The error message (shown in the output below) did nothing to make it clear that the --pre flag is missing from the last two commands. What's more, if the --pre flag is not required with ==, then I would not expect it to be required with the >= pin.

Optimally, pipenv should have installed the package with a warning, or else asked me whether I wanted to install a pre-release, rather than error cryptically as it did. To clarify, if my options are all pre-releases, I think pipenv should succeed installation, either with a warning or after a prompt. At the very least, a better error message should be provided.

I imagine the --pre flag as being relevant only in the context of packages that have full-releases and pre-releases. My perspective is that if I have two choices for the latest, such as 1.3.0 and 2.0a1, I think pipenv should ask which I want. However I understand that this is a feature change and changes how --pre operates. Given the existing behavior but also that people new to pipenv are unaware of the --pre flag, it would be nice to print a message about the flag if an distribution like 2.0a1 is available but the --pre flag has not been specified.

Imagined behavior:

$ pipenv install black # command should succeed ...
Warning! pipenv installed pre-release 18.4a2.

$ pipenv install black # ... or else ask for correct behavior
black does not have a full-release. Install pre-release 18.4a2? y
Installed black 18.4a2

$ pipenv install imagined-package # add warning text about --pre due to alpha version
Warning! imagined-package pre-release 2.0a1 is available. Stable version 1.3.0 was installed. Please use the --pre flag with pipenv if you meant to install version 2.0a1.

$ pipenv install imagined-package --pre # no changed behavior
Installed pre-release 2.0a1

Steps to replicate

Setup:

$ python -V
Python 3.6.4

$ pip -V
pip 10.0.0 from /Users/andrew/.virtualenvs/static-site-test/lib/python3.6/site-packages/pip (python 3.6)

$ rm -r ~/Library/Caches/pip

$ rm -r ~/Library/Caches/pipenv

$ rm -r Pipfile*

$ pip freeze # nothing installed

$ pip install pipenv
Collecting pipenv
  Downloading https://files.pythonhosted.org/packages/e5/fd/740a41c68c38262265d73191fb686f38d26c4ea1ea6080a65b2b626bedc7/pipenv-11.10.0-py3-none-any.whl (5.6MB)
    100% |████████████████████████████████| 5.6MB 403kB/s 
Collecting virtualenv (from pipenv)
  Downloading https://files.pythonhosted.org/packages/ed/ea/e20b5cbebf45d3096e8138ab74eda139595d827677f38e9dd543e6015bdf/virtualenv-15.2.0-py2.py3-none-any.whl (2.6MB)
    100% |████████████████████████████████| 2.6MB 1.3MB/s 
Requirement already satisfied: setuptools>=36.2.1 in /Users/andrew/.virtualenvs/static-site-test/lib/python3.6/site-packages (from pipenv) (39.0.1)
Collecting virtualenv-clone>=0.2.5 (from pipenv)
  Downloading https://files.pythonhosted.org/packages/6d/c2/dccb5ccf599e0c5d1eea6acbd058af7a71384f9740179db67a9182a24798/virtualenv_clone-0.3.0-py2.py3-none-any.whl
Collecting certifi (from pipenv)
  Downloading https://files.pythonhosted.org/packages/7c/e6/92ad559b7192d846975fc916b65f667c7b8c3a32bea7372340bfe9a15fa5/certifi-2018.4.16-py2.py3-none-any.whl (150kB)
    100% |████████████████████████████████| 153kB 1.7MB/s 
Requirement already satisfied: pip>=9.0.1 in /Users/andrew/.virtualenvs/static-site-test/lib/python3.6/site-packages (from pipenv) (10.0.0)
Installing collected packages: virtualenv, virtualenv-clone, certifi, pipenv
Successfully installed certifi-2018.4.16 pipenv-11.10.0 virtualenv-15.2.0 virtualenv-clone-0.3.0

I try to install black as I would with pip.

$ pipenv install black
Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment and create its own instead.
Creating a Pipfile for this project…
Installing black…
Collecting black
  Downloading https://files.pythonhosted.org/packages/e1/e6/dee648e0b1c7997eafa14e51911eeacbed026b2611605bd34d0f824ab6d7/black-18.4a2-py36-none-any.whl (64kB)
Collecting click (from black)
  Downloading https://files.pythonhosted.org/packages/34/c1/8806f99713ddb993c5366c362b2f908f18269f8d792aff1abfd700775a77/click-6.7-py2.py3-none-any.whl (71kB)
Collecting attrs>=17.4.0 (from black)
  Downloading https://files.pythonhosted.org/packages/b5/60/4e178c1e790fd60f1229a9b3cb2f8bc2f4cc6ff2c8838054c142c70b5adc/attrs-17.4.0-py2.py3-none-any.whl
Installing collected packages: click, attrs, black
Successfully installed attrs-17.4.0 black-18.4a2 click-6.7

Adding black to Pipfile's [packages]…
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…

Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
Could not find a version that matches black
Tried: 18.3a0, 18.3a0, 18.3a1, 18.3a1, 18.3a2, 18.3a2, 18.3a3, 18.3a3, 18.3a4, 18.3a4, 18.4a0, 18.4a0, 18.4a1, 18.4a1, 18.4a2, 18.4a2

The error message is incredibly misleading, and is identical if I instead run pipenv install 'black>=18.4a0'.

[jambonrose]

I missed #1760 on my first pass through the issues, but the two are related. I note that my proposal and the feature in #1760 are compatible. Prompting the user to install a pre-release would enable pipenv to use the feature proposed in #1760.

[techalchemy]

I just want to note that behavioral changes will need Kenneth's review and I doubt he will sign off on requiring user input -- I can pretty much say no to that here from a design perspective. The other question about installing a specific package alone without specifying --pre -- this is something I'm not really sure how I feel about. I believe we intentionally fail resolution here because of the nature of pipenv as a dependency resolver -- we don't want people accidentally installing prerelease packages or pinning >=prerelease and having this mean that any new prerelease should be installed even if there is a stable one.

@ncoghlan, do you have any thoughts?

[ncoghlan]

Short answer:

I think it would make sense to have this "just work" (regardless of the --pre setting), perhaps with a printed warning at install time about only pre-releases being available for that particular dependency.

Long answer:

The guiding principle that Warehouse uses for the main web UI (i.e. https://pypi.org/project/black/ ) is to show the non-pre-release with the highest PEP 440 version number, unless the project only has pre-releases, in which case it shows the pre-release with the highest version number. This is the behaviour that PEP 440 recommends: https://www.python.org/dev/peps/pep-0440/#handling-of-pre-releases

The pip CLI will behave the same way as Warehouse if no version qualifier is given (so the pip install black suggestion given in the web UI works), but currently doesn't accept pre-releases as satisfying ordered comparisons, even when no stable release is available (I'm not sure of the rationale for that behaviour - it may be a deliberate design decision I simply wasn't involved in, or else it may be an oversight).

In the case of pipenv, adopting the recommended PEP 440 behaviour (i.e. implicitly allowing pre-releases when nothing else is available) is likely to be the most user-friendly approach: the lock file will still provide stability by default (especially in combination with --keep-outdated), just as it does for any other dependency with an unstable API.

[techalchemy]

Upon discussion I think this is out of scope for now, this has additional complexity and you can already install prereleases by just using a strict pin