pipenv won't install on current nixpkgs-unstable

[scoates]

Describe the bug

nix-shell -p python3 -p pipenv fails.

This is because of from pip._internal.cli.main import main', "ModuleNotFoundError: No module named 'pip._internal.cli.main'" which was previously addressed in #71178.

I've been having trouble with pipenv on my main machine (MacOS 10.15/Catalina with /nix provisioned via @steshaw's method) for the past little bit, and I thought I had an isolated problem. Today, I fired up a spare Macbook Pro and installed 10.15, fresh, used the same method for /nix, and ran through the regular curl https://nixos.org/nix/install | sh method. It failed there too.

Here is a full log of this command on a fresh nix install.

To Reproduce

Steps to reproduce the behavior:

1. nix-shell -p python3 -p pipenv

Expected behavior

A working pipenv. What I actually got is at the end of the above-linked log.

Screenshots

Additional context
Brand new nix install. No local overlays. Not even anything installed with nix-env -i ….

Nix channel is:

sean@saarcosm ~ % nix-channel --list
nixpkgs https://nixos.org/channels/nixpkgs-unstable

Metadata

sean@saarcosm ~ % nix-shell -p nix-info --run "nix-info -m"
these paths will be fetched (0.01 MiB download, 0.03 MiB unpacked):
  /nix/store/01ksiwn033npsbhkpj4n6kf8z2inkrmn-nix-info
  /nix/store/wjm15sdvqrpkbxn4hmz1s3hjnsxg5gcd-DarwinTools-1
copying path '/nix/store/wjm15sdvqrpkbxn4hmz1s3hjnsxg5gcd-DarwinTools-1' from 'https://cache.nixos.org'...
copying path '/nix/store/01ksiwn033npsbhkpj4n6kf8z2inkrmn-nix-info' from 'https://cache.nixos.org'...
 - system: `"x86_64-darwin"`
 - host os: `Darwin 19.3.0, macOS 10.15.3`
 - multi-user?: `no`
 - sandbox: `no`
 - version: `nix-env (Nix) 2.3.2`
 - channels(sean): `"nixpkgs-20.03pre212770.cc1ae9f21b9"`
 - nixpkgs: `/Users/sean/.nix-defexpr/channels/nixpkgs`

[matthewmazzanti]

Follow up to this:

TLDR: Python is a pain in the ass. Unset the PYTHONPATH in your shell and you can make it work.

It seems to be that that the PYTHONPATH set by the nix shell clobbers the PYTHONPATH pipenv needs to be able to call pip. When pipenv installs the virtualenv it makes a link to the current python it can find on the PATH and then downloads a NEW pip from pypi into the virtualenv. This new pip executable is basically just a bootstrap script that gets called to find the "real" pip, intending to find the pip it just installed.

So taking a look at the pip executable created by pipenv:

import re
import sys

from pip._internal.cli.main import main

if __name__ == '__main__':
   sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
   sys.exit(main())

The offending line is from pip._internal.cli.main import main. Now the PYTHONPATH set in the nix shell points to the PYTHONPATH of the pipenv's python not of the python pipenv just installed. This seems to get passed through the execution of pipenv, into the above executable, meaning that the first pip source found is the pip installed by Nix, not the pip installed by pipenv. Usually, even though this breaks the purity of pipenv, it doesnt matter because pip is able to execute under a bunch of different versions of python (its still 2.7 compatible...), but the guys managing pip have shifted the location of the main module twice over the last two releases. This means that when the above script intended for the 20.0.2 version pipenv installed tries to import main from _internal.cli.main it errors out because its looking at the 19.3.1 version installed by nix.

Seems that unsetting the python path forces the PYTHONPATH to get set properly by the time pip is called by pipenv. But it also seems like either:

• The way pipenv is being wrapped needs to be changed or...
• Pipenv needs to change how its setting the path for its call of pip.

More research may be needed here to get a good solution.

[scoates]

(Thank you for your analysis. I do really appreciate it.)

Unset the PYTHONPATH in your shell

Is there a good way to do this in shell.nix? I need it to happen after python and before pipenv, right?

[scoates]

FWIW, this appears to fix it: 5ca385e (but also does not yet seem to be in -unstable)

[scoates]

Going to close this one since nix-shell -p pipenv -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/477ea066ca1.tar.gz (which is a build from today) fixes it. nixpkgs-unstable isn't being updated right now for unknown reason, but that's unrelated to this.

[pmyjavec]

Thanks for this!

[DanielRuthardt]

FWIW, this appears to fix it: 5ca385e (but also does not yet seem to be in -unstable)

Just in case anyone else lands here as well, just as I did - this is now in nixpkgs-unstable.