pipenv creates virtualenvs in ~/Library/Application Support on OSX, so binaries can be unrunnable

[Julian]

Need to fill this in with full steps to reproduce, since I suspect this should be easily reproduced in a much simpler project than the first one that I'm trying pipenv with, but pipenv is putting virtualenvs in ~/Library/Application Support (which is at first thought the right place to put them on OSX, but:), which can make even pipenv install itself fail:

⊙  pipenv install                                                                                                                                                                                                                                                                                                                                           Julian@Macnetic
Warning: Your Pipfile requires python_version pypy, but you are using 2.7.13 (/Users/Julian/L/A/v/M/bin/python).
  $ pipenv check will surely fail.
Installing dependencies from Pipfile.lock (ee6476)…
Traceback (most recent call last):▉▉▉▉▉ 0/65 — 00:00:00
  File "/Users/Julian/.local/bin/pipenv", line 11, in <module>
    sys.exit(cli())
  File "/Users/Julian/.local/share/virtualenvs/pipenv/site-packages/pipenv/vendor/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/Users/Julian/.local/share/virtualenvs/pipenv/site-packages/pipenv/vendor/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/Users/Julian/.local/share/virtualenvs/pipenv/site-packages/pipenv/vendor/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/Julian/.local/share/virtualenvs/pipenv/site-packages/pipenv/vendor/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/Julian/.local/share/virtualenvs/pipenv/site-packages/pipenv/vendor/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/Users/Julian/.local/share/virtualenvs/pipenv/site-packages/pipenv/cli.py", line 195, in install
    selective_upgrade=selective_upgrade
  File "/Users/Julian/.local/share/virtualenvs/pipenv/site-packages/pipenv/core.py", line 1854, in do_install
    pre=pre, requirements_dir=requirements_directory
  File "/Users/Julian/.local/share/virtualenvs/pipenv/site-packages/pipenv/core.py", line 1394, in do_init
    requirements_dir=requirements_dir.name)
  File "/Users/Julian/.local/share/virtualenvs/pipenv/site-packages/pipenv/core.py", line 877, in do_install_dependencies
    requirements_dir=requirements_dir
  File "/Users/Julian/.local/share/virtualenvs/pipenv/site-packages/pipenv/core.py", line 1495, in pip_install
    c = delegator.run(pip_command, block=block)
  File "/Users/Julian/.local/share/virtualenvs/pipenv/site-packages/pipenv/vendor/delegator.py", line 267, in run
    c.run(block=block, binary=binary)
  File "/Users/Julian/.local/share/virtualenvs/pipenv/site-packages/pipenv/vendor/delegator.py", line 156, in run
    s = PopenSpawn(self._popen_args, **pexpect_kwargs)
  File "/Users/Julian/.local/share/virtualenvs/pipenv/site-packages/pipenv/vendor/pexpect/popen_spawn.py", line 46, in __init__
    self.proc = subprocess.Popen(cmd, **kwargs)
  File "/usr/local/Cellar/pypy/5.10.0_1/libexec/lib-python/2.7/subprocess.py", line 405, in __init__
    errread, errwrite)
  File "/usr/local/Cellar/pypy/5.10.0_1/libexec/lib-python/2.7/subprocess.py", line 1053, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

From dropping into a debugger, what's actually happening there is pipenv is trying to subprocess ~/Library/Application\ Support/virtualenvs/Mux-zp_HsCah/bin/pip, but that file has a shebang with a space in it, so that's not a runnable binary, even though it exists. One could get around that by using ~/Library/Application\ Support/virtualenvs/Mux-zp_HsCah/bin/python -m pip instead (in pipenv.core), but FWIW for this reason in my own virtualenv tool I had to choose to not follow the OSX convention of using Application Support and put binaries in the linux location (~/.local/share/virtualenvs).

[uranusjr]

I don’t think the file being created in ~/Library/Application Support is a problem in on itself (well, it is for Pew, but not Pipenv). The “real” problem is that we should use python -m pip to pip here. Pull requests are welcomed.

[techalchemy]

if virtualenv doesn't properly create shebangs that seems like a virtualenv issue also, I didn't realize that was a problem. But yeah, that pip command can easily be replaced

[uranusjr]

The shebang is a very deeply-rooted problem in POSIX—the spec does not provide a way to handle spaces in executable path at all. The was a long discussion in virtualenv (or pip or Setuptools or bpo, I forget), and the conclusion was something like “POSIX is broken, no can do”. The best thing we can do is to avoid those files altogether.

[Julian]

@uranusjr if pipenv considers the virtualenvs it creates to be public (i.e. if an end user is supposed to be able to run stuff from within it either directly or indirectly), then the same problem would occur for any other binary that the end-user package installs (it wouldn't be runnable without explicitly calling python). If they aren't considered public though, that might not be an issue.

@techalchemy virtualenv makes no guarantees about where you (a user of it) create your virtualenv.

[uranusjr]

Indeed, it would be nice if we could suggest a better default, even provide warnings when user sets it to an ill-adviced value. Again, contributions are welcomed.

[Julian]

@uranusjr how do you feel about the option I chose (putting venvs in ~/.local/share/virtualenvs even on OSX)?

[uranusjr]

Do you happen to know what virtualenvwrapper does on Mac? We can just copy the behaviour (unless they also defaults to ~/Library/Application Support…)

[Julian]

virtualenvwrapper has a WORKON_HOME envvar, which it asks you to set, but if it's not set, it defaults to ~/.virtualenvs -- I find ~/.local/share/virtualenvs to be a better default there, because ~/.local is likely to exist for other reasons even though ~/.local/share technically doesn't have to, but it's one less directory to clutter end-users' $HOME with.

[techalchemy]

doesn't pew do that natively though?

[techalchemy]

(I have no access to a mac, I have literally no clue what happens with pew + mac)

[Julian]

Have never used pew, I use my own tool, but yeah looks like it also uses ~/.local/share even on OSX https://github.com/berdario/pew/blob/master/pew/pew.py#L54

…

On Sun, Jun 10, 2018, 18:32 Dan Ryan ***@***.***> wrote: (I have no access to a mac, I have literally no clue what happens with pew + mac) — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#2321 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAUIXqVf3ZfOltzQtGcB0Y-BG0uTy8pAks5t7Z6JgaJpZM4UeiVW> .

[AlJohri]

happy to test any pew/virtualenvwrapper + mac stuff if needed

[claudeleveille]

I think this has since been fixed, see get_workon_home()'s implementaion in the current latest commit in master.