Virtualenv-provided pip3 doesn't work on macOS

[foolip]

I'm trying to follow the instructions at https://web-platform-tests.org/running-tests/from-local-system.html after not having used ./wpt run in a few months, and I can't get it to work. The instructions assume Python 2, which I assume is now wrong, but I can't figure out how to make it work with Python 3 either.

Here's the error I'm getting on macOS after pip3 install --user virtualenv and trying ./wpt --py3 run firefox css/css-flexbox/:

Traceback (most recent call last):
  File "/Users/foolip/web-platform-tests/wpt/_venv3/bin/pip3", line 5, in <module>
    from pip._internal.cli.main import main
ModuleNotFoundError: No module named 'pip._internal.cli.main'
CRITICAL:tools.wpt.utils:('/Users/foolip/web-platform-tests/wpt/_venv3/bin/pip3', 'install', '--prefer-binary', 'zstandard') exited with return code 1
CRITICAL:tools.wpt.utils:b''
Traceback (most recent call last):
  File "./wpt", line 34, in <module>
    wpt.main()
  File "/Users/foolip/web-platform-tests/wpt/tools/wpt/wpt.py", line 155, in main
    venv = setup_virtualenv(main_args.venv, main_args.skip_venv_setup, props)
  File "/Users/foolip/web-platform-tests/wpt/tools/wpt/wpt.py", line 133, in setup_virtualenv
    venv.install(name)
  File "/Users/foolip/web-platform-tests/wpt/tools/wpt/virtualenv.py", line 117, in install
    call(self.pip_path, "install", "--prefer-binary", *requirements)
  File "/Users/foolip/web-platform-tests/wpt/tools/wpt/utils.py", line 57, in call
    return subprocess.check_output(args).decode('utf8')
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/subprocess.py", line 411, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/subprocess.py", line 512, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '('/Users/foolip/web-platform-tests/wpt/_venv3/bin/pip3', 'install', '--prefer-binary', 'zstandard')' returned non-zero exit status 1.

There was a similar error in #21691 and then it had something to do with a pip/python version mismatch, but I don't know what the mismatch is for me.

Does anyone currently have it working on macOS Big Sur and what setup is required?

cc @stephenmcgruer @gsnedders

[foolip]

I had hoped that #27426 would remove a code path I was accidentally going down, but nope.

Do we have a minimum required version of pip which isn't documented anywhere? I'm on 19.2.3.

[foolip]

Oh, but my pip3 at 19.2.3 is /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/pip, the one actually used by WPT seems to be ./_venv3/bin/pip3 which is 20.3.1.

[foolip]

I still don't know why this is happening, but I'm now suspecting that it's virtualenv not doing the right thing, since the pip3 produced in ./_venv3/bin/pip3 looks like this and has the offending line:

# -*- coding: utf-8 -*-
import re
import sys
from pip._internal.cli.main import main
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(main())

[jgraham]

I don't have a mac to try this on, sorry. I hear that @gsnedders knows something about macOS these days :p

[gsnedders]

I don't really have anything helpful to say here. Presumably /Users/foolip/web-platform-tests/wpt/_venv3/bin/pip3 --version and /Users/foolip/web-platform-tests/wpt/_venv3/bin/pip --version both fail similarly?

[foolip]

OK, so here are some things I've tried.

I can get it running if I use virtualenv from the command line:

virtualenv --version # 20.4.2
virtualenv _virtualenv
source _virtualenv/bin/activate
python3 --version # 3.8.2
pip3 --version # 21.0.1
pip3 install -r tools/wptrunner/requirements.txt
pip3 install -r tools/wptrunner/requirements_firefox.txt
./wpt --venv _virtualenv/ --skip-venv-setup run firefox css/css-flexbox/

I can also get things working using python3 -m venv instead of virtualenv:

python3 --version # 3.8.2
python3 -m venv _venvmodule
source _venvmodule/bin/activate
python3 --version # still 3.8.2
pip3 --version # 19.2.3, which won't work so:
pip3 install --upgrade pip
pip3 --version # now 21.0.1
pip3 install -r tools/wptrunner/requirements.txt
pip3 install -r tools/wptrunner/requirements_firefox.txt
./wpt --venv _virtualenv/ --skip-venv-setup run firefox css/css-flexbox/

I suspect the difference is that source _vsomething/bin/activate sets up things correctly in a way that our Virtualenv.activate() isn't able to. That calls exec and presumably whatever environment changes it makes just disappear with that subprocess? @jgraham do you know how this is supposed to work?

The fact that I could get this working without using virtualenv also makes me want to revisit #20019, I'll comment there.

[foolip]

@gsnedders sorry I missed your question as I was writing my long comment.

I don't really have anything helpful to say here. Presumably /Users/foolip/web-platform-tests/wpt/_venv3/bin/pip3 --version and /Users/foolip/web-platform-tests/wpt/_venv3/bin/pip --version both fail similarly?

After the ./wpt run command has set up my _venv and failed once, indeed _venv3/bin/pip3 and _venv3/bin/pip are both the same version, 21.0.1. As one would then expect, changing "pip3" to "pip" in virtualenv.py's pip_path doesn't change the failure mode.

Since invoking the same ./_venv3/bin/pip3 directly does work, my best guess is that this has something to do with how activate() works, that the sys.path doesn't end up in the same order, or something. However, I have checked by adding import pip and printing pip.__version__ in my pip3 wrapper script that it is actually importing pip 21.0.1, so it's simply getting the wrong one.

@gsnedders have you recently been able to just follow the documentation for getting things working on macOS, with no trouble?

[gsnedders]

@gsnedders have you recently been able to just follow the documentation for getting things working on macOS, with no trouble?

With the proviso that my Python installs aren't necessarily "normal", I've not had any problem, but I also don't touch WPT tooling that much at the moment.

[foolip]

@gsnedders ah, so I take it you're not using the python3 that comes with macOS or Xcode?

I do wonder if I've somehow messed up my setup, or if most people trying to ./wpt run on a mac will have this problem. But I don't have a new mac to test that on :)

[foolip]

After carefully getting rid of non-system versions of Python and site-packages this is still not working for me, so I debugged some more. I think the problem is with the order of things in sys.path. Initially (in ./wpt after import sys) it is this for me:

/Users/foolip/web-platform-tests/wpt
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python38.zip
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/lib-dynload
/Users/foolip/Library/Python/3.8/lib/python/site-packages
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages

My system-provided pip3 is the one with its internals in /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/pip, so at the end of that list. But I don't have any pip in /Users/foolip/Library/Python/3.8/lib/python/site-packages so it's the one that should be found.

sys.path is prepended to with a lot of stuff like /Users/foolip/web-platform-tests/wpt/tools/wptrunner.

I think the problem is what the generated "activate_this.py" ends up doing. After the exec(f.read(), {"__file__": path}) in our virtualenv.py, /Users/foolip/web-platform-tests/wpt/_venv3/lib/python3.8/site-packages is prepended as the very first entry on sys.path. So far so good, because the pip in that directory is the one that should be used for this to work.

But, logging sys.path in _venv3/bin/pip3 as invoked from call(self.pip_path, "install", "--prefer-binary", *requirements) shows a problem:

/Users/foolip/web-platform-tests/wpt/_venv3/bin
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python38.zip
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/lib-dynload
/Users/foolip/Library/Python/3.8/lib/python/site-packages
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages

That's exactly the same as the original sys.path apart from the first entry, which doesn't matter. If _venv3/bin/pip3 is invoked separately, the sys.path will be:

/Users/foolip/web-platform-tests/wpt/_venv3/bin
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python38.zip
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/lib-dynload
/Users/foolip/web-platform-tests/wpt/_venv3/lib/python3.8/site-packages

And that would work, because it get rids of the site-packages dir that has the system pip, and adds in the pip installed by virtualenv.

This seems like a difference that might happen on all platforms but just happens to work if the system pip is sufficient, but I've confirmed that on my Linux workstation, sys.path in _venv3/bin/pip3 is actually the same both when invoked directly and via our virtualenv.py.

Still don't know what to do about this.

[foolip]

OK, so what's the difference between invoking _venv3/bin/pip3 from the command line and from inside ./wpt run? Turns out the relevant difference is a __PYVENV_LAUNCHER__. If I tweak call(*args) to make a filtered_env copy of os.environ that removes __PYVENV_LAUNCHER__ and passes that as subprocess.check_output(args, env=filtered_env), things work!

(The value of os.environ["__PYVENV_LAUNCHER__"] for me is '/Applications/Xcode.app/Contents/Developer/usr/bin/python3'.)

https://stackoverflow.com/q/26323852 gives some clues about what __PYVENV_LAUNCHER__ is, and links to python/cpython#9516. What it boils down to, I think, is that the presence of __PYVENV_LAUNCHER__ changes the behavior of later invocations of system Python binaries, which _venv/bin/python is a symlink to.