support for pypy2 and pypy3 #1482

gaborbernat · 2020-01-06T19:55:35Z

Resolves #1470.
Resolves #1471.

Issues I've run into while doing this:

tests/unit/activation/conftest.py

Julian · 2020-01-08T10:13:14Z

I didn't review this carefully, just really skimmed and then tried to run it, but it segfaults for me here as-is:

~/Development/virtualenv is a git repository on remotes/gaborbernat/pypy2
⊙  virtualenv venv                                                                                                                                                      julian@Air
venv/biUsing real prefix '/usr/local/Cellar/pypy/7.2.0/libexec'
  No LICENSE.txt / LICENSE found in source
Path not in prefix '/Users/julian/.local/share/virtualenvs/venvs/include' '/usr/local/Cellar/pypy/7.2.0/libexec'
New pypy executable in /Users/julian/Development/virtualenv/venv/bin/python
Also creating executable in /Users/julian/Development/virtualenv/venv/bin/pypy
^R
venv/bInstalling setuptools, pip, wheel...
^R
venv/bin/python done.

~/Development/virtualenv is a git repository on remotes/gaborbernat/pypy2
⊙  venv/bin/python -m pip install .                                                                                                                                     julian@Air
Processing /Users/julian/Development/virtualenv
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
/Users/julian/Development/virtualenv/venv/site-packages/pip/_vendor/msgpack/fallback.py:133: DeprecationWarning: encoding is deprecated, Use raw=False instead.
  unpacker = Unpacker(None, max_buffer_size=len(packed), **kwargs)
Collecting six<2,>=1.12.0
  Using cached https://files.pythonhosted.org/packages/65/26/32b8464df2a97e6dd1b656ed26b2c194606c16fe163c695a992b36c11cdf/six-1.13.0-py2.py3-none-any.whl
Collecting appdirs>=1.4.3
  Using cached https://files.pythonhosted.org/packages/56/eb/810e700ed1349edde4cbdc1b2a21e28cdf115f9faf263f6bbf8447c1abf3/appdirs-1.4.3-py2.py3-none-any.whl
Collecting entrypoints<1,>=0.3
  Using cached https://files.pythonhosted.org/packages/ac/c6/44694103f8c221443ee6b0041f69e2740d89a25641e62fb4f2ee568f2f9c/entrypoints-0.3-py2.py3-none-any.whl
Collecting pathlib2<3,>=2.3.3; python_version < "3.4" and sys_platform != "win32"
  Using cached https://files.pythonhosted.org/packages/e9/45/9c82d3666af4ef9f221cbb954e1d77ddbb513faf552aea6df5f37f1a4859/pathlib2-2.3.5-py2.py3-none-any.whl
Collecting configparser>=3.5; python_version == "2.7"
  Using cached https://files.pythonhosted.org/packages/7a/2a/95ed0501cf5d8709490b1d3a3f9b5cf340da6c433f896bbe9ce08dbe6785/configparser-4.0.2-py2.py3-none-any.whl
Processing /Users/julian/Library/Caches/pip/wheels/91/95/75/19c98a91239878abbc7c59970abd3b4e0438a7dd5b61778335/scandir-1.10.0-pp272-pypy_41-macosx_10_14_x86_64.whl
Building wheels for collected packages: virtualenv
  Building wheel for virtualenv (PEP 517) ... done
  Created wheel for virtualenv: filename=virtualenv-16.7.10.dev12+g19f8b04-pp272-none-any.whl size=3357799 sha256=b59e078ae20edf12d224f1c9d46a1761255486c32d45d01182572520874a8919
  Stored in directory: /private/var/folders/cr/h4lk12bj5cn4qdx4ct9hddmm0000gn/T/pip-ephem-wheel-cache-GjhcPN/wheels/df/f3/25/b4167a740fa9825ae39b13a337ac65d202940d11f141a8e59e
Successfully built virtualenv
Installing collected packages: six, appdirs, configparser, entrypoints, scandir, pathlib2, virtualenv
Successfully installed appdirs-1.4.3 configparser-4.0.2 entrypoints-0.3 pathlib2-2.3.5 scandir-1.10.0 six-1.13.0 virtualenv-16.7.10.dev12+g19f8b04

~/Development/virtualenv is a git repository on remotes/gaborbernat/pypy2
⊙  venv/bin/pypy -m virtualenv                                                                                                                                          julian@Air
zsh: segmentation fault  venv/bin/pypy -m virtualenv


~/Development/virtualenv is a git repository on remotes/gaborbernat/pypy2
⊙  pypy --version                                                                                                                                                       julian@Air
Python 2.7.13 (4a68d8d3d2fc1faec2e83bcb4d28559099092574, Nov 25 2019, 23:01:16)
[PyPy 7.2.0 with GCC 4.2.1 Compatible Apple LLVM 11.0.0 (clang-1100.0.33.12)]

Will see if I can tell why.

gaborbernat · 2020-01-08T10:19:12Z

could be also https://bitbucket.org/pypy/pypy/issues/3146/pypy3-pypy2-segmentation-fault-on-720; not works on pypy 7.1.0 😢

Julian · 2020-01-08T10:20:50Z

Yeah obviously what's interesting is it's segfaulting not even after I get to creating a new venv with the rewrite, just trying to see the usage info...

Julian · 2020-01-08T10:24:15Z

OK -- whatever it is, it only looks like it affects virtualenvs. I downloaded a fresh tarball from pypy.org and installed this globally, and it works fine on the "global" pypy, but segfaults when installed into a venv.

~/Development/virtualenv is a git repository on remotes/gaborbernat/pypy2
⊙  ./pypy2.7-v7.3.0-osx64/bin/pypy -m virtualenv                                                                                                                        julian@Air

~/Development/virtualenv is a git repository on remotes/gaborbernat/pypy2
⊙  ./pypy2.7-v7.3.0-osx64/bin/pypy -m virtualenv venv; venv/bin/python -m pip install .                                                                                 julian@Air
Processing /Users/julian/Development/virtualenv
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Requirement already satisfied: six<2,>=1.12.0 in ./venv/site-packages (from virtualenv==16.7.10.dev12+g19f8b04) (1.13.0)
Requirement already satisfied: appdirs>=1.4.3 in ./venv/site-packages (from virtualenv==16.7.10.dev12+g19f8b04) (1.4.3)
Requirement already satisfied: entrypoints<1,>=0.3 in ./venv/site-packages (from virtualenv==16.7.10.dev12+g19f8b04) (0.3)
Requirement already satisfied: pathlib2<3,>=2.3.3; python_version < "3.4" and sys_platform != "win32" in ./venv/site-packages (from virtualenv==16.7.10.dev12+g19f8b04) (2.3.5)
Requirement already satisfied: configparser>=3.5; python_version == "2.7" in ./venv/site-packages (from entrypoints<1,>=0.3->virtualenv==16.7.10.dev12+g19f8b04) (4.0.2)
Requirement already satisfied: scandir; python_version < "3.5" in ./venv/site-packages (from pathlib2<3,>=2.3.3; python_version < "3.4" and sys_platform != "win32"->virtualenv==16.7.10.dev12+g19f8b04) (1.10.0)
Building wheels for collected packages: virtualenv
ve  Building wheel for virtualenv (PEP 517) ... -n\/bi|done
  Created wheel for virtualenv: filename=virtualenv-16.7.10.dev12+g19f8b04-pp273-none-any.whl size=3357799 sha256=844b775dcdae01a9df487442ff0d99870896cd34fc023b5bc59d342567009c15
  Stored in directory: /private/var/folders/cr/h4lk12bj5cn4qdx4ct9hddmm0000gn/T/pip-ephem-wheel-cache-6oNNaV/wheels/df/f3/25/b4167a740fa9825ae39b13a337ac65d202940d11f141a8e59e
pythoSuccessfully built virtualenv
nInstalling collected packages: virtualenv
  Found existing installation: virtualenv 16.7.10.dev12+g19f8b04
    Uninstalling virtualenv-16.7.10.dev12+g19f8b04:
      Successfully uninstalled virtualenv-16.7.10.dev12+g19f8b04
Successfully installed virtualenv-16.7.10.dev12+g19f8b04

~/Development/virtualenv is a git repository on remotes/gaborbernat/pypy2
⊙  venv/bin/python -m virtualenv                                                                                                                                        julian@Air
zsh: segmentation fault  venv/bin/python -m virtualenv

Julian · 2020-01-08T10:25:19Z

Hm, and also interesting is python -m virtualenv segfaults, but python -m virtualenv --help works fine even from a virtualenv (side note, maybe the former should imply the latter?)

gaborbernat · 2020-01-08T10:32:07Z

In your case there we already start creating... all parameters have default values. That being said can you try -h -vvv 👍

Signed-off-by: Bernat Gabor <[email protected]>

gaborbernat · 2020-01-08T11:01:17Z

@Julian python -m virtualenv does not imply python -m virtualenv --help 👍 the dest dir defaults to env. Found the issue for the segfault, now should be ok.

Signed-off-by: Bernat Gabor <[email protected]>

Julian · 2020-01-08T11:36:59Z

Ah nice (on both) well done -- hopping on a flight but will give it another run in a few hours.

Signed-off-by: Bernat Gabor <[email protected]>

rlamy · 2020-01-08T19:29:49Z

Trying this in a pypy3-nightly virtualenv, I get:

(pypy3-0102) ~/dev/virtualenv$ python -m pip install .
...
(pypy3-0102) ~/dev/virtualenv$ python -m virtualenv
Traceback (most recent call last):
  File "/home/ronan/pypy-c-jit-98432-1a321e4e7973-linux64/lib-python/3/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/ronan/pypy-c-jit-98432-1a321e4e7973-linux64/lib-python/3/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/ronan/.virtualenvs/pypy3-0102/site-packages/virtualenv/__main__.py", line 22, in <module>
    run()
  File "/home/ronan/.virtualenvs/pypy3-0102/site-packages/virtualenv/__main__.py", line 13, in run
    run_via_cli(args)
  File "/home/ronan/.virtualenvs/pypy3-0102/site-packages/virtualenv/run.py", line 21, in run_via_cli
    session = session_via_cli(args)
  File "/home/ronan/.virtualenvs/pypy3-0102/site-packages/virtualenv/run.py", line 44, in session_via_cli
    creator, seeder, activators = tuple(next(e) for e in elements)  # create types
  File "/home/ronan/.virtualenvs/pypy3-0102/site-packages/virtualenv/run.py", line 44, in <genexpr>
    creator, seeder, activators = tuple(next(e) for e in elements)  # create types
  File "/home/ronan/.virtualenvs/pypy3-0102/site-packages/virtualenv/run.py", line 125, in _get_creator
    creator = creator_class(options, interpreter)
  File "/home/ronan/.virtualenvs/pypy3-0102/site-packages/virtualenv/interpreters/create/venv.py", line 17, in __init__
    self.can_be_inline = interpreter is CURRENT and interpreter.executable == interpreter.system_executable
  File "/home/ronan/.virtualenvs/pypy3-0102/site-packages/virtualenv/interpreters/discovery/py_info.py", line 143, in system_executable
    return self.find_exe_based_of(inside_folder=env_prefix)
  File "/home/ronan/.virtualenvs/pypy3-0102/site-packages/virtualenv/interpreters/discovery/py_info.py", line 162, in find_exe_based_of
    raise RuntimeError("failed to detect {} in {}".format(what, "|".join(possible_folders)))  # pragma: no cover
RuntimeError: failed to detect PyPy3.6.9-64|PyPy3.6.9|PyPy3.6-64|PyPy3.6|PyPy3-64|PyPy3|PyPy-64|PyPy|python3.6.9-64|python3.6.9|python3.6-64|python3.6|python3-64|python3|python-64|python in /home/ronan/pypy-c-jit-98432-1a321e4e7973-linux64/bin|/home/ronan/pypy-c-jit-98432-1a321e4e7973-linux64

The correct name is pypy3 but possible_names only has PyPy3 and variants thereof.

gaborbernat · 2020-01-08T19:57:52Z

We don't include upper/lowercase variants when the file system is case insensitive, can you do -vv with that command, I think the discovery is okay but the host executable part fails 👍

rlamy · 2020-01-08T21:12:41Z

(pypy3-0102) ~/dev/virtualenv$ python -m virtualenv -vv
117 setup logging to DEBUG [DEBUG report:39]
226 find interpreter for spec PythonSpec(str_spec=/home/ronan/.virtualenvs/pypy3-0102/bin/python, path=/home/ronan/.virtualenvs/pypy3-0102/bin/python) [INFO builtin:41]
227 proposed PythonInfo(spec=PyPy3.6.9.final.0-64, exe=/home/ronan/.virtualenvs/pypy3-0102/bin/python, platform=linux, version='3.6.9 (1a321e4e7973, Jan 02 2020, 02:00:14)\n[PyPy 7.3.1-alpha0 with GCC 7.3.1 20180303 (Red Hat 7.3.1-5)]') [INFO builtin:45]
227 accepted target interpreter PythonInfo(spec=PyPy3.6.9.final.0-64, exe=/home/ronan/.virtualenvs/pypy3-0102/bin/python, platform=linux, version='3.6.9 (1a321e4e7973, Jan 02 2020, 02:00:14)\n[PyPy 7.3.1-alpha0 with GCC 7.3.1 20180303 (Red Hat 7.3.1-5)]') [INFO builtin:47]
Traceback (most recent call last):
  File "/home/ronan/pypy-c-jit-98432-1a321e4e7973-linux64/lib-python/3/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/ronan/pypy-c-jit-98432-1a321e4e7973-linux64/lib-python/3/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/ronan/.virtualenvs/pypy3-0102/site-packages/virtualenv/__main__.py", line 22, in <module>
    run()
  File "/home/ronan/.virtualenvs/pypy3-0102/site-packages/virtualenv/__main__.py", line 13, in run
    run_via_cli(args)
  File "/home/ronan/.virtualenvs/pypy3-0102/site-packages/virtualenv/run.py", line 21, in run_via_cli
    session = session_via_cli(args)
  File "/home/ronan/.virtualenvs/pypy3-0102/site-packages/virtualenv/run.py", line 44, in session_via_cli
    creator, seeder, activators = tuple(next(e) for e in elements)  # create types
  File "/home/ronan/.virtualenvs/pypy3-0102/site-packages/virtualenv/run.py", line 44, in <genexpr>
    creator, seeder, activators = tuple(next(e) for e in elements)  # create types
  File "/home/ronan/.virtualenvs/pypy3-0102/site-packages/virtualenv/run.py", line 125, in _get_creator
    creator = creator_class(options, interpreter)
  File "/home/ronan/.virtualenvs/pypy3-0102/site-packages/virtualenv/interpreters/create/venv.py", line 17, in __init__
    self.can_be_inline = interpreter is CURRENT and interpreter.executable == interpreter.system_executable
  File "/home/ronan/.virtualenvs/pypy3-0102/site-packages/virtualenv/interpreters/discovery/py_info.py", line 143, in system_executable
    return self.find_exe_based_of(inside_folder=env_prefix)
  File "/home/ronan/.virtualenvs/pypy3-0102/site-packages/virtualenv/interpreters/discovery/py_info.py", line 162, in find_exe_based_of
    raise RuntimeError("failed to detect {} in {}".format(what, "|".join(possible_folders)))  # pragma: no cover
RuntimeError: failed to detect PyPy3.6.9-64|PyPy3.6.9|PyPy3.6-64|PyPy3.6|PyPy3-64|PyPy3|PyPy-64|PyPy|python3.6.9-64|python3.6.9|python3.6-64|python3.6|python3-64|python3|python-64|python in /home/ronan/pypy-c-jit-98432-1a321e4e7973-linux64/bin|/home/ronan/pypy-c-jit-98432-1a321e4e7973-linux64

gaborbernat · 2020-01-08T21:25:33Z

What system you're running on?

rlamy · 2020-01-09T11:42:34Z

What system you're running on?

Linux (Ubuntu 18.04, to be precise).

Julian · 2020-01-09T12:54:13Z

Gave running this another shot, definitely didn't segfault, though the output on macOS needs tweaking -- it produces for me:

~/Development/virtualenv is a git repository on remotes/gaborbernat/pypy2
⊙  venv/bin/python -m virtualenv                                                                                                                                        julian@Air
filesystem under is%r case-sensitive

Where the output string is literally containing %r because the code mixes % with .format, but probably even without that it shouldn't show that to end users just because macOS is macOS (or really HFS is HFS)?

Julian · 2020-01-09T12:54:47Z

It's super fast though, well done.

Julian · 2020-01-09T12:58:04Z

Just for posterity, posting the difference I get from running python -m site from a virtualenv configured with virtualenv pre and post rewrite:

⊙  env/bin/pypy -m site                                                                                                                                               julian@Air ●
sys.path = [
    '/Users/julian/Development/virtualenv',
    '/usr/local/Cellar/pypy/7.3.0/libexec/lib_pypy/__extensions__',
    '/usr/local/Cellar/pypy/7.3.0/libexec/lib_pypy',
    '/usr/local/Cellar/pypy/7.3.0/libexec/lib-python/2.7',
    '/usr/local/Cellar/pypy/7.3.0/libexec/lib-python/2.7/lib-tk',
    '/usr/local/Cellar/pypy/7.3.0/libexec/lib-python/2.7/plat-darwin',
    '/usr/local/Cellar/pypy/7.3.0/libexec/lib-python/2.7/plat-mac',
    '/usr/local/Cellar/pypy/7.3.0/libexec/lib-python/2.7/plat-mac/lib-scriptpackages',
    '/Users/julian/Development/virtualenv/env/site-packages',
]
USER_BASE: '/Users/julian/.local' (exists)
USER_SITE: '/Users/julian/.local/lib/pypy2.7/site-packages' (doesn't exist)
ENABLE_USER_SITE: False

~/Development/virtualenv is a git repository on remotes/gaborbernat/pypy2
⊙  venv/bin/pypy -m site                                                                                                                                              julian@Air ●
sys.path = [
    '/Users/julian/Development/virtualenv',
    '/Users/julian/Development/virtualenv/venv/lib_pypy/__extensions__',
    '/Users/julian/Development/virtualenv/venv/lib_pypy',
    '/Users/julian/Development/virtualenv/venv/lib-python/2.7',
    '/Users/julian/Development/virtualenv/venv/lib-python/2.7/lib-tk',
    '/Users/julian/Development/virtualenv/venv/lib-python/2.7/plat-darwin',
    '/Users/julian/Development/virtualenv/venv/lib-python/2.7/plat-mac',
    '/Users/julian/Development/virtualenv/venv/lib-python/2.7/plat-mac/lib-scriptpackages',
    '/usr/local/Cellar/pypy/7.3.0/libexec/lib_pypy',
    '/usr/local/Cellar/pypy/7.3.0/libexec/lib-python/2.7',
    '/usr/local/Cellar/pypy/7.3.0/libexec/lib-python/2.7/plat-darwin',
    '/usr/local/Cellar/pypy/7.3.0/libexec/lib-python/2.7/lib-tk',
    '/usr/local/Cellar/pypy/7.3.0/libexec/lib-python/2.7/plat-mac',
    '/usr/local/Cellar/pypy/7.3.0/libexec/lib-python/2.7/plat-mac/lib-scriptpackages',
    '/Users/julian/Development/virtualenv/venv/site-packages',
]
USER_BASE: '/Users/julian/.local' (exists)
USER_SITE: '/Users/julian/.local/lib/python2.7/site-packages' (doesn't exist)
ENABLE_USER_SITE: False

That looks if anything better now with the rewrite? Specifically the paths with lib_pypy relative to cwd look suspicious on pre-rewrite virtualenv, but just posting them here anyhow in case it looks amiss to someone with a better eye.

gaborbernat · 2020-01-09T13:05:59Z

Where the output string is literally containing %r because the code mixes % with .format, but probably even without that it shouldn't show that to end users just because macOS is macOS (or really HFS is HFS)?

the wrong level wanted to provide it as debug, for detecting @rlamy reported issue

Signed-off-by: Bernat Gabor <[email protected]>

gaborbernat · 2020-01-09T16:19:44Z

I'll merge this in as is because I want to push other changes on top; we'll open fixup PRs if we find issues 👍

gaborbernat · 2020-01-09T16:22:26Z

If anyone finds why on CI only (works locally for me) pypy with Windows fails on test https://github.com/pypa/virtualenv/blob/rewrite/tests/unit/activation/test_powershell.py#L14-L18 that would be much appreciated. I've spent three hours trying to figure it out and failed, so I'll cut my further time investment on this to spend on other issues.

gaborbernat added the enhancement label Jan 6, 2020

gaborbernat changed the title ~~support for pypy2~~ support for pypy2 and pypy3 Jan 7, 2020

Julian reviewed Jan 8, 2020

View reviewed changes

tests/unit/activation/conftest.py Outdated Show resolved Hide resolved

Julian reviewed Jan 8, 2020

View reviewed changes

tests/unit/activation/conftest.py Outdated Show resolved Hide resolved

support for pypy2

aba017a

Signed-off-by: Bernat Gabor <[email protected]>

gaborbernat added 2 commits January 8, 2020 11:03

fix that segfault

8ddb5e4

Signed-off-by: Bernat Gabor <[email protected]>

fix Windows

4c42bc8

gaborbernat added 2 commits January 8, 2020 12:21

fix pypy failures

5e8b24f

Signed-off-by: Bernat Gabor <[email protected]>

fix

7fde36b

gaborbernat mentioned this pull request Jan 9, 2020

[Rewrite] Creating a Python 2.7 environment does not copy python27.dll #1484

Closed

gaborbernat added 3 commits January 9, 2020 14:22

fix windows

3e96c7e

Signed-off-by: Bernat Gabor <[email protected]>

fix

4483d66

message

df78c26

Signed-off-by: Bernat Gabor <[email protected]>

gaborbernat merged commit d141da5 into pypa:rewrite Jan 9, 2020

gaborbernat deleted the pypy2 branch January 9, 2020 16:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

support for pypy2 and pypy3 #1482

support for pypy2 and pypy3 #1482

gaborbernat commented Jan 6, 2020 •

edited

Loading

Julian commented Jan 8, 2020 •

edited

Loading

gaborbernat commented Jan 8, 2020 •

edited

Loading

Julian commented Jan 8, 2020

Julian commented Jan 8, 2020

Julian commented Jan 8, 2020 •

edited

Loading

gaborbernat commented Jan 8, 2020

gaborbernat commented Jan 8, 2020 •

edited

Loading

Julian commented Jan 8, 2020

rlamy commented Jan 8, 2020

gaborbernat commented Jan 8, 2020

rlamy commented Jan 8, 2020

gaborbernat commented Jan 8, 2020

rlamy commented Jan 9, 2020

Julian commented Jan 9, 2020

Julian commented Jan 9, 2020

Julian commented Jan 9, 2020

gaborbernat commented Jan 9, 2020

gaborbernat commented Jan 9, 2020

gaborbernat commented Jan 9, 2020

support for pypy2 and pypy3 #1482

support for pypy2 and pypy3 #1482

Conversation

gaborbernat commented Jan 6, 2020 • edited Loading

Julian commented Jan 8, 2020 • edited Loading

gaborbernat commented Jan 8, 2020 • edited Loading

Julian commented Jan 8, 2020

Julian commented Jan 8, 2020

Julian commented Jan 8, 2020 • edited Loading

gaborbernat commented Jan 8, 2020

gaborbernat commented Jan 8, 2020 • edited Loading

Julian commented Jan 8, 2020

rlamy commented Jan 8, 2020

gaborbernat commented Jan 8, 2020

rlamy commented Jan 8, 2020

gaborbernat commented Jan 8, 2020

rlamy commented Jan 9, 2020

Julian commented Jan 9, 2020

Julian commented Jan 9, 2020

Julian commented Jan 9, 2020

gaborbernat commented Jan 9, 2020

gaborbernat commented Jan 9, 2020

gaborbernat commented Jan 9, 2020

gaborbernat commented Jan 6, 2020 •

edited

Loading

Julian commented Jan 8, 2020 •

edited

Loading

gaborbernat commented Jan 8, 2020 •

edited

Loading

Julian commented Jan 8, 2020 •

edited

Loading

gaborbernat commented Jan 8, 2020 •

edited

Loading