Tags missing from pypy3 on Windows 10

[brettcannon]

Comparing what pip's tags and packaging.tags under pypy3 7.0.0 on Windows 10, the following tags are only found by pip:

1. pp370-pp370-win32
2. pp3-none-any
3. py370-none-any

The first one could be considered legitimate, but the last two do not make sense.

[brettcannon]

Might need to tweak things so that in the generic case if no ABI is specified not only is none used but also an ABI for the name of the interpreter as another fallback.

[brettcannon]

Flagging as a question to ask the PyPy folks, but based on their answer either this issue should be closed or treated as a bug.

[pradyunsg]

@brettcannon Did we manage to ask PyPy folks about this at some point, and which ones does packaging.tags generate today?

[brettcannon]

@pradyunsg nope, I haven't gotten around to worrying about this issue yet.

[pradyunsg]

I'm a tweety bird: https://twitter.com/pradyunsg/status/1180465257426210818

[mattip]

Hi. PyPy dev here. Where did those tags come from, what produced them? I see tags such as cp3-none-any on PyPI, what makes pp3-none-any invalid?

[pradyunsg]

Thanks a lot for chiming in @mattip! ^>^

Where did those tags come from, what produced them?

It's produced as a part of pip's PEP 425 checks: https://www.python.org/dev/peps/pep-0425/.

You can see what pip generates as "compatible with", by running pip debug --verbose on the latest pip, on the relevant interpreter. (warning: it's a long list)

---

I think pp3 makes sense, but I'm not so sure -- my intuition on PEP 425 isn't good. :)

[mattip]

Here is the list for the upcoming pypy release which is python3.6-language compatible pypy7.2, where "language compatible" means "cpython interpreter + stdlib"

Compatible tags: 15
  pp372-pypy3_72-manylinux2010_x86_64
  pp372-pypy3_72-manylinux1_x86_64
  pp372-pypy3_72-linux_x86_64
  pp372-none-manylinux2010_x86_64
  pp372-none-manylinux1_x86_64
  pp372-none-linux_x86_64
  py3-none-manylinux2010_x86_64
  py3-none-manylinux1_x86_64
  py3-none-linux_x86_64
  pp372-none-any
  pp3-none-any
  py372-none-any
  py3-none-any
  py371-none-any
  py370-none-any

The list for cpython3.6 is much longer, it has 34 entries.

The tag pp3-none-any is for packages that have pypy-specific code (like cliquet does for cffi. The tag pp372-*does seem confusing at first glance. It is meant to reflect the PEP 425 format {python tag}-{abi tag}-{platform tag}, where the python tag is "python3 language specification (pp3)" + "pypy version 7.2 (72)". The text says " PyPy should probably use its own versions here pp18, pp19" without diving into the meaning, and we chose a format where the major-minor pypy version is always the suffix. So far we do not have concurrent python3 releases (python3.6 and 3.7 both) so it is clear which python minor version is targeted because the pypy version will always increment.

If we do release two or more concurrent minor versions, we should add the minor version to the python tag, so the "python tag" would become pp3772 for our first python 3.7 release.

[mattip]

The previous comment explains (1) and (2) from the original issue, I agree that (3) is an artifact that could be removed, py371 and py370 make no sense. For reference, cpython produces the following py python tags:

  py3-none-manylinux2010_x86_64
  py3-none-manylinux1_x86_64
  py3-none-linux_x86_64
  py36-none-any
  py3-none-any
  py35-none-any
  py34-none-any
  py33-none-any
  py32-none-any
  py31-none-any
  py30-none-any

So perhaps pypy should produce all those as well. It think this is connected to pypa/pip#3075, in particular a change that was made to get the sys.pypy_version_info as part of get_impl_version_info. While that seems to match the function name, it is used in get_supported where a non-implementation get_version (which does not exist) might have sufficed.

[pradyunsg]

Thanks for your responses here @mattip! Much appreciated! I'll defer to @brettcannon here since I have no intuition for PEP 425's operation).

---

BTW, sorry that I didn't provide more context earlier -- packaging.tags is the intended replacement for pip's pep425tags.py module (pypa/pip#6908).

---

W.R.T. how the tags differ between these two generation mechanisms, I'd suggest using this script (which uses pip's internals):

from __future__ import print_function  # Python 2.7 compatibility
from pip._internal.pep425tags import get_supported
from packaging.tags import sys_tags

pip_tags = {'{}-{}-{}'.format(*tag) for tag in get_supported()}
packaging_tags = {str(tag) for tag in sys_tags()}

li = [
    ("Only in pep425tags:", pip_tags - packaging_tags),
    ("Only in packaging.tags:", packaging_tags - pip_tags),
    ("In both groups:", packaging_tags & pip_tags),
    ("All tags:", packaging_tags | pip_tags),
]
for what, tags in li:
    print(what, "\n  ".join(sorted(tags)), sep="\n  ")
for what, tags in li:
    print(what, len(tags))

[mattip]

It seems (3) is only shows up in pip and could be considered a bug there. I would expect packaging.tags to support both cp3-none-any and pp3-none-any, which currently it does not. packaging.tags also supports extra tags (in both cpython and pypy) that pip did not: ones of the form py33-none-linux_x86_64 with the minor python version. Is that on purpose?

[brettcannon]

@mattip So I think there are two questions for me to answer here, but if I miss any let me know!

First, I consider pp3-none-any invalid because that's a tag saying "I have code specific to PyPy version 3.* of the interpreter"(e.g. PyPy 3.7); this has nothing to do with Python version support. As of now there is no cp3 on purpose because it makes no sense (at least right now) to have code tied to CPython 3.* but not any specific feature version of CPython as the compatibility guarantees for thing uinque to CPython are not the same as the language itself. Now if PyPy is different then we could consider pp3 legitimate, but it would be unique in that regard.

As for your question about whether the difference between packaging.tags and what pip provides is on purpose, the answer is emphatically yes (there were discussions on distutils-sig about this and I spent months diving into all of this). 😄 I wrote packaging.tags specifically to plug holes that pip currently has in its wheel tag support and all the changes are on purpose (especially for PyPy). The goal is to eventually rip out pip's pep425tags module that it has vendored and rely on packaging.tags instead. This issue is just an extension of that work to make sure PyPy is being supported appropriately based on what packaging.tags emits on Windows ATM.

[mattip]

Thanks for explaining, I am slowly getting up to speed. I agree (3) is not useful, and (1) would be better expressed as pp3671 : 36 from sys.version_info and 71 from sys.pypy_version_info

As for (2), I claim that there are reasons to have py3-none-any, cp3-none-any and pp3-none-any since there are implementation detail differences and performance related differences between the implementations that package authors may want to exploit. For instance, on CPython it may be better to depend on third-party c-exension packages, on PyPy it is always better to use CFFI extension modules. In the example I used above (cliquet), they would prefer to use psycopg2>2.5 on CPython and psycopg2cffi>2.7.0 on PyPy. If there was only a py3-none-any tag, how could they differentiate?

[brettcannon]

@mattip so is there the possibility of PyPy3 7.1 supporting another version of Python in some unique way that changes what's required to be in a wheel? If not then the Python version number should be left out as that part of the wheel tag specifies the interpreter version only (the py3/py37 interpreter-agnostic versions are special-cases to cross interpreter implementations). Remember this is all about support for pre-compiled wheels, so unless knowing the Python version on top of PyPy interpreter version somehow conveys a different ABI it should be left out.

For cp3, that concept doesn't exist as CPython is not structured to have extension modules compiled to support across feature releases. And if you do come up with some way then you can use compressed tags like cp37.cp36.cp35 if you wanted to support that concept.

Basically I'm not prepared to bring in a new tag kind for interpreters just for CPython 3 and PyPy3 version-agnostic wheels when there's currently as to my knowledge no package maintainers have asked for them. We can consider bringing this in later, but one thing at a time. 😉

[hodgestar]

Two thoughts:

1. It seems odd to declare that a tag like "cp3-none" is nonsensical given that PEP 425 gives it a clear meaning of "this works with any CPython 3.x but doesn't require a specific ABI". If one wrote a game that used PyGame, supported only Python 3, and had no C-extensions of its own, then it seems that its binary distributions might have "pp3-none" (for pypy, and require pygame_cffi) and "cp3-none" (for CPython, and require ordinary pygame). I do get that it would be very strange for a Python module with a C-extension of its own to have a "cp3-none" tag (and I think it might be odd for such a package to have a "pp3-none" under PyPy too).

2. Python tags like "pp372" seem really confusing. Is that "PyPy 3.7.2" or "PyPy 3 (build 7.2)"? Further, "pp372-none" and "pp372-pypy3_72" seem to say exactly the same thing (i.e. I really need this specific build of pypy). It'd be less confusing if the Python tag only specified the PyPy python version, and information about the precise build of PyPy required was relegated to the ABI tag.

Hopefully even if a lot of what I've written is wrong, it's at least somewhat useful as an example of how a non-bdist-packaging expert reads these tags.

[brettcannon]

@hodgestar consider "nonsensical" as "it doesn't already exist and this stuff is hard enough to get right without adding to it by introducing a new potential tag" 😉 . IOW this is definitely not an area to guess at what people want until someone shows a need for it and then comes asking for it. So if the PyGame folks come to us and ask for that tag, then we can talk.

As for pp372 being confusing, so is cp37dm, so this wouldn't be the first time that a tag isn't easy to understand in isolation. 😉 Once again, we're not trying to reinvent anything as that's what the PyPy3 interpreter tag already is in pip. If the pip or PyPy team want to change it to something like pp3_72 then we can, but that would lead to preexisting wheels to no longer work without also shoving in the old name as well (plus what would you want to do with PyPy for Python 2: keep the underscore for consistency or drop it since the number is smaller?). Basically I'm not ready to make that call on our own nor do I have the time right now to drive that conversation as getting this into pip is way more important.

[mattip]

it doesn't already exist and this stuff is hard enough to get right without adding to it by introducing a new potential tag

The cp3-none-any (and corresponding pp3-none-any) tag exists today.

$ python -mpip debug --verbose 2>/dev/null |grep cp3-
  cp3-none-any

I gave an example of an existing package that has a need for this tag and uses it on PyPI. Removing this tag feels like a regression.

[brettcannon]

@mattip That's an example from pip. It was purposefully dropped through discussion with distutils-sig when I did this work. I even queried PyPI's BigQuery data to make sure it wasn't being used in any significant way (and it wasn't), so this has been thought through. Please consider this repo the canonical list of tags.

And you did give an example of a potential use, but PyGame has not come to us asking for this and they currently do not have a wheel using this tag. As I said, bringing in new tags is not a priority right now. I'm just trying to make sure there are not tag triples using preexisting tags which are missing. It seems that pp370-pp370-win32 is at least a legitimate triple that's missing and so I will work to add that in.

[mattip]

We added a valid sysconfig.get_config_vars('SOABI') for pypy on windows, it will be part of an upcoming release. Other platforms already return a valid value. This means that the ABI tag on PyPy is controlled by the implementation, which I think makes more sense that foisting it upon the packaging tool, and is also within the realm of the comment in PEP 425 ""Each implementation's community may decide how to best use the ABI tag".

[brettcannon]

@mattip thanks for the info! I will then close this issue once #231 gets merged as that will use SOABI by default.