It's not obvious how to construct a set of compatible tags for a system

[pfmoore]

The packaging.tags module exposes a number of functions, but it's not particularly obvious how to construct a list of supported tags for a given system from them (other than the current interpreter).

Pip uses a complicated algorithm (see here) to build the tags. The default result doesn't seem to always match packaging.tags.sys_tags() - on my system (Windows 10 64-bit, Python 3.8) pip debug -v includes cp38-none-any, but sys_tags() does not. And the process for building tags for a different system seems very complex, and there's no indication in the packaging.tags documentation that it's the intended or correct approach.

I understand that in some situations (notably manylinux) it's not possible to determine the correct supported tags from "outside" the system. But as things stand it's not at all clear how to correctly use any of the functions exposed here apart from parse_tag() and sys_tags().

(Background: I'm trying to implement a module that replicates pip's "package finder" logic, and that needs to allow the user to specify the platform, etc, to find packages for).

[pradyunsg]

You've gotta do what pip does -- call the relevant sub-functions with the corresponding optional arguments "populated":

https://github.com/pypa/pip/blob/cb8d81d1356aadf74953bc2a72fc7d10dd1eab17/src/pip/_internal/utils/compatibility_tags.py#L152-L156

Any arguments that are not specified take a "reasonable default" (like none) or derive the appropriate value from the currently-running-system (see lines that say platforms or _platform_tags()).

---

IMO pip's code is not that complicated -- everything above line 140 is "process a value (or default) to respect pip's CLI-design-and-legacy-behavior" and everything beyond that is "call the relevant packaging.tags functions with 100% overrides".

[pradyunsg]

FWIW, once you start passing in the platform to any of the packaging.tags functions, you're also bypassing the logic for detecting manylinux and other platforms' compatibility -- they are populated based on the current interpreter if platforms is not passed in.

[pfmoore]

Note that pip's defaults give a different value than sys_tags(). I don't know whether that counts as "complicated enough that pip got it wrong" or "there's a bug in sys_tags()"...

I am inclined to think that the docs could do with some extra work, though, to clarify that this is the intended use of the lower level parts. I'm happy to work up a documentation patch, and then we can debate whether it's "too complicated" and would benefit from some helper functions 😉.

To be honest, though, I don't really see any point in exposing the lower-level functions except to implement something like pip's behaviour. Is there a good reason why we don't include (an equivalent of) pip's get_supported here, and save everyone the trouble of re-implementing it? And conversely, are there use cases for compatible_tags/cpython_tags/generic_tags except for writing the code pip uses? (I'm asking because I'd like to add that to my documentation patch, not because I'm criticising the design here...)

[pradyunsg]

Those are all questions that @brettcannon is in a much better position to answer than me. :)

[brettcannon]

There's a couple of things to unpack here.

One, "can we document how to build tags with the current API?" No, because the API is specifically limited to not allow for that. 😉 The initial reason was no one asked for that level of granularity, so in the spirit of "it's easier to add than remove", it was left out. But it also isn't all there because constructing the tags appropriately to be compatible across tools based on the priority order of tags is really convoluted. There's weird historical things and such which make explaining it a right pain and easy to get wrong (hence why sys_tags() exists; I mean look at the output and try to explain the interleaving clearly just to yourself as a domain expert). See #330 for discussions about widening the API to allow for this possibility, but the complexity of getting this right is very hard, and creating an API that is somehow pluggable but still does the right thing is a major challenge.

Two, "why do we have some functions in the API?" Because those using pep425tags had some very specific needs (including pip), so only what was necessary got exposed. Otherwise I would not have exposed anything beyond sys_tags() and the data class. 😉

Three, "why is cp38-none-any left out?" When I went through and figured out all the tags I couldn't come up with a reason for its existence (and neither could PyPI based on how I think there were like 2 wheels in existence with that sort of tag triple), so I left it out. It has come up before about adding it back in (see #311), but no one has put in the time to make it happen (and I have at this point say I'm not going to stop it but I'm not doing the work either).

[pfmoore]

Thanks for the reply @brettcannon. I'll take the points in reverse order, because that gets the easier one out of the way first 🙂

With regard to (3), arguably pip should remove it as well. I'll take that up over there. It looks like pip calls compatible_tags(None, "cp38", None), which adds that tag. Although maybe compatible_tags should omit it on the same basis that sys_tags does?

Regarding (1) and (2), that's a fair point. I'd happily only use sys_tags if it weren't for the fact that pip allows users to specify platform, interpreter etc, to download files for "another system". I'm not personally convinced that people can realistically work out what values to supply to get what they want, but that's a debate about pip functionality, again. If we can come up with some plausible way to let users specify what system they want to get tags for, I guess we can revisit the question then, with a view to maybe adding a tags_for(...) function. But the onus is on people who want that functionality to clearly state how they expect it to work.

I'd suggest maybe clarifying in the docs that Tag, parse_tag and sys_tags are intended as the "user-level" API, with the other functions being lower-level APIs for specialised needs. Would a PR to that effect be welcome?

[brettcannon]

• If compatible_tags() is differing from sys_tags due to a lack of info; e.g. because it doesn't make sense for CPython doesn't mean it won't make sense for another interpreter; at this point I say we just add it back in so make it less confusing 😄
• I would be totally happy with a doc PR clarifying what users will care about versus low-level API

[dstufft]

cp38-none-any - someone using ctypes to interact with the CPython ABI could use this tag. Dunno if anyone actually does that or cares though.

[brettcannon]

@dstufft I left a comment on #311 saying that anyone who wants to put in the work for adding the tag in can do so and I won't object.

[pfmoore]

I would be totally happy with a doc PR clarifying what users will care about versus low-level API

I just submitted #334

[brettcannon]

... and now that's merged. 😄

Was there something else you wanted to discuss in this issue, @pfmoore , or should we close this issue?

[pfmoore]

Everything else is covered, thanks. The point about cp38-none-any is covered by the comment on #311, and anyone flagging the difference in what pip returns can be directed to that. So we're done here 🙂