feat: make variable substitution for py_wheel abi, python_tag args

[stonier]

As the title states - expands make variable substitution to abi and python tags.

See the example for usage.

NB: It allows variables to be passed in via toolchains (in which case it appears under ctx.var) or via deps (in which case it appears under dep[].variables)

[stonier]

@alexeagle as discussed in slack.

[alexeagle]

nit: this seems like a strange example to me, would you actually expect users to write a custom rule and understand toolchains just to set these?

https://bazel.build/reference/be/make-variables#custom_variables

Rule and toolchain writers can define...

I think that's referring to us, so if users need a thing spelled $(ABI) I'd expect something in rules_python to provide it.

[rickeylev]

Yeah, I agree it'd be best if the rules just told us this directly and users didn't have to write a custom rule. That said...

We could factor out a helper rule so a user doesn't need to write something a custom rule and use TemplateVariableInfo themselves, e.g.:

py_wheel(..., toolchains=[":vars"])
py_wheel_vars(name="vars", python_tag="py3", abi="abi3", platform="any")

But I'm not sure what value that really has -- it'd be easier to just set those values in the py_wheel() call. If you want to customize how it determines e.g. abi, then you're back to writing a custom rule.

Second, the more I learn about these tags, the more complicated it is to know the right value. We can get most of the way there for most users, though, I think.

PyInfo.uses_shared_libraries is probably the biggest indicator -- if there are no shared libraries, we can probably just set python_tag=py3, abi=none, platform=any as the defaults. This logic would have to live in py_wheel (more specifically, something that consumes the libraries to get that info).

If there are shared libraries, then it gets complicated.

For python tag, we need to use the major.minor python version. We can know this by look at the multi-python flags or the --python3_version flag.

For abi, we need the toolchain to tell us the abi value somehow. Normally this comes from sysconfig. A hack would be to call it in an action; this would work as long as host==target.

For platform tag, this one is complicated from what I'm told. I think a simple mapping of e.g. (linux, x86) -> "linux_x86_64" would go a long way (we even suggest such in the docs). I don't know how we'd know to use "manylinux", though -- from what I'm told, it's a specific docker image with specific versions of things used. It really sounds like it'd be modeled as a separate platform in Bazel-land with its own cc toolchain etc.

Some refs I found:

• pep 427
• pep 425 (seems to talk about abi tag more)
• Py PA docs

[stonier]

I don't believe you can infer the abi/python_tag from the toolchain that works for all use cases. It depends on how and what you are packaging into the wheel.

For instance, we're using pybind11 to create bindings around c++ libraries. These bindings are major.minor locked since pybind11 uses features outside the stable C API. Build a wheel with python3.8 and you can't use it with python3.10. End result, we need to set cp38 on wheels that package our python3.8 bindings.

There is work going on to improve that state of affairs (e.g. python/cpython#93012), so at some point, we will need to use, e.g. cp38 for older versions and perhaps abi3 when built with some later version of py3 where pybind11 can guarantee c python stability. In this case you have no way of being able to infer the correct abi tag from the toolchain without knowing what is going on with pybind11.

Internally, we're using a custom rule to introspect the currently resolved toolchain to determine an abi tag that is appropriate for our pybind11 wheels. That then gets fed into py_wheel. With this pipeline, we can build python versions for multiple python toolchains without having to hardcode any toolchain information into the bazel BUILD files. You just specify your py3XY toolchain on the command line (via --extra-toolchains) and the rest just works.

[stonier]

NB: It'd be super useful if python toolchains exposed their major.minor.patch version somehow.

[stonier]

Second, the more I learn about these tags, the more complicated it is to know the right value. We can get most of the way there for most users, though, I think.

Aye, I think the non-shared library case could get automagic handling. Problems in the shared library cases.

[rickeylev]

You just specify your py3XY toolchain on the command line (via --extra-toolchains) and the rest just works.

Nice!

without knowing what is going on with pybind11.

Hm, yeah, the toolchains aren't enough -- we would need e.g. the pybind rules (or whatever extension-building rules) to also communicate their abi needs. Then something like py_wheel could look at it them to figure out what to use. Which may not always have a single answer, which is ok. Actually, I wonder if an aspect would help here...aight i'm just thinking out loud now.

[rickeylev]

Do you have an example of what you're using it for yourself? I'm just thinking that a less contrived example would better demonstrate why you'd want to do this.

[stonier]

See above.