Please consider switching the build-system to flit_core to ease setuptools bootstrap

[mgorny]

Could you please consider switching the build system from setuptools to flit_core? This would help Linux distributions such as Gentoo avoid cyclic dependencies that cause bootstrapping unbundled setuptools a real pain. If you agree, I can submit a pull request doing the conversion.

The problem is that the most recent release of setuptools (66.0.0) started using platformdirs. platformdirs use the hatchling build backend which in turn requires this package. This creates a dependency cycle that we can't install setuptools before installing platformdirs, and we can't build platformdirs before all of hatchling's dependencies are installed, and we effectively end up needing setuptools to build them.

flit_core is a "no dependencies [except for tomli, on Python < 3.11]" by design, so it makes bootstrapping packages much easier.

[pfmoore]

I will look into this, but I’m not a particular fan of flit/flit_core, so I would prefer another backend like hatchling. I’m also a little concerned by the idea that code needs to use a particular backend for bootstrapping reasons - the idea of PEP 517 was that projects should be able to choose a backend without restriction, so being required to use flit seems to me to be as problematic as the previous situation where people had to use setuptools.

To help me here, can you explain the problem you are having? Does setuptools support unbundling, and assuming they do, what do they have to say about this? I note that we added in-tree backend support specifically to handle the bootstrapping issue for backends - is it somehow failing to deliver on that promise?

[mgorny]

I will look into this, but I’m not a particular fan of flit/flit_core, so I would prefer another backend like hatchling.

Unfortunately, using hatchling would be impossible because hatchling directly depends on editables. I've actually discussed the bootstrap problems with hatchling's author and they've pointed that this will not be a problem because their dependencies don't use hatchling… Of course, they could work around the problem by vendoring editables but that's not going to help us.

I’m also a little concerned by the idea that code needs to use a particular backend for bootstrapping reasons - the idea of PEP 517 was that projects should be able to choose a backend without restriction, so being required to use flit seems to me to be as problematic as the previous situation where people had to use setuptools.

Unfortunately, PEP 517 authors have chosen to ignore the existence of world outside virtualenvs and the input of Linux distribution maintainers, and disregarded the bootstrap problem as "let build system vendor everything they need" which is completely against Linux distribution policies of devendoring everything.

To help me here, can you explain the problem you are having? Does setuptools support unbundling, and assuming they do, what do they have to say about this? I note that we added in-tree backend support specifically to handle the bootstrapping issue for backends - is it somehow failing to deliver on that promise?

They don't officially claim to support it but they provide some compatibility code to allow using system packages in place of vendored packages. I haven't discussed this problem in detail with them but I do realize it's a lost cause. We're just going to have to patch and fork more and more packages to keep things working. Our users are counting on it.

[pfmoore]

Thanks for the response. I'll be honest here, I don't really want to get sucked into what TBH sounds to me like a conflict between Linux distributors and build backend authors. This should be raised on the packaging forum if it's a general issue for Linux distributors, and we should look for a general solution, not simply "fix" individual projects/dependencies on a case by case basis.

As I said, I'll consider switching backends, but flit is not high on my list of preferred options, and I want to choose what's best for maintenance of this project, not be forced into a particular choice because other projects have bootstrapping issues they are solving in ways that don't suit repackagers.

At a minimum, can you tell me what backends meet your requirements? What about the backend used by PDM? Or what if I bundled a backend (or wrote my own) and used it via the in-tree backend support mechanism? What if I wrote a backend explicitly designed to be copied into projects with bootstrapping issues? One that was by design not intended to be debundled? Where do we draw the line here?

I would be interested if you have any pointers to formal documentation from build backends such as flit_core which do meet your requirements, stating their policy and what long-term intentions they have. I checked the flit documentation (I'm already a little concerned that flit_core isn't even documented as an independent project) and all I could find was a description of how flit_core addressed the issue of bootstrapping flit. I can't be sure that "being able to bootstrap flit" is a strong enough guarantee for the use case you're describing. Also, if you can point to any discussions on this subject by other projects that decided to switch to flit_core for the same reasons as you're raising here, that would be helpful as well.

[mgorny]

At a minimum, can you tell me what backends meet your requirements? What about the backend used by PDM?

To the best of my knowledge, only flit_core can be easily bootstrapped. With Python 3.11, it has no dependencies, and with older versions of Python it depends only on tomli which means we have to preinstall at most tomli + gpep517 (the wheel build runner).

FWICS pdm-pep517 has 5 dependencies, including dependencies using setuptools, plus optional dependency on setuptools for building C extensions.

Or what if I bundled a backend (or wrote my own) and used it via the in-tree backend support mechanism? What if I wrote a backend explicitly designed to be copied into projects with bootstrapping issues? One that was by design not intended to be debundled? Where do we draw the line here?

Well, if you avoid any external dependencies, i.e. basically do something like using zipfile to make a wheel, that would work. However, I honestly doubt there's a real value in going this route.

I would be interested if you have any pointers to formal documentation from build backends such as flit_core which do meet your requirements, stating their policy and what long-term intentions they have.

To be honest, I can't find it right now but I'm pretty sure I've read somewhere that flit's author doesn't plan to add any more dependencies to flit_core. The closest thing that I can find in the docs is:

The key piece is flit_core. This is a package which can build itself using nothing except Python and the standard library. [...]

As of version 3.6, flit_core bundles the tomli TOML parser, to avoid a dependency cycle. If you need to unbundle it, you will need to special-case installing flit_core and/or tomli to get around that cycle.
(https://flit.pypa.io/en/latest/bootstrap.html)

This works really great for everyone because there's a bundled flit_core (and only that one package) for non-distro builds, and the author supports people unbundling it.

[pfmoore]

However, I honestly doubt there's a real value in going this route.

You're probably right. It was intended as an extreme example, to illustrate my point that this isn't a simple yes/no question, but a scale. From what I can tell, though, you're saying that no published build backend other than flit satisfies your needs. That's disappointing, because it puts (IMO unreasonable) demands on projects that get sucked into the "build backend dependency" food chain¹.

This works really great for everyone because there's a bundled flit_core (and only that one package) for non-distro builds, and the author supports people unbundling it.

Yes, I can see why that would make flit an excellent choice for projects from your perspective. But as a project maintainer, I have other priorities. I don't particularly like setuptools as a backend, so I have a mild interest in changing backends, but not to flit, because it offers me no benefits to my workflow as a maintainer, and it makes some choices that I don't agree with. Switching to flit_core with no CLI is closer to what I have with setuptools (i.e., very basic but sufficient) but the fact that the flit maintainer doesn't document the use of flit_core without flit except in the bootstrapping case, makes me uncomfortable. One specific example is that flit build produces a very different sdist than pyproject-build does, unless I work hard to prevent that, which feels to me like "flit_core without flit" is imperfectly supported at best.

As I said, if you can find examples of other projects that made this transition for the reasons you're quoting here, and you can point me to their discussions on that topic, I'll gladly review their discussions. And if you raise this question in its general form on the packaging discourse, I'll contribute my thoughts there. But as things stand, "making it easier for repackagers to debundle setuptools" isn't a compelling argument for me to change this project's build backend to one I personally dislike.

Footnotes

1. You could just as easily ask platformdirs to switch to flit as a backend, becaus it's a direct dependency of setuptools which is the project giving you issues. Or even ask setuptools to self-bootstrap using flit_core. ↩

[mgorny]

Thank you for your consideration. Your points are valid, and to be honest, this is a lost battle. I've just added platformdirs to the list packages where we're overwriting pyproject.toml to make bootstrap possible for us (I suppose it's common enough theme that I should move the generator to a common library, sigh). I'm sorry but I have no motivation to pursue the topic further, as it is draining me psychically.

[pfmoore]

No problem. Sorry the conclusion isn't what you'd wanted. I may raise the question of dependencies for tools at the base of the food chain myself, because I think you have a valid point and I don't think that in-tree backends have worked out as well in practice as we'd hoped. If I do, please don't feel that I expect you to weigh in - I can fully appreciate how draining this stuff can be, so do whatever's best for you.