pip build command

[pradyunsg]

Phase 2 from #5407

Allow pip to build packages for distribution - pip build

Some notes in #5407 (comment)

---

pip now has PEP 517 support and uses build isolation by default on packages with pyproject.toml files -- which means the way pip builds is difficult to reproduce externally in a trivial manner.

The pip build command would use the same code paths as PEP 517 with the sole difference being that it does not install the packages. It's essentially the pip wheel in a PEP 517 world.

[pradyunsg]

Related Discussion: pypa/packaging-problems#219

[pfmoore]

Some things that need to be considered, transferred from that discussion:

1. Presumably pip build should build wheels and sdists - whether both formats or just one to be controlled by the user.
2. This introduces a huge overlap with pip wheel, so do we remove/deprecate pip wheel?
3. However, pip wheel will download wheels rather than building them if possible - this seems to go against the intent of a pip build command, so do we drop/relocate the functionality of "download existing wheels"? There's also pip download --only-binary :all:.
4. Should pip build build dependencies as well? pip wheel currently does, as a precedent. But setup.py sdist doesn't, as a precedent for the alternative behaviour...
5. We need to make sure we don't lose the ability to support the use case of "get wheels for the following requirements (and their dependencies) by the quickest method possible, only building if necessary", as that's a core use case for populating a local wheelhouse for offline use.

Overall, I think pip needs a review of all of its download/build functionality, with the intention of rationalising it into a better organised set of subcommands.

[dholth]

I've never been convinced that building the sdist was analogous to building the binary distribution or building to install "setup.py install". It would make sense for one command to be able to create more than one kind of binary.

[pradyunsg]

Get wheels for the following requirements (and their dependencies) by the quickest method possible, only building if necessary

Do you reckon the following is acceptable as an alternative to the current pip wheel way of doing this?

$ pip download -r requirements.txt
$ pip build *.tar.gz *.zip

[pfmoore]

I hadn't really thought through how it would get done, but yes, that seems simple enough to be an acceptable approach.

Worth documenting as part of the transition, though - it's not immediately obvious that the average user would think of this. (More accurately I didn't think of it and I'd prefer to believe against all the odds that this means that it's not obvious, rather than just that I'm dumber than average 😉)

[pradyunsg]

I think the more interesting bit is that we now have 2 kinds of dependencies - build time and run time. This means even a pip build that does just "build a wheel from a source distribution" would still need all of our PackageFinder related options (AKA "index options"). :)

I have a draft structure in my head -- I'll post it when I find a bit of time to flesh it out.

[pradyunsg]

FTR - I've forgotten what I had in mind but I'll be working on pip's build logic in the coming months so I might pick this up.

[pganssle]

I would like to suggest that this is probably a bad idea at least if the motivation is to create a recommended build tool to replace setup.py sdist bdist_wheel. I think instead we should have a simple dedicated tool for this purpose rather than a pip subcommand.

One reason for this is that build doesn't obviously fit in either pip or twine - the only reason it makes sense for pip to have a build command is that pip already needs the ability to build packages in order to install them from source, but pip is generally a tool for consumers of packages, not for producers of packages. twine is usually for producers of packages but it has historically only acted upon already existing distributions and so while conceptually it makes sense for twine to be able to do the builds, it would in some sense change the nature of the tool from a tool that uploads things to PyPI into one that builds packages and uploads them to PyPI. To me, the fact that build has a reasonable claim to live in either package makes me think that build is its own thing and should be given its own package.

I'll note that while it seems like pip "already needs the capability to do builds", it in fact currently doesn't need the ability to build source distributions, since it is perfectly capable of building wheels directly from the source. tox both needs and has the ability to build source distributions, but I would not really suggest adding a tox build command to expose this functionality.

Another reason to avoid adding this to pip is that pip already has some unifying themes to its subcommands, including huge reams of command line arguments, some shared and some not. For example, the pip wheel command already exists and is quite valuable for marshalling all the build artifacts one needs to install a given package because it gets you a wheel for the thing you want to install and all its dependencies; this is, however, a terrible and counter-intuitive behavior for a tool for building a wheel for upload - you need to do the magical pip wheel --no-deps -w dist/ . invocation to do what most people actually want to do. If we add a pip build command, we'll suddenly either have to deprecate the existing and useful pip wheel command or we'll confuse everyone by having two slightly different commands with very different purposes and semantics.

The one time where I think it would make sense for pip to have a pip build command would be if pip wants to move to a situation where instead of source builds going source → wheel → install, they would go source → sdist → wheel → install (for a consistent flow that produces artifacts at each stage), that would make sense, but even then I wouldn't want to recommend it as "the way to create distributable artifacts", but rather as a tool for marshalling resources.

[merwok]

It would made sense to me if twine grew or used a PEP 517 builder to produce wheels. In some simple projects where I use flit, the same tool builds and uploads dists.

[pfmoore]

These arguments make sense to me. However, it's worth noting that (at least in my mind) the intention very definitely is to move to a "source → sdist → wheel → install" process.

I'd rather we did that by delegating all of the build processes to a separate library that could be shared among different tools. The pep517 library is probably where that should end up, but we're quite a long way from having that yet. (One big complication is that pep517 uses pip to set up the build environment, but there's no way to configure that copy of pip to work identically to the calling pip in terms of all the various options available).

[pradyunsg]

I'm on board for doing a twine build command instead of pip build. :)

Your comments about the wheel command inspired #6817.

[pganssle]

I'd rather we did that by delegating all of the build processes to a separate library that could be shared among different tools. The pep517 library is probably where that should end up, but we're quite a long way from having that yet.

Yeah, I think we should probably have something that is basically a thin-ish wrapper around pep517 for the build tool with a somewhat more user-friendly name. I don't know if pep517 supports any sort of fallback behavior in the absence of a pyproject.toml file but if not that would probably be the bulk of the "meat" of any wrapper tool (no need for pep517 to grow fallback behavior, IMO), if we want adoption of the builder tool to be semi-independent of adoption of PEP 517.

(One big complication is that pep517 uses pip to set up the build environment, but there's no way to configure that copy of pip to work identically to the calling pip in terms of all the various options available).

Yeah, the biggest complication of the builder tool I think will be that we'll either have to expose the implementation detail that it uses pip under the hood for configuration or we'll create a whole new configuration language for people to learn (that we translate under the hood into pip options).

I would like to see a future where the tool for doing package installations is also configurable, so maybe it's best to expose it as a configurable implementation detail with either a plugin system or some guarantees about how the environment setup utility will be called.

I'm on board for doing a twine build command instead of pip build. :)

@pradyunsg and @merwok's comments illustrate exactly why I think a third tool makes the most sense. build can plausibly but imperfectly fit in any number of places - which I think is a good case for it being conceptually separate from the other tools.

If we're worried about tool proliferation, we may want to create a wrapper tool (ppm, maybe?) that rolls up all the different packaging-related stuff into one CLI and calls out to the individual, unix-philosophy tools (which can be independently installed and used for people who prefer more tightly scoped tools). Yes I realize this creates an "n + 1" tool, but we'll at least be able to have the clear messaging of, "When in doubt, use ppm"

[pradyunsg]

If we're worried about tool proliferation, we may want to create a wrapper tool (ppm, maybe?) that rolls up all the different packaging-related stuff into one CLI and calls out to the individual, unix-philosophy tools (which can be independently installed and used for people who prefer more tightly scoped tools). Yes I realize this creates an "n + 1" tool, but we'll at least be able to have the clear messaging of, "When in doubt, use ppm"

This is where I've reached multiple times, when thinking about this stuff and it makes a lot of sense to me. I know @ncoghlan probably also has thoughts here along similar lines.

ppm

Already a thing. https://pypi.org/project/ppm/

I have a name in mind, if I end up developing this tool but I'd rather not state it publicly; it may get squatted by someone else on PyPI, before me.

I've had 2/3 false-starts of writing such a tool.

[dstufft]

I think worrying about another tool is a tiny bit premature TBH. Quite frankly, I think every time we introduce yet another project/tool people end up more confused.

Libraries are great, splitting up complex topics into multiple libraries are perfectly fine to me, because I don't think most end users are going to interact with the libraries directly, only advanced users who are going to understand the differences better.

This is probably a larger question better suited for discourse, but honestly I think we need to take a holistic view of what our ideal tool looks like. I would try to put the current set of tools out of mind for now, and try to design an experience up front. It can be useful to experiment with a brand new project (or projects) for this, but ultimately the goal shouldn't necessarily be to produce yet another tool, but to come up with an outline of how what we think the destination should be. Once we have a destination in mind, we'll be far better off making the decision about if we should adapt the current tools to provide that OR if we should look at creating new tools. We can also decide then where it makes sense to compromise on our ideal, in order to fit into the real world of what the sheer weight of existing code demands.

Personally I think there are a few major questions we need to ask ourselves:

• Do we want a unified tool experience, or do we want to split the tooling by "user persona" to provide a more focused tool for the different personas.
  • If a unified tool, how do we provide a good experience when commands overlap for the different personas. See for example the "build wheel" case, where consumers are likely to want to pull from PyPI and build for entire dependency chain, whereas producers are likely to want to operate on local artifacts, and only build themselves.
  • If split tools, where do we draw the lines? Consumer and Producer seem like reasonable places but is there better? Personally I think tool wise, drawing the lines around personas rather than functionality is the right way to go, even if that means two tools might have overlap.
• Does our ideal tool include management of environments ala tox or pipenv? Or do we consider environment management to be an external concern and we just operate on an already existing environment?

All that being said, I think trying to follow the "unix philosophy" is a mistake and is actually a pretty poor UX. Yea a lot of nerds grok it because we've caused enough collective brain damage by being forced to use it over time and it works better for the typical unix tools because they generally just come preinstalled. I think it would just add additional complexity to an already confusing landscape of tools for our end users.

Ideally we figure out a rough idea for our ideal "goal", and figure out how to turn our current tools into that, but failing that we can at least better articulate why we needed yet another tool.

[pradyunsg]

I'll start a discourse topic. :)

[pradyunsg]

TADA: https://discuss.python.org/t/building-distributions/2062

[pradyunsg]

I think we should still do this, using pypa/build or our existing code. We already host the logic for this, and I still feel that exposing this to users more directly would be a genuine usability improvement for them.

All frontends that are attempting to encapsulate the user workflow are providing a build command on their CLI as well, for usability reasons.

[marscher]

As a user, I totally agree. Especially since the direct usage of setup.py got discouraged, it'd be an advantage, if pip could create a source archive.

[pfmoore]

PRs are welcome, of course. Otherwise, it'll be a matter of when the pip maintainers get around to this, which might be some time (we're all pretty busy).

[lorengordon]

Ended up here after investigating warnings from setup.py, followed several threads to other discussions, eventually found that this seems to have been addressed in a new(ish) build tool:

• https://github.com/pypa/build