Install all Python packages via pip wheel (or setup.py bdist_wheel), store wheels in $SAGE_LOCAL/var/lib/sage/wheels

[mkoeppe]

Our current version of pip already tries to install packages via bdist_wheel.

In this ticket, we break the build process into these steps by modifying sdh_pip_install:

1. Build the wheel explicitly using pip wheel --no-binary :all: --no-build-isolation -w "$SAGE_SPKG_WHEELS" (or setup.py bdist_wheel), which stores the wheel in SAGE_SPKG_WHEELS=$SAGE_LOCAL/var/lib/sage/wheels.

2. Then install the wheel.

The wheels in $SAGE_SPKG_WHEELS persist after the completed build. We manage them using the DESTDIR staging mechanism -- there will be exactly 1 wheel for each installed package (and removing a package removes the wheel). Example (after rebuilding a few packages on this branch):

$ ls -l local/var/lib/sage/wheels/
...
-rw-r--r--  1 mkoeppe  staff    545293 Sep  8 12:44 Pillow-7.2.0-cp37-cp37m-macosx_10_9_x86_64.whl
-rw-r--r--  1 mkoeppe  staff   4573510 Sep  8 12:06 numpy-1.19.1-cp37-cp37m-macosx_10_9_x86_64.whl
-rw-r--r--  1 mkoeppe  staff      2237 Sep  8 12:44 sage_conf-9.2b12-py3-none-any.whl
-rw-r--r--  1 mkoeppe  staff  17981103 Sep  8 12:59 scipy-1.5.2-cp37-cp37m-macosx_10_9_x86_64.whl

Users can create virtual environments and install the wheels built by the Sage distribution into them, using standard tools such as pip install --find-links.

• Example:

  ```
  $ python3 -m venv some-venv
  $ cd some-venv/
  $ bin/pip3 install -v --no-index --find-links=../local/var/lib/sage/wheels/ Pillow    
  

• In follow-up ticket Create PEP 503 simple repository for wheels built during installation #30527, we create a PEP 503 simple repository for the wheels built during installation, which enables more convenient installation options for users.

In this ticket, we keep using the DESTDIR staging mechanism also for the installation of the package from the wheel.

• In a follow-up ticket, we can change this -- to remove the problems described in Remove DESTDIR staging for Python packages to eliminate race conditions during Python package installations #29585 -- by just holding the pip lock (if it is at all still necessary after the major pip upgrades in the 9.2 series) during the wheel installation (step 2).

---

References:

• https://pip.pypa.io/en/stable/user_guide/#installing-from-local-packages

• https://pip.pypa.io/en/stable/reference/pip_cache/

CC: @slel @tobiasdiez @embray @jhpalmieri @tscrim

Component: build

Author: Matthias Koeppe

Branch: f2e7075

Reviewer: Tobias Diez, John Palmieri

Issue created by migration from https://trac.sagemath.org/ticket/29500

[slel]

comment:1

Would that be gambit, numpy and pillow?

And is this how to find them?

$ git grep "setup.py" build/pkgs

[mkoeppe]

comment:5

... Also consider pipenv + pipfile as suggested in #30317 ...

[embray]

comment:7

I would prefer not to have two separate package installation and management systems. If the only motivation of this is the race condition, like I said this can be solved with an additional file lock to prevent multiple Python packages from being installed simultaneously.

[mkoeppe]

Dependencies: #30024

[mkoeppe]

comment:15

Narrowed the scope of the ticket.

[mkoeppe]

Branch: u/mkoeppe/install_all_python_packages_via_pip_wheel__create_pep_503_simple_repository_for_wheels

[mkoeppe]

New commits:

8aa6fd9

build/bin/sage-dist-helpers (sdh_pip_install): Build a wheel, store it

[mkoeppe]

Commit: 8aa6fd9

[mkoeppe]

Author: Matthias Koeppe

[mkoeppe]

Changed dependencies from #30024 to none

[sagetrac-git]

Changed commit from ca58693 to 5a747c4

[jhpalmieri]

comment:28

• Why sdh_store_and_pip_install_wheel requires . as only argument? Why not just call it with no arguments?

• This function should be documented in developer/packaging.rst.

• Does sage-pip-uninstall leave the old wheels lying around?

• Typo in sage-pip-uninstall (copied from sage-pip-install): "this command hould use"

[mkoeppe]

comment:29

Replying to @jhpalmieri:

• Why sdh_store_and_pip_install_wheel requires . as only argument? Why not just call it with no arguments?

I did this to emphasize the analogy to sage-pip-install.

• This function should be documented in developer/packaging.rst.

Good idea, will do.

• Does sage-pip-uninstall leave the old wheels lying around?

sage-pip-uninstall is not really used for regular uninstalls. Rather, it "force-uninstalls" previously installed versions - in case the site-packages directory was somehow messed up. (This code was previously part of sage-pip-install.)

The wheels are installed with the staging mechanism (DESTDIR), which records the installed files. In this way, the -clean targets etc. automatically install the wheel.

• Typo in sage-pip-uninstall (copied from sage-pip-install): "this command hould use"

Thanks

[mkoeppe]

comment:30

Replying to @mkoeppe:

The wheels are installed with the staging mechanism (DESTDIR), which records the installed files. In this way, the -clean targets etc. automatically install the wheel.

To clarify, both the wheels and the installation made from the wheels uses the staging mechanism.

[sagetrac-git]

Changed commit from 5a747c4 to 4135e8b

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

9ee2110	build/bin/sage-dist-helpers: Also use $sudo for storing the wheel file
d7aac84	src/doc/en/developer/packaging.rst: Update sdh_... documentation
9b7c7a0	build/bin/sage-pip-{install,uninstall}: Fix typo in comment
4135e8b	build/bin/sage-pip-install: Remove an outdated comment

[jhpalmieri]

Changed reviewer from Tobias Diez, ... to Tobias Diez, John Palmieri

[jhpalmieri]

comment:32

Seems okay to me, and it built and passed tests before, so I think it's okay. Of course, I can no longer build Sage anymore, so perhaps take this review with a grain of salt.

[mkoeppe]

comment:33

Thanks!

[vbraun]

comment:34

Merge conflict

[sagetrac-git]

Changed commit from 4135e8b to f2e7075

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

f2e7075

Merge tag '9.2.beta13' into t/29500/install_all_python_packages_via_pip_wheel__create_pep_503_simple_repository_for_wheels

[vbraun]

Changed branch from u/mkoeppe/install_all_python_packages_via_pip_wheel__create_pep_503_simple_repository_for_wheels to f2e7075

[mkoeppe]

comment:39

Follow-up: #30771

[mkoeppe]

Changed commit from f2e7075 to none

[embray]

comment:40

I'm not really sure what the advantage is of this complication. For the follow-up do you propose eliminating the installation manifests installed to /var/lib/sage/installed/? I would very much prefer to not do that. I see no reason for Python packages to be arbitrarily treated differently in this regard. But I'm not sure if that is really the plan.

[mkoeppe]

comment:41

Replying to @embray:

I'm not really sure what the advantage is of this complication.

https://wiki.sagemath.org/ReleaseTours/sage-9.2#Reusable_wheels_for_the_Python_packages_built_by_the_Sage_distribution explains an immediate use case.

[mkoeppe]

comment:42

Replying to @embray:

For the follow-up do you propose eliminating the installation manifests installed to /var/lib/sage/installed/?

I don't have a ticket for this at the moment. But I think it's unavoidable to make some changes. Already now these installation manifests (which duplicate the .egg-info record in site-packages) can fall out of sync with the actual installation when the user uses sage -pip install, which could lead to upgrades of some python package dependencies. It's best to use standard Python tools to manage Python packages.