gh-36358: CI: Fix the multi-stage build in ci-linux.yml, fix conda/ce…

…ntos/archlinux system packages

    
<!-- ^^^^^
Please provide a concise, informative and self-explanatory title.
Don't put issue numbers in there, do this in the PR body below.
For example, instead of "Fixes #1234" use "Introduce new method to
calculate 1+1"
-->
<!-- Describe your changes here in detail -->
Fixes breakage from #36338 seen in https://github.com/sagemath/sage/acti
ons/runs/6332061118/job/17206153694:

The 2nd stage (`...-with-targets`) does not find the container from the
1st stage (`...-with-targets-pre`) because they are keyed to the commit
sha. Fixed here by keying it to the sha before merging the CI fixes.

Also fixing the following failures:
- `conda-forge-minimal` (as seen in https://github.com/sagemath/sage/act
ions/runs/6207962004/job/16858998561); failed because the python3
depcheck packages were missing. Fixed now - check with `tox -e docker-
conda-forge-minimal -- config.status`
- `centos-stream-9-python3.9-minimal` (https://github.com/sagemath/sage/
actions/runs/6207962004/job/16858997494),
`almalinux-9-python3.11-minimal` (https://github.com/sagemath/sage/actio
ns/runs/6207962004/job/16858997663) and `-standard`; fixes #34786
- `archlinux-standard` - libgiac was missing in at least one run

Also adding a few more uses of GH Action output groups for convenience.

Tests running at https://github.com/mkoeppe/sage/actions/workflows/ci-
linux.yml

<!-- Why is this change required? What problem does it solve? -->
<!-- If this PR resolves an open issue, please link to it here. For
example "Fixes #12345". -->
<!-- If your change requires a documentation PR, please link it
appropriately. -->

### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. -->
<!-- If your change requires a documentation PR, please link it
appropriately -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
<!-- Feel free to remove irrelevant items. -->

- [x] The title is concise, informative, and self-explanatory.
- [x] The description explains in detail what this PR is about.
- [ ] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation accordingly.

### ⌛ Dependencies

<!-- List all open PRs that this PR logically depends on
- #12345: short description why this is a dependency
- #34567: ...
-->

<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
    
URL: #36358
Reported by: Matthias Köppe
Reviewer(s): Dima Pasechnik

.github/workflows/docker.yml

@@ -179,12 +179,6 @@ jobs:
           (export PATH=$(pwd)/build/bin:$PATH; (cd upstream && bash -x update-pkgs.sh) && sed -i.bak '/upstream/d' .dockerignore && echo "/:toolchain:/i ADD upstream upstream" | sed -i.bak -f - build/bin/write-dockerfile.sh && git diff)
         if: inputs.upstream_artifact
 
-      - name: Merge CI fixes from sagemath/sage
-        run: |
-          .ci/merge-fixes.sh
-        env:
-          GH_TOKEN: ${{ github.token }}
-
       - name: Try to login to ghcr.io
         if: inputs.docker_push_repository != ''
         # https://docs.github.com/en/actions/reference/workflow-commands-for-github-actions#setting-an-environment-variable
@@ -197,6 +191,9 @@ jobs:
               echo "DOCKER_PUSH_REPOSITORY=$(echo ${{ inputs.docker_push_repository }} | tr "[:upper:]" "[:lower:]")" >> $GITHUB_ENV
               echo "DOCKER_CONFIG_FILE=$HOME/.docker/config.json" >> $GITHUB_ENV
             fi
+            # This line needs to be run before the step "Merge CI fixes from sagemath/sage".
+            DOCKER_TAG="$(git describe --dirty --always)"
+            echo "DOCKER_TAG=$DOCKER_TAG" >> $GITHUB_ENV
             # From the docker documentation via .ci/update-env.sh:
             # "A tag name must be valid ASCII and may
             # contain lowercase and uppercase letters, digits, underscores, periods and
@@ -211,6 +208,15 @@ jobs:
                 EXTRA_DOCKER_TAGS="dev $EXTRA_DOCKER_TAGS";;
             esac
             echo "EXTRA_DOCKER_TAGS=$EXTRA_DOCKER_TAGS" >> $GITHUB_ENV
+
+      - name: Merge CI fixes from sagemath/sage
+        # This step needs to happen after the commit sha is put in DOCKER_TAG
+        # so that multi-stage builds can work correctly.
+        run: |
+          .ci/merge-fixes.sh
+        env:
+          GH_TOKEN: ${{ github.token }}
+
       - name: Configure and build Sage distribution within a Docker container
         run: |
           set -o pipefail; EXTRA_DOCKER_BUILD_ARGS="--build-arg NUMPROC=4 --build-arg USE_MAKEFLAGS=\"-k V=0 SAGE_NUM_THREADS=3\"" tox -e $TOX_ENV -- $TARGETS 2>&1 | sed "/^configure: notice:/s|^|::warning file=artifacts/$LOGS_ARTIFACT_NAME/config.log::|;/^configure: warning:/s|^|::warning file=artifacts/$LOGS_ARTIFACT_NAME/config.log::|;/^configure: error:/s|^|::error file=artifacts/$LOGS_ARTIFACT_NAME/config.log::|;"

.github/workflows/scan-logs.sh

@@ -5,9 +5,13 @@ LOGS=${1-logs}
 for a in $(find "$LOGS" -type f -name "*.log"); do
     if tail -100 "$a" 2>/dev/null | grep "^[A-Za-z]*Error" >/dev/null; then
         echo :":"error file=$a:":" ==== ERROR IN LOG FILE $a ====
+        echo "::group::$a"
         cat "$a"
+        echo "::endgroup::"
     elif tail -20 "$a" 2>/dev/null | grep -E "^(Warning: Error testing|^sage .*doctest.*failed)" >/dev/null; then
         echo :":"warning file=$a:":" ==== TESTSUITE FAILURE IN LOG FILE $a ====
+        echo "::group::$a"
         cat "$a"
+        echo "::endgroup::"
     fi
 done

build/bin/write-dockerfile.sh

@@ -233,11 +233,11 @@ ARG EXTRA_CONFIGURE_ARGS=""
 EOF
 if [ ${WITH_SYSTEM_SPKG} = "force" ]; then
     cat <<EOF
-$RUN echo "****** Configuring: ./configure --enable-build-as-root $CONFIGURE_ARGS \${EXTRA_CONFIGURE_ARGS} *******"; ./configure --enable-build-as-root $CONFIGURE_ARGS \${EXTRA_CONFIGURE_ARGS} || (echo "********** configuring without forcing ***********"; cat config.log; ./configure --enable-build-as-root; cat config.log; exit 1) $ENDRUN
+$RUN echo "****** Configuring: ./configure --enable-build-as-root $CONFIGURE_ARGS \${EXTRA_CONFIGURE_ARGS} *******"; ./configure --enable-build-as-root $CONFIGURE_ARGS \${EXTRA_CONFIGURE_ARGS} || (echo "********** configuring without forcing ***********"; echo "::group::config.log"; cat config.log; echo "::endgroup::"; ./configure --enable-build-as-root; echo "::group::config.log"; cat config.log; echo "::endgroup::"; exit 1) $ENDRUN
 EOF
 else
     cat <<EOF
-$RUN echo "****** Configuring: ./configure --enable-build-as-root $CONFIGURE_ARGS \${EXTRA_CONFIGURE_ARGS} *******"; ./configure --enable-build-as-root $CONFIGURE_ARGS \${EXTRA_CONFIGURE_ARGS} || (cat config.log; exit 1) $ENDRUN
+$RUN echo "****** Configuring: ./configure --enable-build-as-root $CONFIGURE_ARGS \${EXTRA_CONFIGURE_ARGS} *******"; ./configure --enable-build-as-root $CONFIGURE_ARGS \${EXTRA_CONFIGURE_ARGS} || (echo "::group::config.log"; cat config.log; echo "::endgroup::"; exit 1) $ENDRUN
 EOF
 fi
 cat <<EOF

build/pkgs/_bootstrap/distros/arch.txt

@@ -1,4 +1,3 @@
 # Packages needed for ./bootstrap
 autoconf automake libtool
-# Otherwise: Error: could not locate the pkg-config autoconf macros. These are
-pkg-config
+pkgconf

build/pkgs/_python3.10/distros/fedora.txt

@@ -1,2 +1,2 @@
-python310
-python310-devel
+python3.10
+python3.10-devel

build/pkgs/_python3.11/distros/fedora.txt

@@ -1,2 +1,2 @@
-python311
-python311-devel
+python3.11
+python3.11-devel

build/pkgs/_python3.12/distros/fedora.txt

@@ -1,2 +1,2 @@
-python312
-python312-devel
+python3.12
+python3.12-devel

build/pkgs/_python3.9/distros/fedora.txt

@@ -1,2 +1,3 @@
-python39
-python39-devel
+python3.9
+python3.9-devel
+# Except on centos-stream-8 and almalinux-8, where it is called python39 and python39-devel; we special-case this in tox.ini

build/pkgs/freetype/distros/debian.txt

@@ -1 +1 @@
-libfreetype6-dev
+libfreetype-dev

build/pkgs/pkgconf/distros/arch.txt

@@ -0,0 +1 @@
+pkgconf

tox.ini

@@ -142,6 +142,7 @@ passenv =
                   TARGETS_OPTIONAL
     docker:       EXTRA_DOCKER_BUILD_ARGS
     docker:       EXTRA_DOCKER_TAGS
+    docker:       DOCKER_TAG
                   # Use DOCKER_BUILDKIT=1 for new version - for which unfortunately we cannot save failed builds as an image
     docker:       DOCKER_BUILDKIT
     docker:       BUILDKIT_INLINE_CACHE
@@ -221,11 +222,10 @@ setenv =
     ubuntu-focal:     BASE_TAG=focal
     ubuntu-jammy:     BASE_TAG=jammy
     ubuntu-lunar:     BASE_TAG=lunar
-    ubuntu-lunar:                              IGNORE_MISSING_SYSTEM_PACKAGES=yes
     ubuntu-mantic:    BASE_TAG=mantic
-    ubuntu-mantic:                             IGNORE_MISSING_SYSTEM_PACKAGES=yes
     #
     # https://hub.docker.com/_/debian
+    # debian-buster does not have libfreetype-dev (it only has libfreetype6-dev)
     # debian-bullseye does not have libgiac-dev
     # debian-trixie does not have libbrial-dev
     #
@@ -236,6 +236,7 @@ setenv =
     debian-stretch:   BASE_TAG=stretch
     debian-stretch:                            IGNORE_MISSING_SYSTEM_PACKAGES=yes
     debian-buster:    BASE_TAG=buster
+    debian-buster:                             IGNORE_MISSING_SYSTEM_PACKAGES=yes
     debian-bullseye:  BASE_TAG=bullseye
     debian-bullseye:                           IGNORE_MISSING_SYSTEM_PACKAGES=yes
     debian-bookworm:  BASE_TAG=bookworm
@@ -327,10 +328,11 @@ setenv =
     gentoo:                                    IGNORE_MISSING_SYSTEM_PACKAGES=no
     #
     # https://hub.docker.com/_/archlinux/
+    # 2023-09: libgiac went missing, hence IGNORE_MISSING_SYSTEM_PACKAGES=yes
     #
     archlinux:      SYSTEM=arch
     archlinux:      BASE_IMAGE=archlinux
-    archlinux:                                 IGNORE_MISSING_SYSTEM_PACKAGES=no
+    archlinux:                                 IGNORE_MISSING_SYSTEM_PACKAGES=yes
     #
     # https://hub.docker.com/r/vbatts/slackware
     #
@@ -535,7 +537,11 @@ setenv =
     #  - python
     #
     # Setting "--with-system-python3=yes" explicitly in case we change the configure default
-    # to "--with-system-python3=force" as originally proposed in #32060
+    # to "--with-system-python3=force" as originally proposed in #32060.
+    #
+    # For all factors that set "--with-system-python3=force",
+    # we add "_bootstrap liblzma bzip2 libffi libpng zlib" to EXTRA_SAGE_PACKAGES_5
+    # so that the configured python3 can be accepted by configure.
                                                                  PYTHON_MAJOR=3
                                                                    PYTHON_MINOR=10
     python3.9:                                                     PYTHON_MINOR=9
@@ -546,16 +552,22 @@ setenv =
     python3_spkg:                                                    CONFIG_CONFIGURE_ARGS_1=--without-system-python3
     python3.9,python3.10,python3.11,python3.12:                      CONFIG_CONFIGURE_ARGS_1=--with-system-python3=force --with-python=python{env:PYTHON_MAJOR}.{env:PYTHON_MINOR}
     python3.9,python3.10,python3.11,python3.12:                        EXTRA_SAGE_PACKAGES_5=_python{env:PYTHON_MAJOR}.{env:PYTHON_MINOR} _bootstrap liblzma bzip2 libffi libpng zlib
+    # As of 2023-9, Xcode 15.0.0, this is Python 3.9.6.
     macos-python3_xcode:                                             CONFIG_CONFIGURE_ARGS_1=--with-system-python3=force --with-python=/usr/bin/python3
-    macos-{python3_xcode,nohomebrew}-{python3.8}:                    CONFIG_CONFIGURE_ARGS_1=--with-system-python3=force --with-python=/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/{env:PYTHON_MAJOR}.{env:PYTHON_MINOR}/bin/python3
+    macos-python3_xcode:                                               EXTRA_SAGE_PACKAGES_5=_bootstrap liblzma bzip2 libffi libpng zlib
+    macos-{python3_xcode,nohomebrew}-{python3.9}:                    CONFIG_CONFIGURE_ARGS_1=--with-system-python3=force --with-python=/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/{env:PYTHON_MAJOR}.{env:PYTHON_MINOR}/bin/python3
     # Homebrew keg installs
-    homebrew-{python3.9,python3.10,python3.11,python3.12}: CONFIG_CONFIGURE_ARGS_1=--with-system-python3=force --with-python={env:HOMEBREW}/opt/python@{env:PYTHON_MAJOR}.{env:PYTHON_MINOR}/bin/python3
+    homebrew-{python3.9,python3.10,python3.11,python3.12}:           CONFIG_CONFIGURE_ARGS_1=--with-system-python3=force --with-python={env:HOMEBREW}/opt/python@{env:PYTHON_MAJOR}.{env:PYTHON_MINOR}/bin/python3
     # Installers from https://www.python.org/downloads/macos/ (must manually download and install)
     macos-python3_pythonorg:                                         CONFIG_CONFIGURE_ARGS_1=--with-system-python3=force --with-python=/Library/Frameworks/Python.framework/Versions/{env:PYTHON_MAJOR}.{env:PYTHON_MINOR}/bin/python3
+    macos-python3_pythonorg:                                           EXTRA_SAGE_PACKAGES_5=_bootstrap liblzma bzip2 libffi libpng zlib
     # https://github.com/pypa/manylinux
     manylinux-standard:                                              CONFIG_CONFIGURE_ARGS_1=--with-system-python3=force --with-python=/opt/python/cp{env:PYTHON_MAJOR}{env:PYTHON_MINOR}-cp{env:PYTHON_MAJOR}{env:PYTHON_MINOR}/bin/python3
     manylinux-{python3.9,python3.10,python3.11,python3.12}:            EXTRA_SAGE_PACKAGES_5=_bootstrap liblzma bzip2 libffi libpng
     conda:                                                           CONFIG_CONFIGURE_ARGS_1=--with-system-python3=force --with-python=python3
+    conda:                                                             EXTRA_SAGE_PACKAGES_5=_bootstrap liblzma bzip2 libffi libpng zlib
+
+    {centos-stream,almalinux}-8-python3.9:                             EXTRA_SYSTEM_PACKAGES=python39 python39-devel
     #
     #  - toolchain
     #
@@ -706,7 +718,7 @@ commands =
     docker:        bash -c 'for docker_target in {env:DOCKER_TARGETS:with-targets}; do \
     docker:            BUILD_IMAGE_STEM=sage-$(echo {envname} | sed "s/docker-//;s/-incremental//"); \
     docker:            BUILD_IMAGE=$DOCKER_PUSH_REPOSITORY$BUILD_IMAGE_STEM-$docker_target; \
-    docker:            BUILD_TAG=$(git describe --dirty --always); \
+    docker:            BUILD_TAG={env:DOCKER_TAG:$(git describe --dirty --always)}; \
     docker:            TAG_ARGS=$(for tag in $BUILD_TAG {env:EXTRA_DOCKER_TAGS:}; do echo --tag $BUILD_IMAGE:$tag; done); \
     docker:            DOCKER_BUILDKIT={env:DOCKER_BUILDKIT:0} \
     docker:            docker build . -f {envdir}/Dockerfile \