tox / GH Actions: Add ubuntu variants using ppa:ubuntu-toolchain-r #32966

mkoeppe · 2021-12-04T01:59:16Z

see https://askubuntu.com/a/1149383/309919

To test:

https://launchpad.net/~ubuntu-toolchain-r/+archive/ubuntu/test?field.series_filter=trusty:
```
tox -e docker-ubuntu-trusty-standard-toolchain-gcc_9
```
https://launchpad.net/~ubuntu-toolchain-r/+archive/ubuntu/test?field.series_filter=xenial
```
tox -e docker-ubuntu-xenial-standard-toolchain-gcc_9
```

https://launchpad.net/~ubuntu-toolchain-r/+archive/ubuntu/test?field.series_filter=bionic

tox -e docker-ubuntu-bionic-standard-toolchain-gcc_9
tox -e docker-ubuntu-bionic-standard-toolchain-gcc_10
tox -e docker-ubuntu-bionic-standard-toolchain-gcc_11

https://launchpad.net/~ubuntu-toolchain-r/+archive/ubuntu/test?field.series_filter=jammy
```
tox -e docker-ubuntu-jammy-standard-toolchain-gcc_12
```

NB: Your Docker should have enough RAM allowed (I'd advice 8GB). In Docker Desktop this setting is in Preferences->Resources -> Advanced.

Depends on #33187
Depends on #33103

CC: @dimpase @tobiasdiez

Component: porting

Author: Matthias Koeppe

Branch/Commit: d42bc03

Reviewer: Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/32966

The text was updated successfully, but these errors were encountered:

mkoeppe · 2021-12-25T23:11:59Z

Branch: u/mkoeppe/tox___gh_actions__add_ubuntu_variants_using_ppa_ubuntu_toolchain_r

sagetrac-git · 2021-12-25T23:19:43Z

Branch pushed to git repo; I updated commit sha1. New commits:

`de42909`	`build/pkgs/_gcc*/distros: Add homebrew.txt`

sagetrac-git · 2021-12-25T23:19:43Z

Commit: de42909

mkoeppe · 2021-12-28T02:16:09Z

Author: Matthias Koeppe

tobiasdiez · 2022-02-07T12:38:07Z

comment:8

Do we really need to add tests for each possible combination of system packages? This would also mean that we voucher to support these systems (otherwise, what's the point of these tests?). I think maintaining the "standard" system config is already enough overhead. How much people on ubuntu add custom package repos?

Maybe it would be worthwhile to have a general discussion about which systems sage wants to support (how old, which system configs, when to drop support etc) and then based on this outcome add tests. (Please excuse my ignorance if there have been already such a discussion.)

Also this ticket seems to be the exact opposite of #32532 and as far as I followed the discussion on the dev list no final conclusion has been reached there yet.

dimpase · 2022-02-07T13:10:56Z

comment:9

The standard ones on these museum-grade systems are too old to be supported. IMHO we must be much more aggressive in dropping old systems than we currently are...

mkoeppe · 2022-02-07T16:23:26Z

comment:10

Replying to @tobiasdiez:

This would also mean that we vouch to support these systems (otherwise, what's the point of these tests?).

No, this ticket (and all of tox.ini) does not come with a promise that we support a particular system configuration.

It only makes it easy to test it.

mkoeppe · 2022-02-07T16:25:58Z

comment:11

Replying to @dimpase:

The standard ones on these museum-grade systems are too old to be supported.

Exactly, in particular because of upcoming updates of several of our standard packages, see #32074.

tobiasdiez · 2022-02-07T17:55:21Z

comment:12

Replying to @mkoeppe:

Replying to @tobiasdiez:

This would also mean that we vouch to support these systems (otherwise, what's the point of these tests?).

No, this ticket (and all of tox.ini) does not come with a promise that we support a particular system configuration.

It only makes it easy to test it.

But why do you want to test something (and spend resources on implementing and maintaining such tests) if there is no intention to support this particular config?

mkoeppe · 2022-02-07T17:59:59Z

comment:13

Once we can easily test if something works, a decision can be made whether to support it.

mkoeppe · 2022-02-07T18:02:45Z

comment:14

Also, this infrastructure is now used by several upstream projects for portability testing; most recent add: pypa/setuptools#2923

tobiasdiez · 2022-02-07T21:58:20Z

comment:15

Replying to @mkoeppe:

Once we can easily test if something works, a decision can be made whether to support it.

I think you have proven very successfully that the existing infrastructure can easily be extended to incorporate new system configs. Thus, the important question is if we want to support a certain config, and not if its technically possible to test it. I don't have enough experience with the linux environment to judge if custom gcc installations are common enough on recent systems. However, a common theme in the discussion about removing the gcc package was that's a pain in the * * to maintain different configs. So I think there should be good reasons to add another config to the already pretty long list of variations.

mkoeppe · 2022-02-07T22:12:44Z

comment:16

Let me rephrase for clarity:

Once we can easily test if something works on a platform, a decision can be made whether to support SageMath on this platform.

dimpase · 2022-02-07T22:20:00Z

comment:17

As we saw recently a few times, such obsolete systems, where one needs such non-standard toolchains to get anywhere, are well-supported by conda.

Direct support of such configurations is not buying us much.

mkoeppe · 2022-02-07T22:20:55Z

comment:18

Replying to @tobiasdiez:

Replying to @mkoeppe:
a common theme in the discussion about removing the gcc package was that's a pain in the * * to maintain different configs. So I think there should be good reasons to add another config to the already pretty long list of variations.

No, it sounds like you have misunderstood the discussion. We want people to be able to use GCC (and other toolchain components) from their distribution -- to reduce our maintenance burden for our project regarding the gcc spkg.

tobiasdiez · 2022-02-07T22:34:37Z

comment:19

I think a long-term idea of what systems to support and in which way would be preferably. For example, numpy has a clear strategy where they recommend people to install it via conda and are relatively restrictive on which compilers they support for from-source installations. Not saying that sage should go the same path, but I think narrowing down the range of supported distribution channels would open up more resources for actual features and bug fixes.

Anyway, should we transfer this discussion to the dev mailing list?

mkoeppe · 2022-02-07T22:38:35Z

comment:20

I have made a plan for platform support in Sage 9.6, 9.7 in #32074 because major changes are coming from upstream.

mkoeppe · 2022-02-07T22:40:26Z

comment:21

The present ticket does not need more discussion. It's an improvement because it expands the platforms that we can easily test.

Being able to test is better than not being able to test.

dimpase · 2022-02-09T23:44:21Z

comment:22

Sending build context to Docker daemon  519.8MB
Step 1/46 : ARG BASE_IMAGE=ubuntu:latest
Step 2/46 : FROM ${BASE_IMAGE} as with-system-packages
bionic: Pulling from library/ubuntu
68e7bb398b9f: Pull complete 
Digest: sha256:c2aa13782650aa7ade424b12008128b60034c795f25456e8eb552d0a0f447cad
Status: Downloaded newer image for ubuntu:bionic
 ---> dcf4d4bef137
Step 3/46 : RUN DEBIAN_FRONTEND=noninteractive apt-get install -qqq --no-install-recommends --yes software-properties-common
 ---> Running in 600f78310b95
E: Unable to locate package software-properties-common
The command '/bin/sh -c DEBIAN_FRONTEND=noninteractive apt-get install -qqq --no-install-recommends --yes software-properties-common' returned a non-zero code: 100
sha256:18cedbf3aa838b99f9bfd9684434a6f4ed1de5d8be02a579bcb51212082c839f
ERROR: InvocationError for command /bin/bash -c 'for docker_target in with-targets; do  BUILD_TAG=sage-docker-ubuntu-bionic-standard-toolchain-gcc_9-$docker_target:$(git describe --dirty --always);  DOCKER_BUILDKIT=0  docker build . -f /Users/dima/sage/.tox/docker-ubuntu-bionic-standard-toolchain-gcc_9/Dockerfile  --target $docker_target  --tag $BUILD_TAG  --build-arg EXTRA_CONFIGURE_ARGS="--enable-experimental-packages --enable-download-from-upstream-url  --with-system-python3=yes --with-system-gcc=force CC=gcc-9 CXX=g++-9 FC=gfortran-9 "  --build-arg BASE_IMAGE=ubuntu:bionic  --build-arg BOOTSTRAP="./bootstrap"  --build-arg TARGETS_PRE="$(if test -n "$TARGETS_PRE"; then echo $TARGETS_PRE; else echo all-sage-local; fi)"  --build-arg TARGETS="build"  --build-arg TARGETS_OPTIONAL="ptest"  ; status=$?;  if [ $status != 0 ]; then BUILD_TAG="$BUILD_TAG-failed"; docker commit $(docker ps -l -q) $BUILD_TAG; fi;  echo $BUILD_TAG >> /Users/dima/sage/.tox/docker-ubuntu-bionic-standard-toolchain-gcc_9/Dockertags;  if [ x"" != x ]; then echo Pushing $BUILD_TAG; docker push $BUILD_TAG || echo "(ignoring errors)"; fi;  if [ $status != 0 ]; then exit $status; fi; done' (exited with code 100)
_______________________________________________________ summary ________________________________________________________
ERROR:   docker-ubuntu-bionic-standard-toolchain-gcc_9: commands failed

dimpase · 2022-02-09T23:45:34Z

comment:23

Step 1/46 : ARG BASE_IMAGE=ubuntu:latest
Step 2/46 : FROM ${BASE_IMAGE} as with-system-packages
 ---> dcf4d4bef137
Step 3/46 : RUN DEBIAN_FRONTEND=noninteractive apt-get install -qqq --no-install-recommends --yes software-properties-common
 ---> Using cache
 ---> 18cedbf3aa83
Step 4/46 : RUN add-apt-repository ppa:ubuntu-toolchain-r/test
 ---> Running in 8f18951793c2
/bin/sh: 1: add-apt-repository: not found
The command '/bin/sh -c add-apt-repository ppa:ubuntu-toolchain-r/test' returned a non-zero code: 127
sha256:96636bafcff5cb8b60716791c2613a2d9a01ad33e8bf5b825d0f0fd32caaaeb8
ERROR: InvocationError for command /bin/bash -c 'for docker_target in with-targets; do  BUILD_TAG=sage-docker-ubuntu-bionic-standard-toolchain-gcc_10-$docker_target:$(git describe --dirty --always);  DOCKER_BUILDKIT=0  docker build . -f /Users/dima/sage/.tox/docker-ubuntu-bionic-standard-toolchain-gcc_10/Dockerfile  --target $docker_target  --tag $BUILD_TAG  --build-arg EXTRA_CONFIGURE_ARGS="--enable-experimental-packages --enable-download-from-upstream-url  --with-system-python3=yes --with-system-gcc=force CC=gcc-10 CXX=g++-10 FC=gfortran-10 "  --build-arg BASE_IMAGE=ubuntu:bionic  --build-arg BOOTSTRAP="./bootstrap"  --build-arg TARGETS_PRE="$(if test -n "$TARGETS_PRE"; then echo $TARGETS_PRE; else echo all-sage-local; fi)"  --build-arg TARGETS="build"  --build-arg TARGETS_OPTIONAL="ptest"  ; status=$?;  if [ $status != 0 ]; then BUILD_TAG="$BUILD_TAG-failed"; docker commit $(docker ps -l -q) $BUILD_TAG; fi;  echo $BUILD_TAG >> /Users/dima/sage/.tox/docker-ubuntu-bionic-standard-toolchain-gcc_10/Dockertags;  if [ x"" != x ]; then echo Pushing $BUILD_TAG; docker push $BUILD_TAG || echo "(ignoring errors)"; fi;  if [ $status != 0 ]; then exit $status; fi; done' (exited with code 127)
_______________________________________________________ summary ________________________________________________________
ERROR:   docker-ubuntu-bionic-standard-toolchain-gcc_10: commands failed

dimpase · 2022-02-09T23:48:28Z

comment:24

although docker-ubuntu-bionic-standard-toolchain-gcc_11 fares better so far.

sagetrac-git · 2022-02-12T22:22:53Z

Changed commit from 983b4a8 to a78c109

sagetrac-git · 2022-02-12T22:22:53Z

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

`3605076`	`build/pkgs/sagelib/spkg-install: Fix up deleting temp.*`
`d87027c`	`sage --package list: Add option --exclude`
`c02c03c`	`tox.ini (gitpod-standard): Exclude some packages`
`6285cd9`	`.gitpod.Dockerfile: Save space by removing apt lists`
`7e74607`	`.gitpod.Dockerfile: Remove built Sage documentation`
`f71c8b6`	`docker/.gitpod.Dockerfile: Clean more`
`f953228`	`Merge tag '9.5.rc0' into t/33103/gitpod_integration_using_docker_images_from_portability_testing_workflow`
`2d0902a`	`src/doc/en/developer/workspace.rst: Fix markup`
`078f247`	`.github/workflows/tox*.yml: Transform GITHUB_REF_NAME to a valid docker tag`
`a78c109`	`Merge #33103`

mkoeppe · 2022-02-12T22:38:02Z

comment:57

Just noticed that now tox -e docker-ubuntu-xenial-standard-toolchain-gcc_9 -- config.status gives

E: Package 'gpg' has no installation candidate
E: Unable to locate package gpg-agent
The command '/bin/sh -c apt-get update &&  DEBIAN_FRONTEND=noninteractive apt-get install -qqq --no-install-recommends --yes software-properties-common gpg gpg-agent' returned a non-zero code: 100

sagetrac-git · 2022-02-12T22:42:00Z

Changed commit from a78c109 to f98e0dc

sagetrac-git · 2022-02-12T22:42:00Z

Branch pushed to git repo; I updated commit sha1. New commits:

`f98e0dc`	`build/bin/write-dockerfile.sh (debian) [EXTRA_REPOSITORY]: Do not fail if gpg, gpg-agent cannot be installed`

dimpase · 2022-02-13T10:38:36Z

comment:60

Replying to @sagetrac-git:

Branch pushed to git repo; I updated commit sha1. New commits:

`4e2d536`	`build/bin/write-dockerfile.sh (ubuntu): Run apt-get update before installing software-properties-common`

you forgot this one, and without it one gets an error early on.

dimpase · 2022-02-13T13:08:12Z

comment:61

Replying to @dimpase:

Replying to @sagetrac-git:

Branch pushed to git repo; I updated commit sha1. New commits:

4e2d536 build/bin/write-dockerfile.sh (ubuntu): Run apt-get update before installing software-properties-common

you forgot this one, and without it one gets an error early on.

oops, sorry, I messed up my branches. It's good here. Still testing.

dimpase · 2022-02-13T20:21:50Z

comment:62

Please add wiping up /etc/dpkg/dpkg.cfg.d/excludes in handling of Ubuntu images, otherwise apt will drop stuff there, see #32789 comment:61 (and https://bugs.launchpad.net/ubuntu/+source/dpkg/+bug/1960749 mentioned there)

mkoeppe · 2022-02-13T20:48:18Z

comment:63

I've pushed this change to the branch of #32789

dimpase · 2022-02-14T14:45:47Z

comment:64

OK, lgtm

mkoeppe · 2022-02-14T16:44:42Z

comment:65

Thanks!

dimpase · 2022-02-16T12:53:44Z

comment:66

needs a rebase

sagetrac-git · 2022-02-16T16:30:05Z

Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. New commits:

`d42bc03`	`Merge tag '9.6.beta1' into t/32966/tox___gh_actions__add_ubuntu_variants_using_ppa_ubuntu_toolchain_r`

sagetrac-git · 2022-02-16T16:30:05Z

Changed commit from f98e0dc to d42bc03

vbraun · 2022-02-20T13:28:04Z

Changed branch from u/mkoeppe/tox___gh_actions__add_ubuntu_variants_using_ppa_ubuntu_toolchain_r to d42bc03

mkoeppe added this to the sage-9.5 milestone Dec 4, 2021

mkoeppe added c: porting labels Dec 4, 2021

mkoeppe modified the milestones: sage-9.5, sage-9.6 Dec 18, 2021

This comment has been minimized.

Sign in to view

mkoeppe added the s: needs review label Dec 28, 2021

This comment has been minimized.

Sign in to view

mkoeppe added s: needs review and removed s: needs work labels Feb 12, 2022

mkoeppe added s: needs work and removed s: needs review labels Feb 12, 2022

mkoeppe added s: needs review and removed s: needs work labels Feb 12, 2022

dimpase added s: positive review and removed s: needs review labels Feb 14, 2022

sagetrac-git mannequin added s: needs review and removed s: positive review labels Feb 16, 2022

mkoeppe added s: positive review and removed s: needs review labels Feb 16, 2022

vbraun removed the s: positive review label Feb 20, 2022

vbraun closed this as completed in 03d78e2 Feb 20, 2022

dimpase mentioned this issue Jan 6, 2023

Meta-ticket: Remove support for outdated distros #32074

Open

tox / GH Actions: Add ubuntu variants using ppa:ubuntu-toolchain-r #32966

tox / GH Actions: Add ubuntu variants using ppa:ubuntu-toolchain-r #32966

Comments

mkoeppe commented Dec 4, 2021

This comment has been minimized.

mkoeppe commented Dec 25, 2021

sagetrac-git mannequin commented Dec 25, 2021

sagetrac-git mannequin commented Dec 25, 2021

mkoeppe commented Dec 28, 2021

This comment has been minimized.

tobiasdiez commented Feb 7, 2022

dimpase commented Feb 7, 2022

mkoeppe commented Feb 7, 2022

mkoeppe commented Feb 7, 2022

tobiasdiez commented Feb 7, 2022

mkoeppe commented Feb 7, 2022

mkoeppe commented Feb 7, 2022

tobiasdiez commented Feb 7, 2022

mkoeppe commented Feb 7, 2022

dimpase commented Feb 7, 2022

mkoeppe commented Feb 7, 2022

tobiasdiez commented Feb 7, 2022

mkoeppe commented Feb 7, 2022

mkoeppe commented Feb 7, 2022

dimpase commented Feb 9, 2022

dimpase commented Feb 9, 2022

dimpase commented Feb 9, 2022

sagetrac-git mannequin commented Feb 12, 2022

sagetrac-git mannequin commented Feb 12, 2022

mkoeppe commented Feb 12, 2022

sagetrac-git mannequin commented Feb 12, 2022

sagetrac-git mannequin commented Feb 12, 2022

dimpase commented Feb 13, 2022

dimpase commented Feb 13, 2022

dimpase commented Feb 13, 2022

mkoeppe commented Feb 13, 2022

dimpase commented Feb 14, 2022

mkoeppe commented Feb 14, 2022

dimpase commented Feb 16, 2022

sagetrac-git mannequin commented Feb 16, 2022

sagetrac-git mannequin commented Feb 16, 2022

vbraun commented Feb 20, 2022