gitpod integration using Docker images from portability testing workflow #33103

mkoeppe · 2022-01-01T19:34:16Z

Gitpod allows one to setup a complete dev environment in the cloud. It is free to use for up to 50 hours per month. In this ticket the config necessary for making this work with sagemath is added.

You can try this out by going to https://gitpod.io/#https://github.com/mkoeppe/sage/tree/mkoeppe-9.5.beta9%2Bupdates.

It works by building docker images on github, push them to GH packages (https://github.com/mkoeppe?tab=packages&q=gitpod) and then use these ones as the base for the gitpod workspace.

After this ticket is merged, we should activate the automatic prebuild by installing the github app (https://www.gitpod.io/docs/prebuilds#on-github). With this enabled, every push would trigger a prebuild of the complete environment (including the build of all dependencies and cythonizion) so that one has a up-to-date code env very quickly without the need to build anything.

Follow-up: #33113 gitpod usability improvements

Alternative: #32749, which only builds the docker image on gitpod (which is currently not really feasible as they have a time limit of 1h)

Depends on #30933
Depends on #29285

CC: @tobiasdiez @dimpase @saraedum

Component: docker

Author: Tobias Diez, Matthias Koeppe

Branch/Commit: 078f247

Reviewer: Matthias Koeppe, Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/33103

The text was updated successfully, but these errors were encountered:

mkoeppe · 2022-01-01T19:36:53Z

Branch: u/mkoeppe/gitpod_integration_using_docker_images_from_portability_testing_workflow

mkoeppe · 2022-01-01T19:41:25Z

Last 10 new commits:

`e99a09e`	`sed -i.bak 's/ubunty/ubuntu/g' .github/workflows/*.yml`
`edb4364`	`.github/workflows/tox.yml: Replace homebrew-macos-python3_xcode-standard by homebrew-macos-usrlocal-python3_xcode-standard`
`e27e161`	`Merge #32703`
`96a45d1`	`tox.ini, .github/workflows: Remove ubuntu-groovy`
`59ec3d0`	`Merge #33068`
`cbf03d1`	`.github/workflows/tox.yml: Add docker-gitpod`
`4a9edfc`	`build/bin/write-dockerfile.sh: Handle __SUDO`
`d697d94`	`build/bin/write-dockerfile.sh: Use relative workdir 'sage'`
`c7cc5cd`	`.gitpod.yml, docker/.gitpod.Dockerfile: Reverse SAGE_LOCAL symlink`
`bde9f74`	`build/bin/write-dockerfile.sh: Handle $__CHOWN`

mkoeppe · 2022-01-01T19:41:25Z

Commit: bde9f74

mkoeppe · 2022-01-01T19:41:25Z

Author: Tobias Diez, Matthias Koeppe

sagetrac-git · 2022-01-01T19:44:05Z

Branch pushed to git repo; I updated commit sha1. New commits:

`21afe51`	`.github/workflows/tox*.yml, tox.ini: Pass ddocker.pkg.github.com credentials via DOCKER_CONFIG_FILE`
`76b5e76`	`Merge #30933`

sagetrac-git · 2022-01-01T19:44:05Z

Changed commit from bde9f74 to 76b5e76

sagetrac-git · 2022-01-01T20:27:54Z

Branch pushed to git repo; I updated commit sha1. New commits:

`7ef6a6e`	`docker/.gitpod.Dockerfile: Remove, use custom image from GH Packages instead`

sagetrac-git · 2022-01-01T20:27:54Z

Changed commit from 76b5e76 to 7ef6a6e

sagetrac-git · 2022-01-01T20:59:12Z

Branch pushed to git repo; I updated commit sha1. New commits:

`6b41c4a`	`.github/workflows/tox.yml: Also push ...-with-targets-pre`

sagetrac-git · 2022-01-01T20:59:12Z

Changed commit from 7ef6a6e to 6b41c4a

mkoeppe · 2022-01-01T21:35:52Z

comment:9

Running into pyenv issues (#29285) here too:

[pytest] installing. Log file: /home/gitpod/sage/logs/pkgs/pytest.log
  [pytest] error installing, exit status 1. End of log file:
  [pytest]   /home/gitpod/sage-local/var/lib/sage/venv-python3.8/bin/python3: No module named pip
  [pytest] Full log file: /home/gitpod/sage/logs/pkgs/pytest.log
make[4]: *** [Makefile:2756: pytest-no-deps] Error 1

mkoeppe · 2022-01-02T02:01:02Z

Changed dependencies from #33068, #30933 to #33068, #30933, #29285

sagetrac-git · 2022-01-02T02:01:43Z

Changed commit from 6b41c4a to 5f0c36c

sagetrac-git · 2022-01-02T02:01:43Z

Branch pushed to git repo; I updated commit sha1. New commits:

`376932f`	`build/bin/sage-bootstrap-python: Reject pyenv shims`
`5f0c36c`	`Merge #29285`

mkoeppe · 2022-01-02T02:05:38Z

Changed dependencies from #33068, #30933, #29285 to #33068, #30933, #29285, #32868

mkoeppe · 2022-01-02T02:08:41Z

Changed dependencies from #33068, #30933, #29285, #32868 to #33068, #30933, #29285, #32868, #33067

sagetrac-git · 2022-01-02T02:09:46Z

Changed commit from 5f0c36c to af7c8d3

sagetrac-git · 2022-01-02T02:09:46Z

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

`aeca1a0`	`Makefile (pypi-sdists): Add sage_setup`
`cd3dce9`	`pkgs/sage-conf_pypi/setup.py: Update directory of configured sage_conf.py`
`c695921`	`pkgs/sage-conf/sage_conf.py.in (SAGE_SPKG_WHEELS): Update so it works if SAGE_VENV != SAGE_LOCAL`
`cebcda3`	`Merge tag '9.5.beta4' into t/29039/pip_installable_package_sage_bootstrap`
`0026892`	`Merge tag '9.5.beta5' into t/29039/pip_installable_package_sage_bootstrap`
`2a4a323`	`Merge tag '9.5.beta7' into t/29039/pip_installable_package_sage_bootstrap`
`02234ce`	`Merge #29039`
`b1413d4`	`build/bin/write-dockerfile.sh: Move 'ADD pkgs' earlier`
`859be3e`	`build/bin/write-dockerfile.sh: ADD src/*.m4 files for bootstrapping`
`af7c8d3`	`Merge #33067`

sagetrac-git · 2022-01-02T02:26:30Z

Changed commit from af7c8d3 to fad8942

sagetrac-git · 2022-01-02T02:26:30Z

Branch pushed to git repo; I updated commit sha1. New commits:

`fad8942`	`build/bin/write-dockerfile.sh: Use relative workdir 'sage' (again)`

sagetrac-git · 2022-01-02T05:04:34Z

Branch pushed to git repo; I updated commit sha1. New commits:

`601a3b5`	`.gitpod.yml: Remove pyenv workarounds`
`1258c70`	`.gitpod.yml: configure --enable-download-from-upstream-url`

sagetrac-git · 2022-01-12T07:06:38Z

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

`078f247`	`.github/workflows/tox*.yml: Transform GITHUB_REF_NAME to a valid docker tag`

sagetrac-git · 2022-01-12T07:06:38Z

Changed commit from 08048c3 to 078f247

dimpase · 2022-01-13T13:21:31Z

comment:138

Is it easy to configure launching this (with prebuids) on branches in https://github.com/sagemath/sagetrac-mirror ?

tobiasdiez · 2022-01-13T14:31:28Z

comment:139

Yes, it's only a few clicks see: https://www.gitpod.io/docs/prebuilds#projects-and-prebuilds
However, it needs to be done by someone with rights to install apps in the sagemath org.

dimpase · 2022-01-13T17:40:07Z

comment:140

Replying to @tobiasdiez:

Yes, it's only a few clicks see: https://www.gitpod.io/docs/prebuilds#projects-and-prebuilds
However, it needs to be done by someone with rights to install apps in the sagemath org.

I've sent you an invite. I guess I installed their "app", but I don't see how to proceed.
(I think it should be enabled on the branch of this ticket only, before it's merged.)

Did you see my email message about trac access?

tobiasdiez · 2022-01-14T11:59:49Z

comment:141

Thanks! I've now created a "team" sagemath and added the sage and sagetrac-mirror as projects. If you have a gitpod account and want to be invited to this team (in order to access the config), just post your email address and I send an invite.

Currently, the creation of the image fails however due to a permission error:

 > [internal] load metadata for ghcr.io/sagemath/sage/sage-docker-gitpod-standard-with-targets:dev:
------
.gitpod.Dockerfile:3
--------------------
   1 |     ARG BASE_GITHUB_REPOSITORY=sagemath/sage
   2 |     ARG BASE_TAG=dev
   3 | >>> FROM ghcr.io/${BASE_GITHUB_REPOSITORY}/sage-docker-gitpod-standard-with-targets:${BASE_TAG} as with-targets
   4 |     RUN sudo rm -rf /var/cache/debconf/* /var/lib/apt/lists/* /tmp/* /var/tmp/*
   5 |     # Fast doc rebuilds do not work because
--------------------
error: failed to solve: failed to fetch anonymous token: unexpected status: 403 Forbidden

Matthias, can you have a look please?

https://gitpod.io/#prebuild/https://github.com/sagemath/sagetrac-mirror/tree/u/mkoeppe/gitpod_integration_using_docker_images_from_portability_testing_workflow

dimpase · 2022-01-14T12:17:24Z

comment:142

Replying to @tobiasdiez:

Thanks! I've now created a "team" sagemath and added the sage and sagetrac-mirror as projects. If you have a gitpod account and want to be invited to this team (in order to access the config), just post your email address and I send an invite.

my email is shown on my GitHub account https://github.com/dimpase

Thanks!

mkoeppe · 2022-01-14T16:20:50Z

comment:143

Replying to @tobiasdiez:

Currently, the creation of the image fails however due to a permission error:

 > [internal] load metadata for ghcr.io/sagemath/sage/sage-docker-gitpod-standard-with-targets:dev:
------
.gitpod.Dockerfile:3
--------------------
   1 |     ARG BASE_GITHUB_REPOSITORY=sagemath/sage
   2 |     ARG BASE_TAG=dev
   3 | >>> FROM ghcr.io/${BASE_GITHUB_REPOSITORY}/sage-docker-gitpod-standard-with-targets:${BASE_TAG} as with-targets
   4 |     RUN sudo rm -rf /var/cache/debconf/* /var/lib/apt/lists/* /tmp/* /var/tmp/*
   5 |     # Fast doc rebuilds do not work because
--------------------
error: failed to solve: failed to fetch anonymous token: unexpected status: 403 Forbidden

I think you need to deposit a secret providing read access similar to what is explained in
https://www.gitpod.io/docs/self-hosted/latest/configuration/docker-registry

mkoeppe · 2022-01-14T18:51:48Z

comment:144

... alternatively I suppose we could push the required Docker images to DockerHub -- but I don't have write access there; @saraedum, could you deposit an access token for that as a secret here? https://github.com/organizations/sagemath/settings/secrets/actions

tobiasdiez · 2022-01-14T20:09:07Z

comment:146

Replying to @mkoeppe:

Replying to @tobiasdiez:
Currently, the creation of the image fails however due to a permission error:
 > [internal] load metadata for ghcr.io/sagemath/sage/sage-docker-gitpod-standard-with-targets:dev:
------
.gitpod.Dockerfile:3
--------------------
   1 |     ARG BASE_GITHUB_REPOSITORY=sagemath/sage
   2 |     ARG BASE_TAG=dev
   3 | >>> FROM ghcr.io/${BASE_GITHUB_REPOSITORY}/sage-docker-gitpod-standard-with-targets:${BASE_TAG} as with-targets
   4 |     RUN sudo rm -rf /var/cache/debconf/* /var/lib/apt/lists/* /tmp/* /var/tmp/*
   5 |     # Fast doc rebuilds do not work because
--------------------
error: failed to solve: failed to fetch anonymous token: unexpected status: 403 Forbidden
I think you need to deposit a secret providing read access similar to what is explained in
https://www.gitpod.io/docs/self-hosted/latest/configuration/docker-registry

That link is for a self-hosted gitpod and thus does not apply to our situation (we don't have a gitpod config other than what is committed in this branch).

I think the issue is that the image sage-docker-gitpod-standard-with-targets is not yet published under sagemath/sage. But how is this then working with your account Matthias?

mkoeppe · 2022-01-14T20:12:45Z

comment:147

Replying to @tobiasdiez:

the image sage-docker-gitpod-standard-with-targets is not yet published under sagemath/sage. But how is this then working with your account Matthias?

Because on my testing branch used for the prebuild, I am setting the correct repo:
https://github.com/mkoeppe/sage/blob/mkoeppe-9.5.beta9%2Bupdates/docker/.gitpod.Dockerfile

mkoeppe · 2022-01-14T20:14:11Z

comment:148

(And because every GH workflow automatically has the credentials to access its own GH packages (gchr.io)!)

tobiasdiez · 2022-01-14T20:15:32Z

comment:149

Ahh okay, I thought this is an environment variable that is picked up somewhere.

So in this case we just have to wait until this ticket is merged, which would then build and publish the docker image to sagemath/sage, right?

mkoeppe · 2022-01-14T20:20:46Z

comment:150

Replying to @tobiasdiez:

wait until this ticket is merged, which would then build and publish the docker image to sagemath/sage, right?

Yes, on the next release that includes this branch, the Docker image will be pushed (by separate instances of the workflow) both to gchr.io/sagemath/sage and to gchr.io/sagemath/sagetrac-mirror.

We will then see whether the 403 Forbidden will go away. I think it will not, for workflows run on github.com/sagemath/sagetrac-mirror, because it may not have the read:packages credential for gchr.io/sagemath/sage.

mkoeppe · 2022-01-16T21:38:47Z

comment:151

I think we may need to do this:
https://docs.github.com/en/packages/learn-github-packages/configuring-a-packages-access-control-and-visibility#configuring-visibility-of-container-images-for-an-organization

mkoeppe · 2022-01-16T21:44:53Z

comment:152

Well, no, all packages are already set to "public": For example https://github.com/orgs/sagemath/packages/container/sage%2Fsage-docker-ubuntu-focal-standard-configured/settings

mkoeppe · 2022-01-16T21:46:49Z

comment:153

But perhaps this is needed: https://docs.github.com/en/packages/learn-github-packages/configuring-a-packages-access-control-and-visibility#github-actions-access-for-organization-owned-container-images

mkoeppe · 2022-01-16T21:53:49Z

comment:154

No, this makes no sense.

vbraun · 2022-02-12T22:04:53Z

Changed branch from u/mkoeppe/gitpod_integration_using_docker_images_from_portability_testing_workflow to 078f247

mkoeppe added this to the sage-9.5 milestone Jan 1, 2022

mkoeppe added c: docker labels Jan 1, 2022

This comment has been minimized.

Sign in to view

sagetrac-git mannequin added the s: needs review label Jan 12, 2022

mkoeppe added s: positive review and removed s: needs review labels Jan 12, 2022

mkoeppe modified the milestones: sage-9.5, sage-9.6 Jan 28, 2022

mkoeppe added p: critical / 2 and removed p: major / 3 labels Feb 11, 2022

vbraun removed the s: positive review label Feb 12, 2022

vbraun closed this as completed in 88a98d5 Feb 12, 2022

dimpase mentioned this issue Jan 18, 2022

Unbreak the build when pyenv is present #29285

Closed

tobiasdiez mentioned this issue Feb 16, 2022

Add github action running on each push #33263

Closed

gitpod integration using Docker images from portability testing workflow #33103

gitpod integration using Docker images from portability testing workflow #33103

Comments

mkoeppe commented Jan 1, 2022

mkoeppe commented Jan 1, 2022

mkoeppe commented Jan 1, 2022

mkoeppe commented Jan 1, 2022

mkoeppe commented Jan 1, 2022

sagetrac-git mannequin commented Jan 1, 2022

sagetrac-git mannequin commented Jan 1, 2022

This comment has been minimized.

sagetrac-git mannequin commented Jan 1, 2022

sagetrac-git mannequin commented Jan 1, 2022

This comment has been minimized.

This comment has been minimized.

sagetrac-git mannequin commented Jan 1, 2022

sagetrac-git mannequin commented Jan 1, 2022

mkoeppe commented Jan 1, 2022

mkoeppe commented Jan 2, 2022

sagetrac-git mannequin commented Jan 2, 2022

sagetrac-git mannequin commented Jan 2, 2022

mkoeppe commented Jan 2, 2022

mkoeppe commented Jan 2, 2022

sagetrac-git mannequin commented Jan 2, 2022

sagetrac-git mannequin commented Jan 2, 2022

This comment has been minimized.

sagetrac-git mannequin commented Jan 2, 2022

sagetrac-git mannequin commented Jan 2, 2022

This comment has been minimized.

sagetrac-git mannequin commented Jan 2, 2022

sagetrac-git mannequin commented Jan 12, 2022

sagetrac-git mannequin commented Jan 12, 2022

dimpase commented Jan 13, 2022

tobiasdiez commented Jan 13, 2022

dimpase commented Jan 13, 2022

tobiasdiez commented Jan 14, 2022

dimpase commented Jan 14, 2022

mkoeppe commented Jan 14, 2022

mkoeppe commented Jan 14, 2022

tobiasdiez commented Jan 14, 2022

mkoeppe commented Jan 14, 2022

mkoeppe commented Jan 14, 2022

tobiasdiez commented Jan 14, 2022

mkoeppe commented Jan 14, 2022

mkoeppe commented Jan 16, 2022

mkoeppe commented Jan 16, 2022

mkoeppe commented Jan 16, 2022

mkoeppe commented Jan 16, 2022

vbraun commented Feb 12, 2022