Fixes for the conda-for-Sage-developers installation method, add GH Actions workflow #30845

mkoeppe · 2020-11-01T18:57:38Z

Add github action workflow that checks the build of sage in a conda environment, completely bypassing the installation of any sage package. This is based on the steps outlined at https://wiki.sagemath.org/Conda.

This also tests that the generated src/environment*.yml files work correctly. (See documentation added in #28745)

Run: https://github.com/sagemath/sagetrac-mirror/actions/workflows/ci-conda.yml

Fixes:

Conda: Add primecountpy #33330: primecountpy is not installed in the conda env because its not listed in environment.yml
environment-optional: lcalc 1 is installed and then not recognized / leads to error

building 'sage.libs.lcalc.lcalc_Lfunction' extension
    cc1plus: warning: command line option '-Wstrict-prototypes' is valid for C/ObjC but not for C++
    In file included from sage/libs/lcalc/lcalc_Lfunction.cpp:740:
    sage/libs/lcalc/lcalc_sage.h:1:10: fatal error: lcalc/L.h: No such file or directory
        1 | #include "lcalc/L.h"
          |          ^~~~~~~~~~~
    compilation terminated.

environment-optional: probably related, pari 2.11.4 is installed by conda instead of the newest 2.13.2. This version is then rejected by configure as being too old.
Disable conda distro information for gdb because it is broken on macOS.
Add conda distro information for lrcalc_python
Filter out conda-specific ld warnings in doctests
In the feature test for sage_spkg, check whether any Sage packages are actually installed
Include an upper version bound for ptyprocess in sagemath-standard's install-requires

Follow-ups: See Meta-ticket #33331

Depends on #33358
Depends on #33330
Depends on #33361
Depends on #33141

CC: @isuruf @tobiasdiez @dimpase @saraedum

Component: build

Keywords: sd111

Author: Tobias Diez, Matthias Koeppe

Branch/Commit: 93fce5a

Reviewer: Matthias Koeppe, Dima Pasechnik, Tobias Diez

Issue created by migration from https://trac.sagemath.org/ticket/30845

The text was updated successfully, but these errors were encountered:

mkoeppe · 2020-12-06T18:17:38Z

comment:3

Hoping we can make progress on this ticket this week - https://wiki.sagemath.org/days111

mkoeppe · 2020-12-06T18:17:38Z

Changed keywords from none to sd111

tobiasdiez · 2020-12-22T10:44:04Z

comment:7

Something like https://github.com/tobiasdiez/sage/blob/public/build/conda_gh/.github/workflows/ci-conda.yml?

This currently fails with

configure: creating ./config.status
config.status: creating build/make/Makefile-auto
config.status: creating build/make/Makefile
config.status: creating src/bin/sage-env-config
config.status: creating src/bin/sage-src-env-config
config.status: creating build/bin/sage-build-env-config
config.status: creating build/pkgs/sage_conf/src/sage_conf.py
config.status: creating build/pkgs/sage_conf/src/setup.cfg
config.status: executing depfiles commands
config.status: executing mkdirs commands
config.status: creating directory /home/runner/work/sage/sage/logs/pkgs
config.status: creating directory 
mkdir: cannot create directory '': No such file or directory
config.status: error: could not create 
Error: Process completed with exit code 1.

https://github.com/tobiasdiez/sage/runs/1594237745?check_suite_focus=true
Any idea what could be the reason and how to fix it?

tobiasdiez · 2020-12-22T10:46:09Z

Commit: 9af4614

tobiasdiez · 2020-12-22T10:46:09Z

Author: Tobias Diez

tobiasdiez · 2020-12-22T10:46:09Z

New commits:

`c68d262`	`Add first version of workflow`
`9af4614`	`Try with sudo (why is this needed now?!)`

tobiasdiez · 2020-12-22T10:46:09Z

Branch: public/build/conda_gh

sagetrac-git · 2020-12-22T11:14:53Z

Changed commit from 9af4614 to a178b56

sagetrac-git · 2020-12-22T11:14:53Z

Branch pushed to git repo; I updated commit sha1. New commits:

`5ac74ee`	`Try with conda-incubator`
`a178b56`	`Run in the conda shell`

sagetrac-git · 2020-12-22T11:57:25Z

Branch pushed to git repo; I updated commit sha1. New commits:

`199886c`	`Install more conda dependencies`
`10de3f5`	`Fix conda env name`
`86411b8`	`Remove unnececesary install of conda dep`

sagetrac-git · 2020-12-22T11:57:25Z

Changed commit from a178b56 to 86411b8

tobiasdiez · 2020-12-22T11:58:14Z

comment:11

Managed to fix the above error, just to be stuck two lines further down:

configure: creating ./config.status
config.status: creating build/make/Makefile-auto
config.status: creating build/make/Makefile
config.status: creating src/bin/sage-env-config
config.status: creating src/bin/sage-src-env-config
config.status: creating build/bin/sage-build-env-config
config.status: creating build/pkgs/sage_conf/src/sage_conf.py
config.status: creating build/pkgs/sage_conf/src/setup.cfg
config.status: executing depfiles commands
config.status: executing broken-gcc commands
config.status: executing mkdirs commands
config.status: creating directory /home/runner/work/sage/sage/logs/pkgs
config.status: creating directory /usr/share/miniconda/envs/sage-build
config.status: creating directory /usr/share/miniconda/envs/sage-build/bin
config.status: creating directory /usr/share/miniconda/envs/sage-build/etc
config.status: creating directory /usr/share/miniconda/envs/sage-build/include
config.status: creating directory /usr/share/miniconda/envs/sage-build/lib
config.status: creating directory /usr/share/miniconda/envs/sage-build/lib/pkgconfig
config.status: creating directory /usr/share/miniconda/envs/sage-build/share
config.status: creating directory /usr/share/miniconda/envs/sage-build/var/lib/sage/installed
config.status: error: Cannot perform incremental update, run "make distclean && make"
config.status: /usr/share/miniconda/envs/sage-build/lib64 is not a symlink, see Trac #19782
Error: Process completed with exit code 1.

https://github.com/tobiasdiez/sage/runs/1594601374?check_suite_focus=true

tobiasdiez · 2020-12-22T12:22:06Z

comment:12

Found a hack that fixed it by removing the lib64 folder. It only contained libcc1.so (not sure where/how this is needed).

tobiasdiez · 2020-12-22T13:18:37Z

comment:13

Made a bit more progress, but it currently errors while compiling gcc. Moreover, there is a problem in the detection of the conda/system gcc because "sage-env-config" doesn't exist.
https://github.com/tobiasdiez/sage/runs/1594867631?check_suite_focus=true

sagetrac-git · 2020-12-22T13:37:20Z

Branch pushed to git repo; I updated commit sha1. New commits:

`53503dc`	`Try remove lib64 folder by hand`
`e530ac0`	`Add c++ compiler to conda`
`ab47f5e`	`Install directly on top of conda`
`741c55a`	`Make autogen work`

sagetrac-git · 2020-12-22T13:37:20Z

Changed commit from 86411b8 to 741c55a

mkoeppe · 2020-12-22T14:43:24Z

comment:15

The direct variant should use src/environment.yml - which installs the Python packages of Sage in addition to what is listed in environment.yml

mkoeppe · 2020-12-22T14:44:04Z

comment:16

The lib64 looks like a conda packaging bug to me - @isuruf, can you help?

mkoeppe · 2020-12-22T14:45:33Z

comment:17

environment.yml already includes compilers -- I don't think adding cxx-compiler should be necessary

sagetrac-git · 2020-12-22T14:47:01Z

Changed commit from 741c55a to 3aba048

sagetrac-git · 2020-12-22T14:47:01Z

Branch pushed to git repo; I updated commit sha1. New commits:

`1231a24`	`Add env logging and fix SAGE_SRC`
`be024f8`	`Fix build`
`8a46d04`	`Improve output of print_env`
`9802c56`	`Improve print statement`
`3aba048`	`Use correct conda env file`

mkoeppe · 2020-12-22T15:02:30Z

comment:19

Replying to @tobiasdiez:

Made a bit more progress, but it currently errors while compiling gcc. Moreover, there is a problem in the detection of the conda/system gcc because "sage-env-config" doesn't exist.
https://github.com/tobiasdiez/sage/runs/1594867631?check_suite_focus=true

This is now #31097.

mkoeppe · 2020-12-22T15:02:45Z

Changed dependencies from #28745 to #31097

tobiasdiez · 2022-03-18T15:46:11Z

comment:308

Replying to @mkoeppe:

continue-on-error makes no sense because the unconfigured source tree cannot be tested.

That largely depends on the nature of the error. If the error in the config step is critical, then the build step will fail and no tests are run. Moreover, there is no harm in executing also the other steps, the workflow will still be displayed as failed due to the error in the config job. You just get a bit more information.

mkoeppe · 2022-03-18T15:47:58Z

comment:309

Replying to @tobiasdiez:

Replying to @mkoeppe:

continue-on-error makes no sense because the unconfigured source tree cannot be tested.

That largely depends on the nature of the error.

No, it doesn't. On error, the generated files are not generated.

tobiasdiez · 2022-03-18T15:48:29Z

Changed reviewer from Matthias Koeppe, Dima Pasechnik, https://github.com/mkoeppe/sage/actions/runs/2001401113 to Matthias Koeppe, Dima Pasechnik, Tobias Diez

tobiasdiez · 2022-03-18T15:48:29Z

comment:310

Replying to @mkoeppe:

Replying to @mkoeppe:

continue-on-error makes no sense because the unconfigured source tree cannot be tested.

Other than that, it's in good shape now. Could someone review my changes please?

I'm fine with these changes (although they probably should have been extracted to their own tickets as they don't have to do anything with the github workflow; but lets not be overly pedantic).

tobiasdiez · 2022-03-18T16:10:58Z

comment:314

Replying to @mkoeppe:

Replying to @tobiasdiez:

Replying to @mkoeppe:

continue-on-error makes no sense because the unconfigured source tree cannot be tested.

That largely depends on the nature of the error.

No, it doesn't. On error, the generated files are not generated.

I tried it locally and the necessary files seem to be still present when with-system-SPKG=force triggers an error. Having a quick look at the code these errors are also deferred to the very end of the configure script.

mkoeppe · 2022-03-18T16:18:27Z

comment:316

Replying to @tobiasdiez:

I tried it locally and the necessary files seem to be still present when with-system-SPKG=force triggers an error. Having a quick look at the code these errors are also deferred to the very end of the configure script.

You are right.

mkoeppe · 2022-03-18T16:20:14Z

comment:317

Nevertheless, continue-on-error makes no sense as I have explained throughout the thread, comment:141 etc.

mkoeppe · 2022-03-18T16:33:33Z

comment:318

But it's fine for now, let's see that we can get it merged in 9.6.

mkoeppe · 2022-03-18T16:34:40Z

comment:319

Onward to documenting it, #33426?

mkoeppe · 2022-03-18T16:55:17Z

comment:321

Replying to @tobiasdiez:

they probably should have been extracted to their own tickets as they don't have to do anything with the github workflow

Yes, the fix to src/sage/features/pkg_systems.py could perhaps have been made a separate ticket and a dependency here. But it would have been difficult to test it separately because the present ticket is its only use case.

Sometimes the scope of a ticket needs to be adjusted during development. You already know that sometimes the scope turns out to be too big and it is necessary break out smaller, more manageable and more clearly defined tickets.

But sometimes the scope of a ticket also turns out to be too small and the ticket is not viable on its own. This is what happened on the present ticket. The workflow documented in the wiki https://wiki.sagemath.org/Conda has never actually worked. Creating a GH Actions workflow based on a semi-quasi-working variant of it, by itself was not convincing as a ticket (comment:121 - "what does it test"). With the broader scope of actually fixing this way of installing Sage and testing something that we can recommend to developers, the ticket has a viable scope: The changes bring a clear improvement to Sage.

mkoeppe · 2022-03-18T21:29:56Z

comment:322

Some of the test suite failures that we saw are from ipython 8.x; #33170 updates the doctests

tobiasdiez · 2022-03-18T22:12:16Z

comment:323

Replying to @mkoeppe:

But it's fine for now, let's see that we can get it merged in 9.6.

Thanks for the review!

tobiasdiez · 2022-03-18T22:12:34Z

comment:324

Replying to @mkoeppe:

Onward to documenting it, #33426?

Yes!

tobiasdiez · 2022-03-18T22:27:45Z

comment:325

Replying to @mkoeppe:

Replying to @tobiasdiez:

they probably should have been extracted to their own tickets as they don't have to do anything with the github workflow

Yes, the fix to src/sage/features/pkg_systems.py could perhaps have been made a separate ticket and a dependency here. But it would have been difficult to test it separately because the present ticket is its only use case.

Sometimes the scope of a ticket needs to be adjusted during development. You already know that sometimes the scope turns out to be too big and it is necessary break out smaller, more manageable and more clearly defined tickets.

But sometimes the scope of a ticket also turns out to be too small and the ticket is not viable on its own. This is what happened on the present ticket. The workflow documented in the wiki https://wiki.sagemath.org/Conda has never actually worked. Creating a GH Actions workflow based on a semi-quasi-working variant of it, by itself was not convincing as a ticket (comment:121 - "what does it test"). With the broader scope of actually fixing this way of installing Sage and testing something that we can recommend to developers, the ticket has a viable scope: The changes bring a clear improvement to Sage.

In general I agree that a ticket has to have a clear scope and improvement. For this ticket here, the github workflow essentially didn't change in the past 4 weeks, and at this point it was already clear that the workflow works in principle, with the caveat that a lot of docstests failed. In my opinion, it would have been better to merge the ticket already (with the few additional improvements to the workflow that we have discussed), then follow-up tickets could have easily improved different aspects of the conda environment, while the previous runs of the ci would have served as a baseline. Even thinks like the addition of the tests for macos could have been small follow-up tickets. I really think the tendency of the sage developers to create a lot of parallel, inter-dependent tickets makes reviewing harder and slows down the process. A more linear approach would increase the dev experience and velocity in my opinion.

Anyway, I'm happy that this ticket is finally ready to be merged.

mkoeppe · 2022-03-18T22:53:14Z

comment:326

Replying to @tobiasdiez:

it was already clear that the workflow works in principle

The project generally uses a standard for tickets higher than that.
Yes, there's a tradeoff between velocity and stability.

isuruf · 2022-03-21T07:28:40Z

comment:327

Is there a ticket for the remaining failures after this ticket?

tobiasdiez · 2022-03-21T10:09:50Z

comment:328

Yes, they are collected in #33331.

vbraun · 2022-03-27T15:44:10Z

Changed branch from public/build/conda-runci to 93fce5a

mkoeppe added this to the sage-9.3 milestone Nov 1, 2020

mkoeppe added c: build labels Nov 1, 2020

mkoeppe changed the title ~~GH Actions: Add test for conda without SPKG~~ tox.ini, GH Actions: Add test for conda without SPKG Nov 3, 2020

This comment has been minimized.

Sign in to view

mkoeppe changed the title ~~GH Actions: Add test for conda without SPKG~~ Fixes for the conda-for-Sage-developers installation method, add GH Actions workflow Mar 18, 2022

This comment has been minimized.

Sign in to view

mkoeppe added s: positive review and removed s: needs review labels Mar 18, 2022

mkoeppe added p: critical / 2 and removed p: major / 3 labels Mar 18, 2022

vbraun removed the s: positive review label Mar 27, 2022

vbraun closed this as completed in 6dbf093 Mar 27, 2022

tobiasdiez mentioned this issue Jan 29, 2023

Meta-ticket: improve conda setup #33331

Closed

dimpase mentioned this issue Apr 10, 2022

add more details on conda-based installations of Sage #33426

Closed

Fixes for the conda-for-Sage-developers installation method, add GH Actions workflow #30845

Fixes for the conda-for-Sage-developers installation method, add GH Actions workflow #30845

Comments

mkoeppe commented Nov 1, 2020

This comment has been minimized.

mkoeppe commented Dec 6, 2020

mkoeppe commented Dec 6, 2020

This comment has been minimized.

tobiasdiez commented Dec 22, 2020

tobiasdiez commented Dec 22, 2020

tobiasdiez commented Dec 22, 2020

tobiasdiez commented Dec 22, 2020

tobiasdiez commented Dec 22, 2020

sagetrac-git mannequin commented Dec 22, 2020

sagetrac-git mannequin commented Dec 22, 2020

sagetrac-git mannequin commented Dec 22, 2020

sagetrac-git mannequin commented Dec 22, 2020

tobiasdiez commented Dec 22, 2020

tobiasdiez commented Dec 22, 2020

tobiasdiez commented Dec 22, 2020

sagetrac-git mannequin commented Dec 22, 2020

sagetrac-git mannequin commented Dec 22, 2020

mkoeppe commented Dec 22, 2020

mkoeppe commented Dec 22, 2020

mkoeppe commented Dec 22, 2020

sagetrac-git mannequin commented Dec 22, 2020

sagetrac-git mannequin commented Dec 22, 2020

mkoeppe commented Dec 22, 2020

mkoeppe commented Dec 22, 2020

tobiasdiez commented Mar 18, 2022

mkoeppe commented Mar 18, 2022

tobiasdiez commented Mar 18, 2022

tobiasdiez commented Mar 18, 2022

This comment has been minimized.

This comment has been minimized.

tobiasdiez commented Mar 18, 2022

This comment has been minimized.

mkoeppe commented Mar 18, 2022

mkoeppe commented Mar 18, 2022

mkoeppe commented Mar 18, 2022

mkoeppe commented Mar 18, 2022

mkoeppe commented Mar 18, 2022

mkoeppe commented Mar 18, 2022

tobiasdiez commented Mar 18, 2022

tobiasdiez commented Mar 18, 2022

tobiasdiez commented Mar 18, 2022

mkoeppe commented Mar 18, 2022

isuruf commented Mar 21, 2022

tobiasdiez commented Mar 21, 2022

vbraun commented Mar 27, 2022