sagemath: update to 9.7 #39876

tornaria · 2022-10-10T22:19:37Z

[EDIT] this ticket includes:

updates to flintlib, arb, singular, pari, gp2c, eclib, giac, gap (all those require recompilation of sagemath)
revbump for lcalc, python3-cysignals, python3-cypari2 b/c the soname change for libpari

Everything seems to be working ok, this is now ready for merging.

With respect to the original description:

Update fplll to 5.4.2 and python3-fpylll to 0.5.7 #39838, python3-memory_allocator: update to 0.1.3. #39840, python3-sympy: update to 1.11.1, adopt. #39872 are already merged.
SuiteSparse: update to 6.0.1. #39846 is not included here as it won't force rebuild
singular: update to 4.3.1p2 (fix gcc 12 build) #39866 was moved here
matplotlib 3.6.1 is working now

This includes #39838, #39840, #39846, #39872, #39866, also update eclib-20220621.

For now I'm reverting python3-matplotlib to 3.5.3_1 since 3.6.0_1 gives a lot of problems (#39857 adds python3-contourpy which will solve some, but I think there are more issues).

I want to make sure everything works ok and the full testsuite passes as is. Then I'll just move matplotlib back to 3.6.0_1 and try to see if it's feasible to fix or workaround the issues (sagemath-the-distro is still on matplotlib 3.5.2 and I didn't find anything about the update to 3.6.0)

Testing the changes

I tested the changes in this PR: currently running the testsuite in --long after the update of singular.

tornaria · 2022-10-11T12:58:25Z

There was one failure that may be related to singular, or maybe it's just some transient failure with pexpect. I'm copying it here to save it. I'll soon push an updated PR using matplotlib 3.6, the CI will run again then and we'll see.

sage -t --warn-long 30.0 --random-seed=0 /builddir/sage-9.7/pkgs/sagemath-standard/build/lib.linux-x86_64-cpython-310/sage/interfaces/expect.py
**********************************************************************
File "/builddir/sage-9.7/pkgs/sagemath-standard/build/lib.linux-x86_64-cpython-310/sage/interfaces/expect.py", line 937, in sage.interfaces.expect.Expect._eval_line
Failed example:
    singular.interrupt()
Expected:
    True
Got:
    False
**********************************************************************
File "/builddir/sage-9.7/pkgs/sagemath-standard/build/lib.linux-x86_64-cpython-310/sage/interfaces/expect.py", line 943, in sage.interfaces.expect.Expect._eval_line
Failed example:
    singular('2+3')
Expected:
    Singular crashed -- automatically restarting.
    5
Got:
    5
**********************************************************************
1 item had failures:
   2 of  16 in sage.interfaces.expect.Expect._eval_line
    [101 tests, 2 failures, 23.78 s]

dkwo · 2022-10-11T13:51:27Z

Some comments:

please have a look at Python 3.11 #39346, to see whether any changes to sagemath template can be incorporated here
at some point I can build and test on aarch64, if we care about it
I still favor the idea to split a subpackage with all the *devel stuff, needed to pass comprehensive tests

ahesford · 2022-10-11T14:33:20Z

Please make sure to pull python-3.11.patch from 6229f31.

How does sagemath break with matplotlib >= 3.6.0 (aside from the contourpy dependency that I overlooked in the last update)?

tornaria · 2022-10-11T19:13:21Z

Please make sure to pull python-3.11.patch from 6229f31.

Thanks, I'll include that and try to upstream whatever is needed to support python 3.11 (cf https://trac.sagemath.org/ticket/33842)

What's the expected/intended ETA for python 3.11?

How does sagemath break with matplotlib >= 3.6.0 (aside from the contourpy dependency that I overlooked in the last update)?

I think I managed to fix everything -- I didn't get the time to make a proper patch and push, but I'll try to do it soon and upstream it.

ahesford · 2022-10-11T19:23:51Z

The last official word I saw was a Py 3.11 release 24 Oct. I've been regularly building the python world with rc2 but haven't really run-tested much. As we get closer, I'll probably install rc2 on something that I use regularly, then bump to the official release and use it for a week or so. I'll probably aim to merge 3.11 by early November.

tornaria · 2022-10-11T21:37:10Z

I've just pushed a version that passes all doctests (x86_64) using matplotlib 3.6.1.

dkwo · 2022-10-12T18:18:14Z

Trying to natively build and check this on aarch64: I encounter 1 test failure in giac, then it works successfully with ntl and singular (using the libtool PR), but fails with sbcl (in do_build I get mmap: Invalid argument). Any ideas?

dkwo · 2022-10-12T18:18:40Z

@leahneukirchen

tornaria · 2022-10-12T20:17:19Z

Trying to natively build and check this on aarch64: I encounter 1 test failure in giac, then it works successfully with ntl and singular (using the libtool PR), but fails with sbcl (in do_build I get mmap: Invalid argument). Any ideas?

Ouch, yes... I forgot about giac so the CI failed at that. That is a known issue on giac testsuite, safe to ignore (I shall disable the test if I don't find an easy workaround). Other than that, everything passes for me.

I don't know about sbcl. Did you try building maxima with ~sbcl (or just remove aarch64 from the case in maxima template? I think maxima-ecl should be good enough for sagemath.

dkwo · 2022-10-12T22:02:03Z

Thanks, with ~sbcl then it works. After building some nocross stuff (I wish we had native builders :), it only fails a few tests by timeout, so it looks good. I used XBPS_CHECK_PKGS=yes, not full, for now.

tornaria · 2022-10-29T11:02:10Z

The check failures have to do with gap (atlasrep pkg) trying to download files from the network (and printing warnings b/c this fails since neither curl nor wget are installed in the chroot). The actual answers are ok so it seems the download is not really necessary for whatever sage is testing.

I will patch gap so it defaults to not download anything. The way the feature is implemented is not suitable for a system package anyway (wants to have a few shared 1777 directories, which I think is not an option).

Eventually, if these data files are important, we could try to download the data package and install it in the system... If someone needs this please tell me. See for instance https://www.math.rwth-aachen.de/homes/Thomas.Breuer/atlasrep/ where we are offered to download https://www.math.rwth-aachen.de/homes/Thomas.Breuer/atlasrep/atlasrepdata.tar.gz which is 22M so nothing compared to the current size of the gap package)

ahesford · 2022-10-29T11:12:55Z

I am not worried about the checks. As for the data files, let somebody who needs them PR their addition.

Can this be ready to merge within the next week? I plan to bump Python next Saturday (5 Nov) and would like not to break your Sage workflow.

tornaria · 2022-10-29T16:55:31Z

I'm not sure what went wrong with i686. All started with

sage -t --warn-long 30.0 --random-seed=0 /builddir/sage-9.7/pkgs/sagemath-standard/build/lib.linux-x86_64-cpython-310/sage/manifolds/differentiable/pseudo_riemannian_submanifold.py
**********************************************************************
File "/builddir/sage-9.7/pkgs/sagemath-standard/build/lib.linux-x86_64-cpython-310/sage/manifolds/differentiable/pseudo_riemannian_submanifold.py", line 26, in sage.manifolds.differentiable.pseudo_riemannian_submanifold
Failed example:
    E.<w,x,y> = M.chart()
Exception raised:
    Traceback (most recent call last):
      File "sage/symbolic/expression.pyx", line 13959, in sage.symbolic.expression.send_sage_domain_to_maxima (build/cythonized/sage/symbolic/expression.cpp:104209)
        assume(v, 'real')
      File "/builddir/sage-9.7/pkgs/sagemath-standard/build/lib.linux-x86_64-cpython-310/sage/symbolic/assumptions.py", line 676, in assume
        x.assume()
      File "/builddir/sage-9.7/pkgs/sagemath-standard/build/lib.linux-x86_64-cpython-310/sage/symbolic/assumptions.py", line 249, in assume
        self._validate_feature()
      File "/builddir/sage-9.7/pkgs/sagemath-standard/build/lib.linux-x86_64-cpython-310/sage/symbolic/assumptions.py", line 219, in _validate_feature
        _valid_feature_strings.update(repr(x).strip() for x in list(maxima("features")))
      File "sage/misc/lazy_import.pyx", line 391, in sage.misc.lazy_import.LazyImport.__call__ (build/cythonized/sage/misc/lazy_import.c:4393)
        return self.get_object()(*args, **kwds)
      File "sage/misc/lazy_import.pyx", line 216, in sage.misc.lazy_import.LazyImport.get_object (build/cythonized/sage/misc/lazy_import.c:2674)
        return self._get_object()
      File "sage/misc/lazy_import.pyx", line 252, in sage.misc.lazy_import.LazyImport._get_object (build/cythonized/sage/misc/lazy_import.c:2998)
        self._object = getattr(__import__(self._module, {}, {}, [self._name]), self._name)
      File "/builddir/sage-9.7/pkgs/sagemath-standard/build/lib.linux-x86_64-cpython-310/sage/interfaces/maxima_lib.py", line 119, in <module>
        ecl_eval("(set-pathnames)")
      File "sage/libs/ecl.pyx", line 1352, in sage.libs.ecl.ecl_eval (build/cythonized/sage/libs/ecl.c:10922)
        cpdef EclObject ecl_eval(str s):
      File "sage/libs/ecl.pyx", line 1375, in sage.libs.ecl.ecl_eval (build/cythonized/sage/libs/ecl.c:10855)
        o=ecl_safe_eval(python_to_ecl(s, True))
      File "sage/libs/ecl.pyx", line 321, in sage.libs.ecl.ecl_safe_eval (build/cythonized/sage/libs/ecl.c:5637)
        raise RuntimeError("ECL says: {}".format(message))
    RuntimeError: ECL says: Could not create directory "/tmp/.sage/maxima/binary"
    C library error: "File exists"

Probably a race condition, since CI is doctesting using 2 threads. More doctest failures after this are all caused by E not being defined, etc.

I don't think we should delay this PR for this (maybe someone can restart that test to be sure).

tornaria · 2022-10-29T16:58:29Z

@leahneukirchen @dkwo : I think this is ready to merge.

tornaria · 2022-10-30T17:09:41Z

@ahesford: I'm now testing this PR with python 3.11 (using your branch from #39346).

Compilation went fine, but sage fails to start due to some deprecated python stuff that was removed in 3.11. There is a candidate patch in https://trac.sagemath.org/ticket/33842 which I'm trying now. Based on the discussion there, we may be needing more patches in sagemath and/or python3-cypari2.

I'll report back, hopefully we can have this working before the python update.

tornaria · 2022-10-30T23:37:23Z

I added the update of python3-cypari2 to 2.1.3, which seems it will be necessary to support python 3.11.

However, I'm not being able to even start sage (no solution available yet, see https://trac.sagemath.org/ticket/33842#comment:48). This is a major showstopper for me.

tornaria · 2022-10-31T11:56:02Z

Just as a summary of the situation wrt to python 3.11 for @ahesford :

it seems sage is completely broken on python 3.11: it compiles but it doesn't start
sage depends on a custom implementation of WeakValueDictionary (the one in python weakref module is not good for several reasons discussed in https://trac.sagemath.org/ticket/13394)
the dict layout changed in 3.11 breaking everything
the whole thing hangs in a single low-level function which will have to be rewritten (https://git.sagemath.org/sage.git/tree/src/sage/cpython/dict_del_by_value.pyx?h=9.7#n131)
after this, the rest seems close enough, see starting at https://trac.sagemath.org/ticket/33842#comment:62

All of this does not affect the current PR with python 3.10. It would be nice to have it merged since having all the updated packages in the main repo makes it easier to work on they python 3.11 fixes.

dkwo · 2022-10-31T14:40:55Z

It would be nice to have it merged

I agree, for various reasons.

ahesford · 2022-10-31T15:37:55Z

OK, time to focus on getting this ready for Python 3.11.

@tornaria if there is no more obvious fix in the next week, I'm tempted to pull your no-op delete patch in https://trac.sagemath.org/ticket/33842#comment:63. It seems the only side-effect of this patch is that stale weakrefs will persist in the dictionary. Those refs are checked when items are accessed so it should be harmless except for some wasted memory. What say you?

tornaria · 2022-10-31T20:30:04Z

OK, time to focus on getting this ready for Python 3.11.

@tornaria if there is no more obvious fix in the next week, I'm tempted to pull your no-op delete patch in https://trac.sagemath.org/ticket/33842#comment:63. It seems the only side-effect of this patch is that stale weakrefs will persist in the dictionary. Those refs are checked when items are accessed so it should be harmless except for some wasted memory. What say you?

Not completely unreasonable. Let me try to cook up a PR to fix the rest of the issues (there are lots of issues but none look very bad at first sight, I'm already done with a few).

tornaria · 2022-11-03T13:36:20Z

@ahesford: I think I have a fix using standard python api. I'll soon make a PR after I finish testing and can clean up a bit, if you want to stick to your schedule for python 3.11, I think we will be able to do it.

ahesford · 2022-11-03T13:59:26Z

Thanks. I'm planning to start the rebuild around 1300 UTC on Saturday, 5 Nov. The whole process should take 2--3 days. As long as your fixes are ready in the next five days or so, we can support sage without interruption.

tornaria force-pushed the sagemath-update branch from 172e996 to 8961a92 Compare October 11, 2022 21:35

tornaria force-pushed the sagemath-update branch from 8961a92 to 0a9293a Compare October 12, 2022 14:46

tornaria force-pushed the sagemath-update branch 4 times, most recently from b044788 to fb78a5d Compare October 16, 2022 01:52

tornaria mentioned this pull request Oct 28, 2022

Update gap to 4.12.1 #40208

Closed

tornaria force-pushed the sagemath-update branch from 9ec08d3 to 62258e2 Compare October 29, 2022 02:30

tornaria force-pushed the sagemath-update branch from 62258e2 to b381da6 Compare October 29, 2022 12:16

tornaria marked this pull request as ready for review October 29, 2022 16:55

tornaria changed the title ~~DRAFT: sagemath update to 9.7~~ sagemath: update to 9.7 Oct 29, 2022

tornaria and others added 4 commits October 30, 2022 19:06

flintlib: update to 2.9.0.

aa12527

arb: update to 2.23.0.

3aa250b

singular: update to 4.3.1p2 (fix gcc 12 build)

104a573

pari: update to 2.15.0.

ee1fcee

tornaria added 8 commits October 30, 2022 19:06

gp2c: update to 0.0.13.

4148ceb

eclib: update to 20221012.

fc511be

giac: update to 1.9.0.27.

d570a35

gap: update to 4.12.1.

e3f1afa

lcalc: revbump for pari 2.15.0

d17efd8

python3-cysignals: revbump for pari 2.15.0

149004e

python3-cypari2: update to 2.1.3.

7ee5ea8

sagemath: update to 9.7.

c4a6d33

tornaria force-pushed the sagemath-update branch from b381da6 to c4a6d33 Compare October 30, 2022 22:06

tornaria mentioned this pull request Oct 30, 2022

igraph: update to 0.10.2. #40224

Merged

ahesford closed this in 950170e Oct 31, 2022

tornaria deleted the sagemath-update branch November 6, 2022 20:43

tornaria mentioned this pull request Feb 3, 2023

sagemath: update to 9.8 #42048

Merged

tornaria mentioned this pull request Mar 6, 2023

Upgrade: GAP 4.12 sagemath/sage#34391

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

sagemath: update to 9.7 #39876

sagemath: update to 9.7 #39876

tornaria commented Oct 10, 2022 •

edited

Loading

tornaria commented Oct 11, 2022

dkwo commented Oct 11, 2022

ahesford commented Oct 11, 2022

tornaria commented Oct 11, 2022

ahesford commented Oct 11, 2022

tornaria commented Oct 11, 2022

dkwo commented Oct 12, 2022

dkwo commented Oct 12, 2022

tornaria commented Oct 12, 2022

dkwo commented Oct 12, 2022 via email

tornaria commented Oct 29, 2022

ahesford commented Oct 29, 2022

tornaria commented Oct 29, 2022

tornaria commented Oct 29, 2022

tornaria commented Oct 30, 2022

tornaria commented Oct 30, 2022

tornaria commented Oct 31, 2022

dkwo commented Oct 31, 2022

ahesford commented Oct 31, 2022

tornaria commented Oct 31, 2022

tornaria commented Nov 3, 2022

ahesford commented Nov 3, 2022

sagemath: update to 9.7 #39876

sagemath: update to 9.7 #39876

Conversation

tornaria commented Oct 10, 2022 • edited Loading

Testing the changes

tornaria commented Oct 11, 2022

dkwo commented Oct 11, 2022

ahesford commented Oct 11, 2022

tornaria commented Oct 11, 2022

ahesford commented Oct 11, 2022

tornaria commented Oct 11, 2022

dkwo commented Oct 12, 2022

dkwo commented Oct 12, 2022

tornaria commented Oct 12, 2022

dkwo commented Oct 12, 2022 via email

tornaria commented Oct 29, 2022

ahesford commented Oct 29, 2022

tornaria commented Oct 29, 2022

tornaria commented Oct 29, 2022

tornaria commented Oct 30, 2022

tornaria commented Oct 30, 2022

tornaria commented Oct 31, 2022

dkwo commented Oct 31, 2022

ahesford commented Oct 31, 2022

tornaria commented Oct 31, 2022

tornaria commented Nov 3, 2022

ahesford commented Nov 3, 2022

tornaria commented Oct 10, 2022 •

edited

Loading