Upgrade `ipython` to 8.17.x #35251

mkoeppe · 2023-03-09T18:16:06Z

📚 Description

📝 Checklist

This is the upgrade of ipython to 8.17 + upgrades of dependencies.

As the following issue is still open:

Recent SIGINT change breaks ipython / cysignals -- affects sagemath prompt-toolkit/python-prompt-toolkit#1576
ipython now requires prompt-toolkit >= 3.0.30, so we can no longer use the workaround of prompt_toolkit 3.0.25+ breaks Ctrl-C #33428 (pinning prompt-toolkit)
we provide a proper workaround here.
I have made sure that the title is self-explanatory and the description concisely explains the PR.
I have linked an issue or discussion.
I have created tests covering the changes.
I have updated the documentation accordingly.

⌛ Dependencies

tornaria · 2023-05-05T20:45:24Z

@mkoeppe see prompt-toolkit/python-prompt-toolkit#1576 (comment)

Doctest errors with new ipython are already dealt with in #35235 (ipython 8.11) and #35423 (ipython 8.12), both already merged.

I'm currently using ipython 8.13.1, no new changes needed.

Unless upstream approves my proposal, your way forward is to either (a) patch the updated prompt_toolkit and never allow the version from the system or (b) do the saving-restoring signals in sagemath around the call to prompt_toolkit (poc here: #33428 (comment)).

mkoeppe · 2023-05-05T21:03:50Z

Patching prompt_toolkit is not really a long-term option because we need it to be a pip-installable dependency of sagemath-repl.

Let's wait a little bit for a reaction from prompt-toolkit upstream, but your solution in #33428 (comment) looks promising

tornaria · 2023-05-05T21:58:51Z

Let's wait a little bit for a reaction from prompt-toolkit upstream, but your solution in #33428 (comment) looks promising

Only caveat is that eventually we will be saving-restoring signals twice (once in the fixed prompt-toolkit and once in sagemath) but it seems a reasonable workaround.

The solution I proposed can be adjusted as well to use the python stable API instead of cysignals as in my new patch for prompt-toolkit. Maybe that's useful (I don't know if sagemath-repl will depend on cysignals).

orlitzky · 2023-09-13T04:30:02Z

Let's wait a little bit for a reaction from prompt-toolkit upstream, but your solution in #33428 (comment) looks promising

The solution I proposed can be adjusted as well to use the python stable API instead of cysignals as in my new patch for prompt-toolkit. Maybe that's useful (I don't know if sagemath-repl will depend on cysignals).

I think we should give up waiting on upstream and go ahead with this. Distros are adopting the new prompt-toolkit (https://repology.org/project/python:prompt-toolkit/versions), and patching it with a fix not accepted by upstream is a hard sell. The corresponding distro sage will just suffer from the bug until something changes.

orlitzky · 2023-11-04T01:56:31Z

@tornaria is this the solution you had in mind?

diff --git a/src/sage/repl/interpreter.py b/src/sage/repl/interpreter.py
index b9a222c12ef..4ddb8aa94a3 100644
--- a/src/sage/repl/interpreter.py
+++ b/src/sage/repl/interpreter.py
@@ -153,6 +153,18 @@ from IPython.terminal.embed import InteractiveShellEmbed
 from IPython.terminal.ipapp import TerminalIPythonApp, IPAppCrashHandler
 from IPython.core.crashhandler import CrashHandler
 
+from ctypes import pythonapi, c_int, c_void_p
+# The following functions are part of the stable ABI since python 3.2
+# See: https://docs.python.org/3/c-api/sys.html#c.PyOS_getsig
+
+# PyOS_sighandler_t PyOS_getsig(int i)
+pythonapi.PyOS_getsig.restype = c_void_p
+pythonapi.PyOS_getsig.argtypes = c_int,
+
+# PyOS_sighandler_t PyOS_setsig(int i, PyOS_sighandler_t h)
+pythonapi.PyOS_setsig.restype = c_void_p
+pythonapi.PyOS_setsig.argtypes = c_int, c_void_p,
+
 
 # TODO: This global variable _do_preparse should be associated with an
 # IPython InteractiveShell as opposed to a global variable in this
@@ -287,6 +299,18 @@ class SageTerminalInteractiveShell(SageShellOverride, TerminalInteractiveShell):
         backend = BackendIPythonCommandline()
         backend.get_display_manager().switch_backend(backend, shell=self)
 
+    def prompt_for_code(self):
+        # save sigint handlers (python and os level)
+        # https://github.com/prompt-toolkit/python-prompt-toolkit/issues/1576
+        # https://github.com/sagemath/sage/issues/33428
+        # https://github.com/sagemath/sage/pull/35251
+        import signal
+        sigint = signal.getsignal(signal.SIGINT)
+        sigint_os = pythonapi.PyOS_getsig(signal.SIGINT)
+        text = TerminalInteractiveShell.prompt_for_code(self)
+        signal.signal(signal.SIGINT, sigint)
+        pythonapi.PyOS_setsig(signal.SIGINT, sigint_os)
+        return text

(It seems to work for me but I haven't had the energy to read through the documentation for everything yet.)

If so, can we add it to this PR? It'd be nice to have a modern version of ipython in 10.2.

dimpase · 2023-11-09T09:43:51Z

I've added the workaround in #35251 (comment) to the branch. Is there a quick way to test it?

tornaria · 2023-11-09T11:52:52Z

I've added the workaround in #35251 (comment) to the branch. Is there a quick way to test it?

See #33428 for the original way to reproduce it.

There are other ways to reproduce it directly in ipython in prompt-toolkit/python-prompt-toolkit#1576 and void-linux/void-packages#35712.

@orlitzky: your diff is what I had in mind. My actual solution is to patch prompt_toolkit to default handle_sigint to False (https://github.com/void-linux/void-packages/blob/master/srcpkgs/python3-prompt_toolkit/patches/dont-handle-sigint.patch).

An alternative would be to somehow "fix" ipython to pass handle_sigint=False to prompt_toolkit.

dimpase · 2023-11-09T23:55:33Z

OK, so forever test, as in prompt-toolkit/python-prompt-toolkit#1576, passes just fine.

sage: import forever
sage: forever.run()
^C---------------------------------------------------------------------------
KeyboardInterrupt                         Traceback (most recent call last)
Cell In[2], line 1
----> 1 forever.run()

File /mnt/opt/Sage/sage-dev/forever.pyx:4, in forever.run()
      2 
      3 def run():
----> 4     sig_on()
      5     while True:
      6         pass

KeyboardInterrupt: 
sage:

dimpase · 2023-11-10T09:29:24Z

it goes without saying that Sage's ipython does not behave as sage, i.e. forever.run() cannot be interrupted by Ctrl-C, as opposed to being run in Sage's CLI.

orlitzky · 2023-11-10T13:43:23Z

I finally read the python docs and now understand what that code is doing. It should do the right thing.

orlitzky · 2023-11-12T20:29:17Z

The comments in ipython/install-requires.txt and prompt_toolkit/install-requires.txt are no longer needed, otherwise LGTM. Everything is working with system packages on Gentoo.

orlitzky · 2023-11-12T20:30:27Z

Feel free to re-positive-review it on my behalf if you push those two comment fixes.

mkoeppe · 2023-11-12T23:40:50Z

Also the constraint added in #36659 needs to be reverted

…e as in install-requires.txt" This reverts commit 7dabb0a.

github-actions · 2023-11-24T15:35:21Z

Documentation preview for this PR (built with commit 563a31a; changes) is ready! 🎉

mkoeppe · 2023-11-24T17:35:11Z

Doctest failures. Same that are fixed already in #36129.

dimpase · 2023-11-24T17:49:23Z

what I see are

2023-11-24T17:08:48.2168346Z sage -t --random-seed=122800131133740979597912118662376657835 src/sage/interacts/test_jupyter.rst  # 2 doctests failed
2023-11-24T17:08:48.2168996Z sage -t --random-seed=122800131133740979597912118662376657835 src/sage/repl/ipython_kernel/interact.py  # 2 doctests failed
2023-11-24T17:08:48.2169517Z sage -t --random-seed=122800131133740979597912118662376657835 src/sage/repl/ipython_kernel/widgets.py  # 2 doctests failed
2023-11-24T17:08:48.2170067Z sage -t --random-seed=122800131133740979597912118662376657835 src/sage/repl/ipython_kernel/widgets_sagenb.py  # 2 doctests failed

and this is all boiling down to AttributeError: module 'ipykernel.comm.comm' has no attribute 'BaseComm'
(no idea what it is though)
anything else?

mkoeppe · 2023-11-24T17:50:57Z

So why not just test the PR that already works - #36129.

dimpase · 2023-11-24T19:27:14Z

this PR has an important technical fix, enabling an update to the modern prompt_toolkit.

it's not logical to put it into a mega-update PR (which imho should be "remove stuff" PR)

This is now based on JupyterLab. There are many changes in dependencies. We are trying to use platform-independent wheels when available, to avoid any Node.JS activity during our build. Hence we remove our `nodeenv`, `nodejs`, `jupyter_packaging`, `hatch_nodejs_version` packages. There is trouble on the horizon regarding our model of building everything from source: The latest versions of `jsonschema` have switched from `pyrsistent` to a Rust-based package. We use the newest versions that don't pull in the build dependency on the Rust compiler.   - Resolves sagemath#24904 - Resolves sagemath#30399 - Resolves sagemath#33772  ### 📝 Checklist     - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies  - Depends on sagemath#35251 (merged here)  URL: sagemath#36129 Reported by: Matthias Köppe Reviewer(s): Eric Gourgoulhon, Kwankyu Lee, Matthias Köppe, Michael Orlitzky

tornaria · 2023-12-12T15:44:26Z

@tornaria is this the solution you had in mind?
diff --git a/src/sage/repl/interpreter.py b/src/sage/repl/interpreter.py
index b9a222c12ef..4ddb8aa94a3 100644
--- a/src/sage/repl/interpreter.py
+++ b/src/sage/repl/interpreter.py
...
(It seems to work for me but I haven't had the energy to read through the documentation for everything yet.)

If so, can we add it to this PR? It'd be nice to have a modern version of ipython in 10.2.

Just as a heads up: prompt_toolkit 3.0.42 was released today, and it fixes the issue for good. This means the workaround above will not be necessary as long as prompt_toolkit is >= 3.0.42.

@orlitzky @mkoeppe

mkoeppe · 2023-12-12T16:06:15Z

Excellent!

mkoeppe added the c: packages: standard label Mar 9, 2023

orlitzky mentioned this pull request Aug 26, 2023

Use system python packages with --enable-system-site-packages #36141

Merged

orlitzky mentioned this pull request Oct 28, 2023

Support a few more Cython system packages #36548

Merged

mkoeppe added this to the sage-10.3 milestone Nov 8, 2023

dimpase force-pushed the ipython_3_11 branch from 642b1a1 to 195e94f Compare November 8, 2023 23:56

mkoeppe changed the title ~~Upgrade ipython to 8.11~~ Upgrade ipython to 8.17.x Nov 9, 2023

dimpase marked this pull request as ready for review November 10, 2023 15:44

dimpase added the s: needs review label Nov 10, 2023

vbraun force-pushed the develop branch 2 times, most recently from 883e05f to e349b00 Compare November 12, 2023 16:25

orlitzky added s: positive review and removed s: needs review labels Nov 12, 2023

mkoeppe added s: needs work and removed s: positive review labels Nov 12, 2023

dimpase force-pushed the ipython_3_11 branch from f97fbe1 to 5d48b9e Compare November 13, 2023 13:15

dimpase added s: needs review and removed s: needs work labels Nov 13, 2023

Matthias Koeppe and others added 14 commits November 24, 2023 12:10

build/pkgs/jedi: Update to 0.18.2

975cd41

build/pkgs/wcwidth: Update to 0.2.6

b89786f

build/pkgs/pygments: Update to 2.14.0

74fe4fe

build/pkgs/stack_data: Update to 0.6.2

4aea03e

build/pkgs/asttokens: Update to 2.2.1

692df57

build/pkgs/prompt_toolkit: Update to 3.0.38

3460d50

build/pkgs/prompt_toolkit/install-requires.txt: Tentative update

213872d

build/pkgs/ipywidgets: Update to 8.0.4

adfe8a7

ipython 8.17.2, rebase

971ab2d

fix for prompt-toolkit

1be8323

bump ipywidgets to 8.1.1

d3b7454

Revert "build/pkgs/prompt_toolkit/distros/conda.txt: Use version rang…

9db3eeb

…e as in install-requires.txt" This reverts commit 7dabb0a.

remove obsolete comments, sync the version of prompt_toolkit for conda

82fafb3

remove space in conda.txt

09bbe60

dimpase force-pushed the ipython_3_11 branch from 1947710 to 09bbe60 Compare November 24, 2023 12:10

Matthias Koeppe and others added 3 commits November 24, 2023 14:20

build/pkgs/comm: New (ipywidget dep)

2285fc3

trim license

c122241

minimise changes wrt rc4

563a31a

mkoeppe added the r: duplicate label Nov 24, 2023

dimpase closed this Nov 25, 2023

github-actions bot removed the s: needs work label Nov 25, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade `ipython` to 8.17.x #35251

Upgrade `ipython` to 8.17.x #35251

mkoeppe commented Mar 9, 2023 •

edited by dimpase

Loading

tornaria commented May 5, 2023

mkoeppe commented May 5, 2023

tornaria commented May 5, 2023

orlitzky commented Sep 13, 2023

orlitzky commented Nov 4, 2023

dimpase commented Nov 9, 2023

tornaria commented Nov 9, 2023

dimpase commented Nov 9, 2023

dimpase commented Nov 10, 2023

orlitzky commented Nov 10, 2023

orlitzky commented Nov 12, 2023

orlitzky commented Nov 12, 2023

mkoeppe commented Nov 12, 2023

github-actions bot commented Nov 24, 2023

mkoeppe commented Nov 24, 2023

dimpase commented Nov 24, 2023

mkoeppe commented Nov 24, 2023

dimpase commented Nov 24, 2023

tornaria commented Dec 12, 2023

mkoeppe commented Dec 12, 2023

Upgrade ipython to 8.17.x #35251

Upgrade ipython to 8.17.x #35251

Conversation

mkoeppe commented Mar 9, 2023 • edited by dimpase Loading

📚 Description

📝 Checklist

⌛ Dependencies

tornaria commented May 5, 2023

mkoeppe commented May 5, 2023

tornaria commented May 5, 2023

orlitzky commented Sep 13, 2023

orlitzky commented Nov 4, 2023

dimpase commented Nov 9, 2023

tornaria commented Nov 9, 2023

dimpase commented Nov 9, 2023

dimpase commented Nov 10, 2023

orlitzky commented Nov 10, 2023

orlitzky commented Nov 12, 2023

orlitzky commented Nov 12, 2023

mkoeppe commented Nov 12, 2023

github-actions bot commented Nov 24, 2023

mkoeppe commented Nov 24, 2023

dimpase commented Nov 24, 2023

mkoeppe commented Nov 24, 2023

dimpase commented Nov 24, 2023

tornaria commented Dec 12, 2023

mkoeppe commented Dec 12, 2023

Upgrade `ipython` to 8.17.x #35251

Upgrade `ipython` to 8.17.x #35251

mkoeppe commented Mar 9, 2023 •

edited by dimpase

Loading