Support IPython 5.0

[gregsexton]

http://blog.jupyter.org/2016/07/08/ipython-5-0-released/

Ok, great. What broke this time?

As this is now LTS maybe we should make this the minimum required version for ob-ipython? I'm fed up of supporting multiple API versions.

[singletS]

One thing that broke is accessing the IPython REPL in ob-ipython via C-c C-v C-z; the new prompt-toolkit features cause emacs to hang as they fight over things like tab completion. There's a new IPython option, "ipython --simple-prompt" that disables prompt-toolkit for emacs inferior-python (mentioned in the "what's new in v5.0 IPython" docs).

However, there's an orthogonal problem: "ipython --existing" is deprecated and one must now use "jupyter console --existing", but jupyter-console doesn't have/pass the --simple-prompt option to the IPython kernel. Hopefully jupyter/jupyter_console#93 will get addressed, and thus help fix ob-ipython.

BTW, the recent merge of #67 helps, now that everything will probably need to through jupyter.

[ajsteven130]

I just ran into this problem as well and found the exact same issue as @singletS .

A workaround exists that has the same basic functionality as the --simple-prompt for ipython.

When starting up, jupyter console will look for an environment variable called JUPYTER_CONSOLE_TEST that will put it into a virtually identical mode to ipython --simple-prompt (the code that executes to change the prompt is virtually identical).

Thus, until jupyter/jupyter_console#93 can get addressed, the ob-ipython REPL will work by running (setenv "JUPYTER_CONSOLE_TEST" "1") before starting the console using the command jupyter console. This could be set in your init.el, or it can just be run by hand with M-:. With this, you'll get to use the REPL in emacs and still have access to the --existing functionality.

Looking through the source code for jupyter_console, this should not produce any other side effects (beyond the effects of the prompt) as it is only used to set the prompt.

This also works for solving jorgenschaefer/elpy#908 if you want to keep a consistent environment between ob-ipython and elpy, both of which are awesome in different ways.

[osdf]

Thank you @ajsteven130 this works (combined with #67 of course) mostly. In the REPL (opened with C-c C-v C-z) I'm getting a strange behaviour. If I try to define a function (def test(x):), hitting return after the : results in a SyntaxError: unexpected EOF while parsing. Same thing happens for any other blocks, e.g. while True:. Any idea why that happens and how to fix?

[ajsteven130]

hi @osdf. Using the simple-prompt does not currently support multiline editing, but they are trying some different things over at ipython/ipython#9816 . According to that discussion, you might try updating prompt_toolkit and launching without the workarounds discussed here. I'll test when I get a chance.

[ajsteven130]

I got a chance to try with prompt_toolkit 1.0.4, as suggested in ipython/ipython#9816 and it did not seem to help. You can see the traceback and details at ipython/ipython#9816 (comment)

[osdf]

Yes, I just tried it out myself. Without the above (set ...) thing, I see garbeled output of the jupyter console.

[osdf]

There is a new comment now for ipython/ipython#9816. What does that mean with respect to this issue (or actually the prompt problem)?

[ajsteven130]

It means there has not been any publicly visible progress. To get multiline editing, they prefer using other emacs/elpy/org-mode features. For the time being, I would suggest working like tacaswell - do your multiline editing in orgmode code blocks, python buffers, or temporary python buffers. There is also EIN, the emacs ipython notebook for ipython/jupyter integration in emacs.

It seems like the mode used for the REPL, comint mode, may be very, very hard to integrate with multiline editing and other nice things like displaying figures, etc. The choice seems to be to either use other awesome features at your disposal through ob-ipython, elpy, etc.,, or write a richer mode for the repl which is a non-trivial task.

[tacaswell]

Another possible path is something like https://github.com/Carreau/xonshpiration which is a proof-of-concept of programmaticly driving prompt-toolkit from another thread. I do not have a clear idea of how that would help, but it seems to be going in the right direction.

I (personally) view the multi-line editing stuff in the REPL a work-around to not having access to an emacs buffer in python-mode. One UI thing that would be cool (that I lack the elisp to do), is an equivalent org-edit-special from the comint prompt that drops you into a temporary python buffer and executes the code when you close it. Not clear how to integrate that with the history feature and also not sure if this is the right place to be pitching this idea.

sorry for the peanut-gallery quality of my comments on this.

[EmperorDali]

I'm running into a lot of lag running code, which I think may be related to this change in ipython. I'm running ipython 5.1.0. After the kernel and client driver startup, executing ipython babel blocks seems to work normally - the code executes normally and inserts the result into the buffer. However, it takes several seconds to run an operation that should be instantaneous

However, if I try to type commands into the REPL, there's an incredible delay in getting output. For example,

#+BEGIN_SRC ipython :session mysession :results output
print("hi")
#+END_SRC

hangs emacs for ~5 seconds, then prints. Similarly running this command in the REPL hangs for 5 seconds before printing, then another 5 seconds before printing the next ipython prompt.

Any ideas?

Update: I got a backtrace during one of these hangs - it looks like emacs is stuck in url-retrieve-synchronously. Maybe this means that the tornado server or jupyter itself is hanging momentarily for some reason?

[gregsexton]

Not sure what's happening there. Admittedly I'm running 5.0.0 but everything is instant here. Until my package manager has 5.1, I'm not really willing to upgrade. You'll need to provide more info as I can't reproduce.

[gregsexton]

5.0 is working for me now and I haven't run in to any issues: closing this.

@EmperorDali, if you're still experiencing issues and can give me more info or - better - repro steps feel free to cut a new issue.