support interactive plots in CoCalc Jupyter notebooks

[williamstein]

Also, ensure "%matplotlib notebook" works, which maybe uses widgets?

WORKAROUND: Use classical Jupyter mode, where widgets are fully supported.

[williamstein]

I will be starting implementation of this very soon.

[williamstein]

dup of #1930

[haraldschilly]

I don't think this is a dup. Here is the implementation of nbAgg: https://github.com/matplotlib/matplotlib/blob/master/lib/matplotlib/backends/backend_nbagg.py

I've changed the title to interactive plots instead of widgets, too.

[williamstein]

I guess this is really -- make it so

%matplotlib notebook
from pylab import *
plot([1,2,7], [2,4,5])

works. It does NOT work yet, because it tries to load some sort of extension that cocalc widgets can't or doesn't support, I guess.

So we have widgets, but not enough of widgets...

[williamstein]

This is definitely not supported still. It requires "custom widgets"...

[haraldschilly]

this came up today in an support email, well, still open. a possible workaround is to use bokeh … but then you have to rewrite a little bit how you code the plots.

[williamstein]

Another workaround might be to use plotly, which also involves rewriting code a bit and has other disadvantages (involving where the code lives).

This PR is about adding support for one complicated 3rd party widgets, and as a part of it I've increased the possibility of using others: #5902 As part of that, I'll look into what matplotlib tries to use; maybe it is now easy to support.

[williamstein]

In our current JupyterLab deployment, this happens:

%matplotlib notebook
from pylab import *
plot([1,2,7], [2,4,5])

OUTPUT: "Javascript Error: IPython is not defined".

I think this means there is some Jupyter classic extension trying to be used. Is there a JupyterLab version of %matplotlib notebook?

(And wow, evidently I can't copy output from JupyterLab right now. That's a big bug.)

[haraldschilly]

There is %matplotlib widget, needs this extension https://github.com/matplotlib/ipympl

so, maybe it's worth looking into this more modern way instead

[haraldschilly]

their website is interesting, explains for various online platforms how this backend works: https://matplotlib.org/ipympl/

[williamstein]

Thanks. The basic process is to find the corresponding npm module, which is:

https://www.npmjs.com/package/jupyter-matplotlib

install it, hook it into our widget system, and then see all the things that break due to subtle assumptions that might be satisfied in my implementation of ipywidgets (which is in some ways different than the official one), then try to fix those issue. I'll try.

[williamstein]

%matplotlib widget does work in our Jupyer classic server install. The surprise for me is that it kind of sucks (?). The functionality is very limited, but also just trying to pan around is extremely laggy.

I also tried installing ipympl and plugging it in as a custom widget into cocalc and it just silently draws an empty canvas. It would be possible to debug, but take quite a while, probably. It's very odd that there are no error messages at all.

I suspect it might be much better to invest effort into bqplot than this. That said I've also never used bqplot, since it isn't installed in either our jupyterlab or jupyter classic environments either. I just suspect it might be much better than this, due to many years of serious development at Bloomberg by Sylvaine...

[williamstein]

I just thought about this some more and looked at the javascript source code of this https://github.com/matplotlib/ipympl, and it does have the nice advantage of being typescript, pretty small, and with a significant increase in work on it recently. Also, rendering using canvas instead of png/svg seems pretty cool... Plus full support for matplotlib is of course very nice, given that it is so popular. Also, if they haven't already fixed panning being slow in master, maybe I can fix that sometime.

[mforbes]

Why does this not work with the CoCalc backend? Is it not just displaying some HTML and Javascript?

import numpy as np, matplotlib.pyplot as plt
import mpld3

Nxy = (64, 65)
Lxy = (5., 5.)

x, y = np.meshgrid(*(np.arange(_N)*_L/_N - _L/2 for _N, _L in zip(Nxy, Lxy)), indexing='ij', sparse=True)
z = (x+y)*np.exp(-abs(-(x+1j*y))**2)
fig, ax = plt.subplots()
ax.pcolormesh(x.ravel(), y.ravel(), z.T, shading='auto')
ax.set(aspect=1)
mpld3.display()

Returns WARNING: 1 intermediate output message was discarded. Is this issue #4590?

[MathMiller]

I have a related issue running Julia in the CoCalc Jupyter environment. The following code returns an image when run in CoCalc in a Jupyter Lab or Jupyter Classic server, but not in the CoCalc default:

using Plots

logisticiteration(r,x) = r * x * (1-x)

x₀ = 0.5

rvalues = 0:0.005:4

discardediterations = 200;
maxoscillatoryvalues = 100;

points = Vector{Tuple{Float64,Float64}}(undef, 0)

for rvalue ∈ rvalues
    x = 0.5
    for i ∈ 1:discardediterations
        x = logisticiteration(rvalue,x)
    end
    maxdiscardedvalue = x
    for i ∈ 1:maxoscillatoryvalues
        x = logisticiteration(rvalue,x)
        push!(points,(rvalue, x))
        if abs(x - maxdiscardedvalue) < 0.001
            break
        end
    end
end

bifurcationplot = scatter(points,
    markersize = 1,
    markercolor = :black,
    legend = false,
    xlims = (min(rvalues...),max(rvalues...)),
    ylims = (-0.5,1.5))

Running it in CoCalc asks me to "Fetch additional ouput" and then gives a WARNING: 1 intermediate output message was discarded. error.

[williamstein]

Why does this not work with the CoCalc backend? Is it not just displaying some HTML and Javascript?

@mforbes, this will work after #7687 is merged:

[williamstein]

I think there is no need or reason to support "%matplotlib notebook", which is the ancient deprecated JupyterClassic extension. It's not even supported by JupyterLab (hence Jupyter classic v7+) anymore:

So this issue is about support %matplotlib widgets, which is a custom widget, and definitely something we should support, and a goal of #7687 and something I'm making progress on:

[williamstein]

%matplotlib widget
from pylab import plot
plot([1,2,7], [2,4,5])

[williamstein]

As of right now, %matplotlib widget works. Also mpld3 works as well. So I'm finally closing this!