Spike: See what it would take to have plot.ly, bokeh, matpotlib work

[parente]

matplotlib notebook mode references IPython.notebook.kernel.*. I suspect the others do similar things. What shims would we need to provide to have the three libs in the title work in the dashboard server?

[jhpedemonte]

matplotlib request in ipywidgets: jupyter-widgets/ipywidgets#378

[jdfreder]

Here's a related conversation on the Plotly side of things: plotly/plotly.py#408

[jdfreder]

You should be able to used the output widget to capture these things. Have you tried that yet?

[parente]

@jdfreder Have not. We're using the jupyter-js-notebook OutputAreaModel/View as the top-level wrapper for dashboard "cells". Are you saying that putting OutputWidget in those cells where we know there's custom JS coming from the backend will help somehow?

Most of the problem I see with matplotlib's notebook implementation, for example, is that it makes direct references to IPython.notebook which just isn't there. I'm missing how OutputWidget will help (mostly out of ignorance).

[jdfreder]

Ah no, it's my fault, I misread what the problem was - the missing globals. I thought you were having trouble capturing the output because a lack of an output area. Sounds like you've got that covered though.

[parente]

An outcome I'd like to see from this is a statement of compatibility in the README here about what libraries we do and don't work with. I don't think we're going to be able to make everything work during the transition period in which libraries are adapting from working in classic notebook to alternative application containers (jupyter lab, standalone, ...)

[parente]

https://github.com/jupyter-incubator/dashboards_server/blob/master/public/js/widget-manager.js#L174

Try adding more functions here with good comments about what shims are provided and for what library.

[dalogsdon]

For matplotlib, I was able to add some shim code alongside the DeclWidgets shims in our widget manager. It's mostly just stubbing out some notebook functionality that dashboard server doesn't support, but one function does need to access output area data which requires us to poke into the internals of the output area (phosphor observable list) to extract the required data.

It also requires a list of all output areas. I extracted a list from the existing _pendingExecution object, but we may want to look into storing those in a dedicated place for when they are needed.

Initial shim code can be seen in dalogsdon@d0e13e5.

[parente]

Thanks for giving this a shot. That's not a terrible starting shim for matplotlib. Now I wonder what it looks like if you do bokeh too. Any overlap or all new shims?

[dalogsdon]

For Bokeh things mostly worked. It accessed the comm manager from Jupyter rather than IPython.

There is a comm-related error causing push_notebook() to stop working (which breaks the sliders updating the chart). In the notebook, Bokeh registers a comm target immediately, but in the dashboard server it is delayed a bit (seems to be jQuery ready function is delayed). A comm_open message is received before the Bokeh target is registered, causing the comm to not work even after the target is registered.

I'm currently digging into this issue to see if we can get the target registered or the handler delayed until it is registered.

[dalogsdon]

Ah, I didn't notice that the same error happens in the notebook (initial run doesn't work with the interact sliders). I'm seeing it work only when re-running the notebook cells.

@parente @nitind Is this a known issue? Also, the link to a Bokeh example does not work from the bokeh_demo notebook in dashboards.

[dalogsdon]

The actual error I am seeing is:

Class 418b81d5-6076-4769-97e5-bc62a3925bc8 not found in registry @ jupyter-js-services.js:3130

Looks like the same error is happening in the notebook.

[dalogsdon]

The error doesn't happen when running cells individually, specifically when running output_notebook() cell and allowing time for the JS to load. This is not an option in dashboard server, however.

The issue bokeh/bokeh#3639 gives the same error in other cases and looks to be related to the loading of JS from an external (CDN) source. An inline option is mentioned; I can see if that is possible for us to set.

[dalogsdon]

One way to solve it is to use inline mode by calling output_notebook(bokeh.resources.INLINE) to load Bokeh.

[dalogsdon]

There is also a rendering issue since Bokeh uses a table and the output area has the class rendered_html. This causes some of our styling to leak into the plot. We might need to make our rendered_html handling more sophisticated to ignore content added by other libraries.

[jhpedemonte]

My initial thought was to somehow monkey-patch Bokeh to wait for its scripts to load before executing anything. But this is probably going to be an issue with other toolkits as well.

So here's my new proposal: we execute all code cells in a loop, then wait for result to come back and pass that to jupyter-js-output-area to handle. The latter is what adds HTML/JS to the DOM. When handling results, we can add a check to see if any <script> were added to the DOM. If so, we add onLoad handlers and queue up any remaining results until those scripts have loaded.

This would be a generic solution to this issue that would work for Bokeh as well as other toolkits.

[dalogsdon]

@jhpedemonte This is an interesting potential solution. I opened #135 to investigate. For this issue we can recommend specific coding style to make things work.

[dalogsdon]

Trying plotly, I am able to get it working in the notebook, but there is an issue that breaks dashboards because it undefines requirejs globals (see comments above). They do have a fix in plotly/plotly.py#412, but it is not yet released.

[dalogsdon]

I'm also seeing the "run all" problem like with Bokeh. The plot does not render because it is looking for a global Plotly which is not yet defined. Looking to see if there is a similar workaround like the Bokeh inline mode.

[dalogsdon]

The problem with 'run all' with plotly was that requirejs was used to create a 'plotly' module, and then the plotly module was required to set window.Plotly. However, this operation is asynchronous, and the plot rendering function attempts to access window.Plotly before it is set. There needs to be another fix to plotly to wrap the plot creation in a require call. I made a fork to try this and it worked. I will see about creating a PR on plotly.

[dalogsdon]

I opened a PR with the libraries working and with an example notebook showing all 3. Currently still 2 issues:

1. Bokeh plot has some borders around it. I opened Jupyter Notebook CSS is tied to the notebook structure bokeh/bokeh#4012 to deal with this. Update: fix notebook and toolbar css handling bokeh/bokeh#4013 was created to resolve
2. Plotly has a requirejs error. I did a quick fix in a fork, and so will need to update the example notebook once it is fixed. I opened Jupyter Notebook: Fix window.Plotly not defined error when running all cells plotly/plotly.py#420 to deal with this.

[parente]

🍻 for the other bug reports and 2 line fix.

[bryevdv]

FYI going to push a PR to Bokeh today to add try/catch around the notebook comms bit, which should ameliorate the problem in bokeh/bokeh#3639 It just means that if notebook comms cannot be set up for whatever reason, that things will not fall over (push_notebook will not work int that case, of course)

update see bokeh/bokeh#4144