Expose Scheduler Status

[mrocklin]

We need to expose the state of the network to expert users who are generally curious about why their jobs are taking a while, the health and state of their workers, etc..

People want to access this information from a variety of locations, including interactive terminals and notebooks, web pages, and programming and network interfaces. I'm taking the following approach:

1. Built HTTP endpoints that serve valuable information
2. Use console/notebook interactions to drive point 1
3. Hope that other better qualified people come by and help build a Web UI

cc @canavandl @quasiben @martindurant

[mrocklin]

I've updated the header text with my approach. Here is a janky start.

$ dscheduler
$ dworker 127.0.0.1:8786
$ dworker 127.0.0.1:8786

This uses IPython widgets at the moment, but most of the actual logic is well separated. It might be better to use Bokeh plots in the future just so that we can more easily cross the notebook/web-ui divide.

Some questions:

1. Can we style the dataframes more effectively? (cc @TomAugspurger)
2. Do we want to switch to something more graphical, like an IPython bar or Bokeh plot?
3. What other information do we want to publish about the Workers? I plan to get full memory usage and availability soon. Do we care about hardware, hostnames, aliases?
4. Can we improve the names used for tasks?

[mrocklin]

Note that the scheduler status widget shown here differs from the progressbar in that the new widget is all about scheduler and worker state, not the state of a particular set of tasks.

I included the progressbar here though so that we can raise our visual standards a bit.

cc @broxtronix

[nfaggian]

This work looks really exciting! I think a dynamically updating graph of the nodes (like your dot visuals) would be amazing.

[mrocklin]

The dot visuals don't scale well past 100 or so tasks. It would be great to find an alternative for larger numbers of tasks. If you have ideas on this then we should probably talk about it on another issue.

[TomAugspurger]

@mrocklin Sorry I haven't had a chance to look closely at this yet, I've been having trouble getting the widgets to display, even progress isn't showing up. Haven't been able to track down why yet.

In general I'd say just be on the lookout for when the .style stuff starts to get in your way. It's really flexible, since you can store just about anything in the DataFrame and you have a lot of control over how each cell is displayed as of pandas-dev/pandas#12162. I had Bokeh plots in a DataFrame just for fun. But if this grows complicated enough the structure provided by the Dataframe may become restrictive. Will look more later.

[astrojuanlu]

I stumbled upon this PR looking for a way to embed the task graphs into the notebook itself. Is there a current incarnation of the live_info you display in the first GIF?

[mrocklin]

@Juanlu001 no, not currently. I think that the right way to do this today would be to find some way to embed the bokeh graphs we're serving on ports 8787 and 8788 into the notebook, perhaps with an iframe. I'm not entirely sure about the best way to do this though.