javascript rendering of system diagrams

[RussTedrake]

(I wrote this in an email for someone potentially willing to help with the cause; but realized it deserves to be here in an issue for anybody that might want to help).

Quick background:

• Drake builds system diagrams like Simulink, etc (but programmatically in python/c++, instead of visually). Example here:
  https://mybinder.org/v2/gh/RobotLocomotion/drake/nightly-release?filepath=tutorials/dynamical_systems.ipynb

• You’ll see in the second-to-last cell on that tutorial that, because of it’s c++ origins, we never spent much time visualizing the resulting diagrams. But we offer the ability to kick-out a graphviz .dot file. In jupyter notebooks, we render that to a PNG and display the PNG.

• I took a spin earlier this summer and made a version that renders to a js diagramming tool, instead:
  implement javascript system diagram visualization #13644
  i did the javascript coding for this in
  https://observablehq.com/@russtedrake/gojs-for-drake-system-diagrams

• It turns out the particular diagramming tool that i choose was rejected because of the licensing implications:
  Remove GenerateHtml #13705
  but, as you see at the bottom of that issue, it might be possible to simply render the .dot file in js.

I think a few hours of trying some of the js diagramming libraries and deciding what works would result in an immense improvement in the user/student experience. I think one key feature is the ability to expand/collapse nested subdiagrams.

[RussTedrake]

I spent a little more time looking through the options yesterday, but haven't found one that is obviously good.

To address #13975, I realized pydot could create svg. For moderate size diagrams, that is a reasonable solution for diagram visualization, too.

I went through the entire list of javascript packages in https://codefreezr.github.io/awesome-graphviz/#language-bindings , as recommended by @jwnimmer-tri in #13705. Those links took me to https://github.com/dagrejs, which is the most promising path I've seen from graphviz to a javascript rendering. But even this package doesn't appear to support collapsing/expanding subdiagrams, so I feel the additional value it adds over the python-based svg rendering is not worth the trouble.

Because of the existing SVG solution, I am going to upgrade the feature request of collapsing/expanding here to a "must have". I honestly haven't found a good gojs alternative. I'm almost of the opinion that we should just write our own in d3. I bet I could do some pretty nice graph layout optimization in mathematicalprogram (probably a MILP).

I'm also ok if it takes a different form from collapse/expand. It might be possible to use a combination of max_depth on the graphviz and links in something like dagrejs to open subdiagrams.

[rpoyner-tri]

Seems like someone with js chops (not me) could do something decent with https://github.com/magjac/d3-graphviz (BSD 3-clause license). If it is possible to enumerate the set of expanded/collapsed graphs ahead of time, it may not be too hard to provide some navigation among them.

[jwnimmer-tri]

Does @trowell-tri fit that bill?

[RussTedrake]

There is one deprecated example of using GenerateHtml left in https://deepnote.com/project/Ch2-Lets-get-you-a-robot-j4YXK7WXTOubrZLRGsemzA/%2Frobot.ipynb which shows the old feature that we are trying to replace.

[jwnimmer-tri]

The recent https://github.blog/2022-02-14-include-diagrams-markdown-files-mermaid/ caught my eye. Possibly its https://github.com/mermaid-js/mermaid#readme will be a winner in terms of a durable graph modeling language for us to build upon.

[RussTedrake]

mermaid looks great, but it doesn't look to support collapsable subdiagrams. I think that's a required feature for this.

[shensquared]

I was just searching for interactive (i.e. collapsable) mindmap/diagram libraries myself and came across this issue. https://github.com/gera2ld/markmap (their demo) fit my bill. Not too sure about the specific needs here, but sharing fwiw.

[shensquared]

Chatted with Russ a bit yesterday and I understand the needs better now. https://js.cytoscape.org/ and its plugin https://ivis-at-bilkent.github.io/cytoscape.js-expand-collapse/ looked promising.

Building on top of those (and some other auxiliary libs, all MIT license), I cobbled together https://shensquared.github.io/diagram-proto/demo.html. The interaction isn't intuitive yet, so here's a gif.

Comparing with the one in https://manipulation.csail.mit.edu/intro.html#example3 that Russ made with GoJS, I think we'd still need:

• more polished one-click collapse/expand
• more complex graph that’s hopefully easily declared/populated.

Both feel doable. I think I actually know how to tweak the listener/event. For the visual, cloning Russ's graph is for sure possible; in fact, Cytoscape's nodes/edges API looks pretty similar to GoJS (JSON based, uses keys like from, to, group, etc). I do need to look closer if their layout also has a smart ‘auto’ mode like GoJS, but even if it's not built in, seeing that they have a lively community (building extensions, plugins, etc), I’m optimistic.

Before we pursue this further though: wonder if I missed anything on the wish list?

[RussTedrake]

Thanks @shensquared ! I'm not so concerned about how "polished" the collapse/expand looks; as long as you can connect multiple ports in one diagram and get somewhat reasonable layouts, then this looks great!

[shensquared]

Sounds good. My to-do plan then, in order:

• One-to-many and many-to-one connection
• Easier/auto layout (my demo involved setting positions manually)
• A Generating function (i.e. takes a .yaml or a python dictionary and pops out the .html automatically)
• Beautify and polish (e.g. maybe also match the Drake color scheme)

Will let you know if/when the first two (the essential ones) are done.

[shensquared]

@RussTedrake Updated the demo towards the first 2 items on my list. Does it look like what you had in mind?

In parallel, I plan to start trying auto-generate a real Drake diagram (instead of using 'Lorem ipsum' nodes/edges). Maybe from e.g. a .yaml? Wonder how/where did you get the port names and connections for your GoJS diagram?

[RussTedrake]

Looks great!

…

On Tue, Mar 15, 2022 at 10:26 PM, Shen Shen ***@***.***> wrote: @RussTedrake <https://github.com/RussTedrake> Updated the demo <https://shensquared.github.io/diagram-proto/demo.html> towards the first 2 items on my list. — Reply to this email directly, view it on GitHub <#13874 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABRE2NAYFNALL3Y53SNLCMDVAFBETANCNFSM4QAFRCSA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[shensquared]

Sorry it's been a while.

Below is a mimic of Russ's manip_station diagram:
The demo; it's based on Cytoscape.js, and line 24-34 toggles different edge styles and layouts.

Some shortcomings when compared with gojs:

• Edge routing is not as smart; e.g., I haven't been able to get zig-zag edges that 'try' to avoid crossing nodes
• Initial layout looks reasonable; but it gets much more messier much more quickly than gojs as more nodes get expanded (this compounds with the edge routing issue).

As I was trying to improve these, questions/answers like this makes me a bit less optimistic about the base library.

So while I'm still trying, I'm also keeping an eye for other libraries. I'm also happy to bring this one in as is into GenerateHTML first (e.g. if we wanna first see how other Drake diagrams might look like with this?)

@RussTedrake any preference? thanks!

[shensquared]

The demo looks/works a lot better to me now.

The rendering is via Cytoscape; the collapse/expand is via cytoscape.js-expand-collapse; the layout is done by elk and then wrapped/parsed via cytoscape.js-elk. While not perfect, this combo is my favorite so far :D

Fwiw, other libraries that didn't quite work out but probably would be great depending on the need (so 'good-to-know's):

• G6: The layout breaks my heart; dagre and comboCombined layouts are actually the best I got it to look :( But it has otherwise very rich and cool interactions.
• X6: When collapsing, edge is 'ghosting' (see demo, change line 187 'aa' to 'aaa', and then collapse node 'aa'). But also very rich features and very actively maintained.
• Joint: A different edge update issue (see demo, collapse the single node group on the right); also doesn't support mouse-wheel zoom in/out natively. But easy setup, solid basic graphing features, and detailed docs.
• sprotty : Simply not enough docs yet so I didn't get very far. But it's developed by the same people behind elk, so potentially better scalability/compatibility (if these become an issue down the road).

[RussTedrake]

That demo does look great. Thanks for doing it! Is there any chance that the input ports / output ports for a single system (e.g. the plant in this demo diagram) can be left/right justified?

[shensquared]

seems possible in elk though the wrapper has some lost-in-translation; I'll look more into it.

[shensquared]

Is there any chance the input ports / output ports for a single system (e.g. the plant in this demo diagram) can be left/right justified?

Check this out?

Also added in that demo:

• a mini navigation map
• mouse over ports to highlight their in/out connections
• click on ports to go to a URL (e.g. that port/system's code on github or their docs page or sth; I just put drake.mit.edu as a placeholder for now)

Feel free to let me know too if these are irrelevant/counterproductive? Thanks.

[zapplelove]

I think the WebGME is a good choice, it can achieve the rendering of system diagrams and drag/drop modeling. But unfortunately, I'm not good at coding. Can anyone achieve it?

@RussTedrake @shensquared @rpoyner-tri @jwnimmer-tri