Merge website, doxygen pages, and wiki?

[derekbruening]

Split from #3194

Most of our user-facing content is prose inside the doxygen pages, leaving just a few pages on the main website. And the two have completely different styles and layout. We could consider merging in one direction or another: we could remove all the non-API content from doxygen control and move it to the Pages and embed the API content in an iframe (maybe remove the navbar, etc. there); or we could go the other way and try to put all the top-level pages inside doxygen auto-generation.

We could also consider removing the wiki and moving its content into the website, to have everything integrated in one place.

[derekbruening]

The downside of moving content out of doxygen is that the local docs in a package or local build would then lose that content. However, I do not know that the typical user looks at the local docs in any case. It may be better to have nicer online docs.

[derekbruening]

@AssadHashmi , @johnfxgalea -- looking for some more opinions.

Also, now that the website is editable by others in the dynamorio.github.io repository, it can be improved by the community: e.g., #4112, or other style changes.

[derekbruening]

Another downside of moving the tool docs is that they would no longer live next to their code: e.g., instead of drcachesim.dox.in right in the drcachesim/ code directory, it would live in the separate dynamorio.github.io repository. (I suppose we could have dynamorio be a submodule of the docs repo and have jekyll pull out just some targeted tool doc files.)

[derekbruening]

An upside is that we could add a search box over all the docs, versus the current search box in the doxygen-generated html which searches only the code tokens and does not search the prose at all.

[derekbruening]

I'm leaning toward:

1. Moving all the wiki pages into the website and removing the wiki.

2. Putting in the work to integrate the doxygen output into the jekyll site. This would involve removing the side frame and having doxygen generate plain html pages, pushing them into a subdir of dynamorio.github.io with a script from Travis, and building up navigation menus using the doxygen file names. This could allow us to keep the content in both release packages and online.

[derekbruening]

My new plan is to duplicate:

• Keep the wiki, but place a copy of it (updated w/ the weekly cronbuild) inside the web site.
• Generate both standalone framed doxygen html and doxygen suitable for embedding.

I put some work into integration of doxygen and put it up on dynamorio.org.

I have a menu system using doxygen's own javascript modified to be extendable (had to scrap their "skip list" indexing and inline all the submenus).

I added a whole-site search using lunr which searches the prose docs and the other pages. However it is not perfect for API searches: #4740.

Still to do beyond improving search which as mentioned is #4740:

• Automate updating the embedded doxygen. This requires generating a separate copy with no frames, and then editing all the html files to remove the headers and editing all the menu .js files to use jekyll includes to inline themselves. Then pushing the result into the dynamorio.github.io repo. And probably deleting the dynamorio_docs repo.

• The group pages are empty: I think this is a problem with doxygen 1.9.1 on the machine where I generated this content. Compare http://dynamorio.org/docs/group__drwrap.html and https://dynamorio.org/dynamorio_docs/group__drwrap.html

• Header file anchors have no menu entries. E.g., click on upper-right "Functions" at https://dynamorio.org/docs/dr__tools_8h.html which goes to https://dynamorio.org/docs/dr__tools_8h.html#func-members and loses the menu place. Although it seems that pure doxygen has the exact same problem: https://dynamorio.org/dynamorio_docs/dr__tools_8h.html#func-members

• Add a copy of the wiki. Once it's there, we can change the Download, etc. top-level menu links to be in-site instead of going off externally.

[johnfxgalea]

Another option might be to move the wiki & website to Docusaurus, which supports document search powered by Algolia DocSearch (or perhaps use this directly). However, I'm not sure if it works on HTML produced by doxygen, and might be an overkill anyways given our current setup.

https://v2.docusaurus.io/docs/search

[derekbruening]

The doxygen docs are now nicely integrated.

But I'm looking for help deciding what to do with the wiki.

I added Debugging to the live web site with menu entries for all sections in
DynamoRIO/dynamorio.github.io@02e5f61 to visualize what it would look like.

There are a couple of goals here:

• Better navigation for wiki pages. Currently they are very long. We would like a table of contents and sidebar menus for each.
• Better integration of content. Some wiki content is relevant to users: debugging, etc., and should perhaps be integrated into the doxygen pages.
• Searching integrated with all the other content.

I was hoping we could just make a copy of the wiki markup files and they would work as-is for jekyll. But there are complications:

• Some are mediawiki format which would require a converter. IIRC we only used that format to get a table of contents inserted.
• We need to insert the jekyll header on every file.
• We need to generate the javascript menu files.

So we need a script for all that. Github Pages disallows jekyll plugins, so we have to do it ahead of time. So either we build a script to generate JS menus, insert header + toc, and convert mediawiki and run at package deploy time, or we move the wiki content instead of having a copy.

Pros of deleting the wiki and moving its content into the web pages:

• Have TOC and sidebar menus for all content
• No fiddly, fragile script to get copy looking good
• No converting from mediawiki to kramdown for jekyll
• No 1-week delay before changes appear on the web site

Cons of moving:

• Editing web menus is currently manual: could be painful when
  adding/changing subsections
• No WYSIWYG easy editing interface: but how often do we edit anyway?
• Broken old wiki links: but that's temporary

An alternative is to move the wiki into the doxygen pages to get auto-menu-generation, better content integration (e.g., Debugging should be with the rest of the user docs). But, the doxygen prose pages are not quite as nice to edit as the jekyll markdown.

We could also move some existing web content into doxygen: e.g., Tutorials and the API Usage Tutorial should perhaps be combined.

Any opinions?

[derekbruening]

Editing web menus is currently manual: could be painful when
adding/changing subsections

Although I abandoned menu generation with Liquid when it had to generate all the expansion and highlighting, maybe we can use Liquid to auto-generate the JavaScript arrays for the wiki pages. If that works out, I would lean toward deleting the wiki and moving its content into dynamorio.github.io.

[johnfxgalea]

@derekbruening What are the cons of moving everything to Doxygen? Seems to be the easiest option to integrate everything, no?

[derekbruening]

@derekbruening What are the cons of moving everything to Doxygen? Seems to be the easiest option to integrate everything, no?

Yes, it may well be the best option. Given that we will always have some doxygen output for our function and data structure interfaces, we'll always have to deal with integrating it somehow. I also ended up using doxygen's javascript-based menu system for the web site, because it has the most pages and a ready-made system.

The only cons IMHO are:

• We have less control over high-level layout: doxygen puts all prose pages as peers of API docs, while we might want a layer on top of that as we have now. That may be harder to accomplish with post-doxygen editing as we might have to rip out menus instead of just adding/tweaking, but still doable.
• It feels a little uglier using the doxygen markup inside a C comment for large amounts of prose versus using jekyll's kramdown markup: but not a major issue.
• We have less direct control over some html output details that doxygen controls: but we can auto-edit the files in build post-processing (already doing that for some things) and tweak the web site style files to override doxygen and get it looking how we want.

[derekbruening]

Looks like doxygen 1.8.0+ has new support for markdown that looks more like the wiki or jekyll kramdown .md: https://www.doxygen.nl/manual/markdown.html

GA CI is using 1.8.11 so we could switch to this (instead of raw html <a href=, <em>, etc.); it shoul dmake it easier to move wiki pages to doxygen.

[derekbruening]

I played with moving some wiki pages into doxygen with the new markdown and it worked pretty well. I plan to go ahead and do that for all the wiki pages.

[AssadHashmi]

New integrated website looks really good!
Improved layout and navigation features 👍