How should the structure of the technical documentation be organized?

[dorrab]

The omr/doc/ directory currently contains the coding standard as well as some technical documentation about the compiler. There are some efforts underway to add to this documentation, such as:

#339
#340

Based on the earlier discussions on the mailing list, it seems the general consensus is to have the docs and code in the same repository, so we should continue to contribute under this doc directory. But I suggest we figure out how it should be organized.

There are a few options that we can consider:

• https://readthedocs.org/
• https://www.gitbook.com/
• other suggestions?

[mgaudet]

I have a small amount of experience with ReadTheDocs. I've been quite impressed with what I've seen.

• Big projects are using it, including things like Python.
• It's reasonably extensible. We could eventually hook up the Breathe plugin, and integrate our Doxygen Documentation.
• Commit Hook builds means the documentations is up to date, but the tool also supports doc versioning.

[SueChaplain]

This is a useful resource for comparing documentation solutions: https://www.staticgen.com/
Popular, stable, and here for the future would be one good criteria.
Quantity of doc and type of documentation (concept/task/reference) would be another. Some solutions lean more towards small and simple rather than a ton of doc.
Do you already use Doxygen for API documentation? Currently it looks like ReadTheDocs does not explicitly support it.

[mgaudet]

We do use Doxygen comments -- not as much as we would like, but we're trying :D

Whatever solution we have should also support large sections of prose to discussion higher level concepts; We've already checked some Markdown files into the doc directory, and I definitely want to write more.

IMO, some navigation support should also be provided. We have some experience using Doxygen's pages, and personally, I didn't love it.

[fjeremic]

How does everyone feel about most of the documentation / design living alongside the code and getting parsed out by tools for external display? For example all of the design documentation could be Doxygen'd to live alongside the relevant code and whatever tool can then be used to extract, format, and present in web form.

[SueChaplain]

I assumed that the Doxygen docs would cover mostly API type reference information - I've yet to look at it's capabilities. If it doesn't, then to cover some higher level concept topics, which @mgaudet mentioned, we might have to look at what else could be used. If the concept topics are not too large, we might be able to include those files under the existing gh-pages soln.

[mgaudet]

Regarding @fjeremic's question about docs: I desperately want all documentation to end up in tree, revision controlled with the code (ruling out an orphan branch solution like gh-pages).

I also think there's a home for prose description both inside the code (doxygen comments, and doxygen \page etc), and in stand-alone documents (docs/*.md).

[SueChaplain]

@mgaudet - It is possible to have docs in the master tree, run a doc build and push the results to a gh-pages branch. That way you can ensure version control. I had this working for Gitbook, where a Gitbook-cli build generates a /_book directory in master, which you can then git push to an orphan gh-pages branch.

[mgaudet]

I'd like to resuscitate this conversation. IMO, we should choose a solution that has low overhead on boarding and get started on forming a coherent documentation set, even if it's not where we land finally for documentation. Some structured docs are better than none.

I stand by my original pitch for ReadTheDocs, as I think it'll involve no more than getting a committer to get in touch with Eclipse, and having them turn it on for us. If we do that, we can then start structuring our Docs and building them out in a reasonably easy to use system.

[SueChaplain]

Can you help me understand what it is you need to produce and how much of that exists vs needs creating (high level overview / detailed documentation / API style reference). That will help this discussion.

[mgaudet]

The background is that we're starting to get the barest nibbles of community interest, but invariably, we're unable to point at a place that has answers to questions. We talked a bit about this yesterday in one of our local weekly meetings. What were were talking about is that OMR needs to have documents on things like

What:

• What is the OMR project? (*)
• What are the components of the OMR project (*)
• What are the concepts inside / organization of each component

Why:

• Why should someone consider using the OMR components?

How:

• How do you integrate the OMR components?
• How do you modify the OMR components
• How do you
  • ... add an optimization
  • ... modify a garbage collection policy
  • ...

The elements I've marked with (*) are things that I think would do well on the existing OMR site. Other points however I think would be best elsewhere: In particular, I think they have to be kept in the same repo as the code as we aren't interface-stable enough to separate them out,

The value I see from something like ReadTheDocs is a tightly integrated workflow the ensures our docs look consistent, and provide us a structure in which to write.

IMO, Doxygen style code level documentation will also be important, but for now, is secondary to prose.

[SueChaplain]

Thanks for the explanation.
I would suggest that you need some of the "Why" on the website too, even if the bulk of the documentation covers it in more detail.

Regarding the main part of the documentation that will live alongside the code. There looks to be a lot of material to curate unless some of that doc already exists in other repositories. If I look under the /doc directory there are currently a few pieces on the JIT. I guess these fit into the section "What are the concepts inside / organization of each component".

Do you have a more detailed structure in mind for the main documentation set that could be organised into a readthedocs or similar build solution?
Do you have a feel for how quickly some of the remaining content can be pulled together?

[mgaudet]

I think this is a bit of a 'if you build it they will come' situation.

By having a good documentation tool setup, we'll be able to setup organization and documentation will then come.

Having said that, I really appreciate the structure of the creatonne docs.

[SueChaplain]

I agree that the creatonne docs look nice. Doc build solutions other than ReadtheDocs (Sphinx) can produce similar results (E.g. gitbook, mkdocs) ... we are testing a few at the moment.

Sphinx primarily supports source files in Restructured text (ReST), although you can add reCommonMark for basic markdown support (note there are limitations with this, such as no support for tables). How does the community feel about writing doc in ReST format rather than markdown? Do you think this will be a barrier to doc contribution? The formats are similar in that they are relatively simple, but ReST does add some extra complexity.

I'm also keen to understand the requirement to produce and/or integrate API reference doc. Doxygen is mentioned in an earlier comment as a potential tool for extracting it. What is the status of the OMR API doc?

[mgaudet]

While I would personally be willing to write in reST, it do think that we should support people writing in Markdown. It's incredibly pervasive, and as such smooths contribution.

IMO, API reference documentation can be left as a secondary priority for now. We encourage people to write comments in Doxygen format, but are still a ways away from having cohesive doxygen documentation.

[0xdaryl]

The community discussed this at the June 4, 2020 OMR Architecture Meeting. The consensus was that we will use Markdown format for now while the project continues to accumulate a corpus of documentation. The format of the documentation does not appear to be the main barrier to producing more of it, so choosing a form that is simple and integrated with the code seems appropriate at this time. This issue will be closed.