Linux friendly manpage generation

[kravemir]

I'm using picocli for my project, and I would like to create manual page for it.

Is there any recommended approach to generate a manual page for standard linux's command man ?

Issue type, depending on possibilities: question, feature suggestion, or article/documentation improvement request.

[remkop]

That’s a great idea!
We can add a DocGen class that generates documentation from a CommandSpec in various formats. This should probably live in a separate package because I can see this grow in the future. The first format could be manpage.

[remkop]

See also #299, which suggests using pandoc for this. (Update: looks like pandoc uses the GPL license so we cannot use it.)

Do you feel like providing a PR for this?

[kravemir]

See also #299, which suggests using pandoc for this. (Update: looks like pandoc uses the GPL license so we cannot use it.)

Actually, if we use pandoc as a command-line application, and launch it through POSIX, then we wouldn't violate GPL license. Correct me, if I'm wrong.

Do you feel like providing a PR for this?

Though, my free time is quite limited, but knowing this feature is welcomed and would be accepted, and also I want to use it, I would like to provide a PR for this.

We can add a DocGen class that generates documentation from a CommandSpec in various formats. This should probably live in a separate package because I can see this grow in the future. The first format could be manpage.

Could you provide whole high level concept, or solution design, for this feature? Ie.

• add project dependency group:artifactId,
• create class DocGen in package picocli.x.y.z,
  • constructor (or create generate method with) argument CommandSpec,
• extend existing class CommandLine:
  • add method generateDocs to generate documentation for the commandline,
• ...

[remkop]

All good points.

• a new artifact is a good idea. I’m thinking info.picocli:picocli-docgen. Let’s make a new module for this in the picocli project.
• package: I’m thinking picocli.docgen
• class DocGen (no need to extend anything)
• one idea is to have a static method generateDocs(String contentType, CommandSpec spec) : String.

The idea/hope is that content type (mime type) would make it extensible: ”text/html” for HTML, etc. Not sure if this is a good idea. Do markdown and man pages have a standard mime type, for example? (I couldn’t find them.)

Also not sure what the return type of the method should be. Either return a String, or void and print to standard out? (The latter would allow binary output formats.)

Thoughts?

[kravemir]

a new artifact is a good idea. I’m thinking info.picocli:picocli-docgen. Let’s make a new module for this in the picocli project.

I thought of new dependency for picocli, which would be used to generate outputs.

But, documentation generation isn't needed for application/tool at runtime. Documentation is generated during build. Ie. once, then all docgen code isn't needed anymore. And, putting it into picocli core would just make runtime dependencies unnecessarily bigger.

And, so you think of whole new artifact, and/or github project? Or, what's yours idea about new artifact picocli-docgen.

one idea is to have a static method generateDocs(String contentType, CommandSpec spec) : String.

The idea/hope is that content type (mime type) would make it extensible: ”text/html” for HTML, etc. Not sure if this is a good idea. Do markdown and man pages have a standard mime type, for example? (I couldn’t find them.)

Also not sure what the return type of the method should be. Either return a String, or void and print to standard out? (The latter would allow binary output formats.)

From my point of view, the docgen process is defined by three parts:

• input: probably only CommandSpec,
• output: can be returned as String, argument of type Appendable to write output to, argument of type OutputStream, or also Consumer<???> which would write output to something writable,
• generationOptions: defines what would generated output look like, there can be contentType, style, verbosity, ...

I dislike idea to use standard output for generated outputs.. It just complicates things. I like to think in functional way, where methods have got input, and generate output. Then, such methods can be used, and composed, in various ways without limitations.

[kravemir]

Also, output type can be DocBook SGML, which can be then converted to manpage using docbook-to-man

[remkop]

And, so you think of whole new artifact, and/or github project? Or, what's yours idea about new artifact picocli-docgen.

I was thinking to create a module picocli-docgen in the picocli project. The build would create a new picocli-docgen-x.x.x.jar in addition to the picocli-y.y.y.jar. No need for a new GitHub project. The picocli-docgen module may have external dependencies in addition to depending on picocli.

About the DocGen API, what you’re saying makes a lot of sense. We can evolve this over time, I would be surprised if we got it perfectly right from the beginning. :-)

[remkop]

About mine type for man pages:

According to
http://www.engr.iupui.edu/~dskim/tutorials/misc/mime_type.htm,

Files with extension .man have mime type application/x-troff-man, meaning Troff with MAN macros.

Potentially related:

Currently, troff documents are listed under the nonstandard application/x-troff and text/x-troff. However, the IANA approved the standard text/troff MIME type on 2005-06-10.

according to https://bugs.freedesktop.org/show_bug.cgi?id=5072

[remkop]

I will rename the existing examples module to picocli-examples.

If you want, I can create a docgen branch and add a picocli-docgen module in that branch. That may make it easy to work together on this.

[remkop]

Perhaps doxygen is a good tool to use for this ticket.

Alternatively we can just generate raw man groff macros. This page and this page show some examples. This article also gives a nice overview. All in all it looks doable.

@kravemir Are you still interested in this? Should I go ahead and create the picocli-docgen module?

[kravemir]

@remkop: yes, I'm interested in this. It's on my todo-list, but there are still some task before this. One was to create picocli debian package and get it uploaded to debian official repositories.

I saw readme, and thanks for attribution :-) Though, if you want to refer me on debian, you can use https://udd.debian.org/dmd/?kravec.miroslav%40gmail.com or https://qa.debian.org/developer.php?login=kravec.miroslav%40gmail.com.

And, regarding this feature. I thought about first implementing some solution directly for my tool, then make it as reusable solution, and create PR for picocli containing the implementation.

[remkop]

Understood. Thank you again for creating the debian package! I updated the links on the README page, let me know if these links need further correction.

About this ticket, glad to hear you are still interested.
Feel free to create a picocli-docgen module (#498) as part of the PR if this module does not exist by then.

[hupfdule]

Also, output type can be DocBook SGML, which can be then converted to manpage using docbook-to-man

I think DocBook is a very flexible way to support multiple output formats.
You should even consider generating an AsciiDoc document, which then can be converted to manpages directly (or HTML or PDF) or to DocBook which in turn can be converted to other formats.

There exists the https://github.com/asciidoctor/asciidoctorj project to parse or convert AsciiDoc documents programmatically.

AsciiDoc is very easy to write for humans (also and very readable), however I don't know how easy it is for a machine to generate AsciiDoc or whether AsciidoctorJ provides an API to create AsciiDoc documents.

[remkop]

I love AsciiDoc: all picocli documentation is written in AsciiDoc. :-)

Can you give details on how a text in AsciiDoc format can be transformed to man page format?

[hupfdule]

A description on how to do it via commandline is given in
https://asciidoctor.org/docs/user-manual/#man-pages.

As far as I know, AsciidoctorJ also supports this.

There are some restrictions on the format of the AsciiDoc document, but that shouldn't be a problem.

[remkop]

An initial version of picocli.codegen.docgen.manpage.ManPageGenerator has landed in the picocli-codegen module in master.

You can test by checking out the latest master and building with:

gradlew clean publishToMavenLocal

That should publish picocli-4.2.0-SNAPSHOT to your local .m2 Maven cache. You can then try this in a project that uses the info.picocli:picocli:4.2.0-SNAPSHOT dependency.

Here is an example Gradle build.gradle snippet:

dependencies {
    compile "info.picocli:picocli:4.2.0-SNAPSHOT"
    annotationProcessor "info.picocli:picocli-codegen:4.2.0-SNAPSHOT"
}

mainClassName = "my.pkg.MyCommand"

task generateManpageAsciiDoc(type: JavaExec) {
    dependsOn(classes)
    group = "Documentation"
    description = "Generate AsciiDoc manpage"
    classpath(configurations.compile, configurations.annotationProcessor, sourceSets.main.runtimeClasspath)
    main 'picocli.codegen.docgen.manpage.ManPageGenerator'
    args mainClassName, "--outdir=${project.buildDir}/generated/docs", "-v" //, "--template-dir=src/docs/mantemplates"
}

apply plugin: 'org.asciidoctor.convert'
asciidoctor {
    dependsOn(generateManpageAsciiDoc)
    sourceDir = file("${project.buildDir}/generated/docs")
    outputDir = file("${project.buildDir}/docs")
    logDocuments = true
    backends 'manpage'
}

The generateManpageAsciiDoc task generates <cmd>.adoc files with doctype manpage in build/generated/docs for each command and subcommand.

The asciidoctor task converts the generated <cmd>.adoc files in build/generated/docs to <cmd>.1 manpage files in build/docs/manpage/. You could then use the Gradle distribution plugin to include the generated manpage files in the distribution archive:

apply plugin: "distribution"
distributions {
    main {
        contents {
            from ("${project.buildDir}/docs/manpage") {
                into('docs')
            }
        }
    }
}

Customization

The generated man page is very similar to the online help generated by the command itself when a user specifies the --help option. You may want to add more details or extra sections to the man page. This can be achieved by leveraging AsciiDoctor's include mechanism.

To create customized man pages, first point the sourceDir for the asciidoctor task to a different directory than the directory where the AsciiDoc files with doctype manpage are generated.

Let's call this new directory src/docs/manpage. In this directory, we keep a manually maintained xxx.adoc file for each generated AsciiDoc file with doctype manpage. Now we can customize the resulting manpage by editing the manually maintained files in src/docs/manpage.

In its simplest form we can include the full generated page with the generated-manpage tag. This allows us to add some sections to the end of the page. For example:

// src/docs/manpage/xxx.adoc
:includedir: ../../../build/generated/docs
include::{includedir}/xxx.adoc[tag=generated-manpage]

== Authors
My, myself and I.

It is also possible to include the picocli-generated sections individually, so that the generated sections can be customized with additional text that follows the generated text for the section. For example:

// src/docs/manpage/yyy.adoc
:includedir: ../../../build/generated/docs
include::{includedir}/yyy.adoc[tag=generated-man-header]

include::{includedir}/yyy.adoc[tag=generated-man-name-section]

This is a very good tool.

include::{includedir}/yyy.adoc[tag=generated-man-synopsis]

*mycommand* [-hVv] [a=VERY] [--different=synopsis]

include::{includedir}/yyy.adoc[tag=generated-man-description]

Here are some additional description lines.

include::{includedir}/yyy.adoc[tag=generated-man-arguments]

include::{includedir}/yyy.adoc[tag=generated-man-options]

include::{includedir}/yyy.adoc[tag=generated-man-commands]

include::{includedir}/yyy.adoc[tag=generated-man-exit-status]

include::{includedir}/yyy.adoc[tag=generated-man-footer]

[%hardbreaks]
Here is an extra example:
abc def ghi jkl mno pq
abc def ghi jkl mno pq
abc def ghi jkl mno pq

== Authors
All of us.

== Copyright
Just hand it out, it's fine.

(These tag names may be changed before the picocli 4.2.0 release.)

Known issues:

• custom option sections (arggroup option sections) are still ignored
• embedded markup like @|bold hello|@ in descriptions and header/footer text is rendered as ANSI escape codes and not yet converted to AsciiDoc markup
• @<filename> is included in the synopsis but not in the ARGUMENTS section

Update: the above known issues are all resolved now in master.

Feedback welcome!

[remkop]

@kravemir, @hupfdule, All,

picocli 4.2.0 has been released, including this functionality. Enjoy!