Standard documentation format for Node.js

[Qix-]

cc @sindresorhus and @dthree, from dthree/wat@419654c.

---

The javascript community needs a standard, authoritative way to document libraries and modules. Markdown documentation is great, and it works, but actually using it other than reading is nearly impossible - nobody does it the same.

Preferably something that can be parsed, used, transcribed (to Markdown, JSON, HTML, etc.) simply because it's a structured documentation format.

It should also be human-readable by itself (similar to Markdown).

---

Discuss!

[Qix-]

JSON will be awful for this. If we write a DSL, I'd be happy to write the PEG.js parser for it (I love writing PEG parsers).

[sindresorhus]

Can you include something about what the other programming communities do? Go, Python, Ruby, etc.

[Qix-]

@sindresorhus I don't know what any of those specific communities do to be quite honest. I know that Java people use Javadocs (a real, parsable, built-in thing) and usually convert them to documentation using some stock tools or a tool like Doxygen.

Native people usually use manpages, or Doxygen as well. Same with PHP - usually doxygen with PHPDocs.

[iamstarkov]

we as community have a JSDoc for a while, if we are talking about having documentation in the code.

[Qix-]

@iamstarkov Sure, viable enough, and I know we've had JSDoc. My personal feelings towards it have always been a bit "meh" though. @sindresorhus why don't more modules use JSDoc?

By the way, WebStorm and other IDEs that work with javascript support JSDoc by default, offering helpful hints and such.

[RangerMauve]

I personally don't like JSDoc because some of the modules I make are too dynamic to really work well with it. I prefer just documenting apis with small examples in markdown where I can show how things should be used.

[dthree]

Perhaps a smart markdown parser that knows the general patterns people use in writing up their apps. It won't be perfect, what I was looking for was not a complete docset generator. JSDoc is that, so if it's not adopted, creating another one isn't going to get them more adopted. I think the thing inherently wrong with JSDoc is that it's too impersonal IMHO.

I thinking parsing markdown is important as what you see in the README are the salient points that the author wanted you to see. It has the human touch that JSDoc will never have.

READMEs are also probably the most maintained documentation a library has.

[RangerMauve]

In terms of markdown, there's DaSpec which might be similar to what you're looking for.

[dthree]

Thanks. Checking it out.

[RangerMauve]

Also, for code examples there's evalmd

[dthree]

evalmd is interesting - not sure how I'd use it, but definitely an interesting concept.

[RangerMauve]

Yeah, I was thinking it'd be cool for unit tests where you can have a description of what the test is doing and why it's there, and then having it get evaluated.

[dthree]

Yeah I can see that. I'm going to take a stab at a smart markdown parser. DaSpec looks pretty promising, but I have to get my feet wet.

[sindresorhus]

Might want to look into @wooorm's mdast parser.

[dthree]

Thanks

[wooorm]

Thanks @sindresorhus for the ping.

A few weeks back there was some movement on a related topic, originating in the feross/standard repo by maxogden. It’s called standard-readme. The project is a bit stale currently. But there is useful stuff in the issues (especially, related to API syntax).

Also: mdast reads markdown very well, and it’s relatively easy to convert to something else due to its syntax tree!

[wooorm]

P.S. I’m not sure if we’re talking about documentation in markdown in comments in JS, or documentation in markdown?

[dthree]

@wooorm we're taking about the latter - parsing readme / api / wiki markdown for general documentation.

mdast looks really good!

[ruyadorno]

relevant to the discussion, back from 2011: http://blog.millermedeiros.com/inline-docs/

unfortunately his mdoc documentation generator never really took off, but it has a really nice interface once converted to html, you can see an example of a very simple generated doc here

[AdrienFromToulouse]

Hello all, sorry for the may be stupid question but why woudl you like to use markdown?

From what I know npm takes the README from github as the documentation of the pkg.
That is why the documentations could sound unconsistent.

By forcing to comment the code itself with a jsdoc style syntax and then we use the tool dgeni done by the Angular team to generate the templates, we might "solve" the issue.

We can tweak it to suit the specific needs of nodejs if needed.

So after each publish to npm the doc is regenerated with a gulp task.
The module could not be pushed if any errors is returned from the doc build.

Am I on the good track? Does it sounds relevant to you?

[dthree]

@wooorm @Qix- @sindresorhus

I'm actively working on a smart readme / markdown parser that builds API docs using mdast. wooorm was talking about the standard-readme project above, which looks promising but is unfortunately stagnating.

Since I am doing a ton of work parsing readmes, do you guys think it would be a good idea for me to create a sort of mdlint project that sets a standard for a readme in terms of information, grammar, code linting as well as API doc standards? For example,

1. It could use Alex for proper word usage,
2. A general grammar language lib for general mis-spellings,
3. An existing project such as this for linting code,
4. The work I am doing to standardize API docs,
5. Validate test badges, etc.
6. General agreed upon standards, based on best practices learned over time; such as an introduction, table of contents and license section.
7. (ideas accepted)...

I think this would be cool, maybe we could team up if it interests any of you. Providing an actual lint library could perhaps push community agreement over the edge to provide standardized readmes for the first time. which could then be a stable and maintainable source of documentation.

With this, a readme could get a badge or recognition as being linted or standard md compatible, and wat could automatically generate high quality API docs in addition to local access to the readme with no human intervention.

What do you guys think?

[sindresorhus]

create a sort of mdlint project that sets a standard for a readme

Have you seen mdast-lint?

What do you guys think?

Sounds awesome, but it's going to be a lot of work. I think we should first focus on standardizing the readme API docs. That is what matters for parsing. The rest of the readme is just for consistency and quality, and will be a lot harder to standardize.

Obligatory xkcd:

[dthree]

Haha - amazing! For sure. Yeah, I was thinking (eventually, not now) just combining all of the work that's been done in to a single standard, it doesn't have to be a new one. I.E. Alex, mdast-lint, the most popular api standard, etc. mdast-lint looks like the perfect starting point. And I agree with you: standardizing the layout of a readme isn't necessarily necessary. I was more referring to the types of things mdast-lint handles, things like ## Header instead of ##Header because some md interpreters like npm don't recognize the latter, etc. etc.

---

But yes, less talking for me and more coding. I'll get back with you all when I have something workable to show.

[Qix-]

I was waiting for that XKCD to show up.

[stevemao]

https://github.com/verbose/verb
https://github.com/zeke/package-json-to-readme

[sindresorhus]

@stevemao Instead of just posting random links, can you elaborate on what they are and why you posted them?

[stevemao]

They both generate a readme for you from package.json. With verb you can even define a template. According to the title of this thread you wanna standardise docs so I guess the opinionated, uncustomisable package-json-to-readme might be ideal to achieve this.

[dthree]

Mm. package-json-to-readme really doesn't do much other than boilerplate, and it's not totally real either. I've never seen someone include dependencies and dev-dependencies on their readme. Ironically, it doesn't even fully document what it does with its own auto-generated readme, e.g. where the usage data came from.

Verb basically seems to aggregate data from other markdown files, with a few fancy template options. Doesn't really solve the question at hand.

[dthree]

@sindresorhus @Qix- I think we need to discuss:

1. Why jsdoc doesn't fly, and why can't it become the standard doc format for Node,
2. What would need to be different about it in order to be acceptable.

In other words, we need an ideal scene of what we're trying to accomplish.

[Qix-]

@RangerMauve My earlier post explains why. Sifting through code with Node projects in particular is rough.

---

Keep in mind this is just an idea. My word definitely isn't law ;)

[dthree]

@sindresorhus what do you think about this?

[RangerMauve]

@Qix- So it'd just be easier for a tool to find? Why not just have a file format for declaring the API? Maybe JSON, and then you could have something like api.json in the project root as well as the package. You could have all the structured fun of json to have the format you want for the api, have it be easily located, and not clutter up your markdown with a bunch of comments that nobody will bother to read.

[dthree]

"foo": {
  "bar": {
    "type": "method"
    "args": [
      "my": {
        "optional": false,
        description: "Some argument."
      },
      "preference": {
        "optional": true,
        description: "Some arg description."
      },
      "for": {
        "optional": true,
        description: "Some arg description."
      },
      "optional": {
        "optional": true,
        description: "Some arg description."
      },
    ],
    examples: [
      {
        header: "Using this to do so and so:",
        code: "err... i dont know how this would work in JSON"
      }
    ]
  }
} 

A little bloated...

[dthree]

@RangerMauve of course no one is going to read the .md comments - the point is to use a tool to then build the docs.

[Qix-]

@RangerMauve agreed with @dthree - that's a trainwreck. Further, the comments aren't for people to read, they're for machines to read. :)

[dthree]

@Qix- I'm not really into a million <!-- comments --> either. Hmmm....

[RangerMauve]

Well, maybe JSON isn't the best. But I still think that having it in its own file would be better than cluttering up the source of the markdown. If it's not for people to read, and it's made for this specific tool, then why not just have a file that the tool looks for? Is there any benefit to anyone for having it inline over a separate file?

[dthree]

Maybe YAML?

[Qix-]

@RangerMauve @sindresorhus mentioned, and I agree, yet another "dotfile" type of thing in a repository is just making an already bad dotfile problem worse.

[RangerMauve]

I personally still like the idea of having the headers and content of the markdown specify the API, but I agree that it'd be hard to get people to agree on a way to structure it since they like their own style.

@Qix- I don't think that inlining parts of what's essentially a dotfile into a readme is really getting rid of it. It just makes the readme less of a readme.

[dthree]

I still don't know why we can't do it in markdown, with just a little standard.

---

calc.add(a, b[, additional...]):sum

Returns the sum of two or more numbers.

calc.add(2, 3, 4); // 9

---

It can be another .md file than the readme, such as api.md, and then can be processed by a machine, but it also can be viewed by humans, and is ridiculously easy to write.

[RangerMauve]

I think it could happen if enough people signed onto the idea. Standard style and xo seem to be getting traction and they're about having a particular style for code, if enough people start following this I'm sure it's not a stretch to say that there could be standard styles for markdown readmes.

[dthree]

For example, using wat, I auto-generated all of d3's docs in about 10 seconds, creating a perfect API of about 600 methods and properites. My code parsed them without error. and this worked simply because d3's docs are clean and uniform. It can be done.

[dthree]

@RangerMauve I agree. It's all just agreement, really. I think it's practical enough.

[RangerMauve]

Maybe it'd make sense to go over some of the popular styles out there and see what one are most popular and easiest to parse?

[dthree]

Yeah. This is what standard-readme was doing, and then the discussion just died. it just needs to be pushed through.

[RangerMauve]

Maybe there just really needs to be at least a simple implementation out there for people to start playing with and then have the community improve it as they go. If we just discuss hypotheticals, I think this discussion will die out in a similar way. I don't think standard because what it is right now because it was discussed for a long time, I think people just started using it and then got around to any finer details.

[dthree]

Well then why don't declare something, and just sound ridiculously certain about it, and then adopt it on as many things as we can, and then list out what we adopted it on, and then start telling people about it. @RangerMauve I agree that many methods work - there isn't a perfect method, so we just need to be definitive. And then if 20 ppl adopt it, and someone has an argument to improve something, they can bring it up.

[RangerMauve]

I personally don't think I'd have time to work on such a tool, but the moment someone makes it, I'll adopt it in my projects. It might also be useful to make a not to have people using the standard add a link to the tool so that people reading the readme will gain awareness of it.

[dthree]

Yeah - like this:

[dthree]

By the time I'm done with wat, I'll have readme API parsing pretty well taped, so maybe the parsing can become its own module that can be used by a readme API linter.

[joakimbeng]

👍 Would really like something in this direction!

I created pj2md which I use to generate readme's for my modules, which uses get-api to get some API information from a module.

[dthree]

Nice @joakimbeng . You should include some links examples of what a few popular modules' markdown would look like if fully generated through pj2md.

[nektro]

If a custom style could be made, I think it would nicely fitting we made a format that could be compiled and run through https://github.com/tabatkins/bikeshed

[Richienb]

I think the best seamless solution would be to generate markdown from typescript typings or fallback to jsdoc comments.

[SiddharthShyniben]

One problem would be that there are some libraries which are too complex to be documented by comments. So my ideas are:

• Let the creators document the dynamic stuff by themselves
• Let the creators add certain comments to the source code which creates an API Reference (Think JSDoc). We can't call an API Reference "documentation", so the first point. And writing an API Reference is terribly hard, so this "tool" would help.

[SiddharthShyniben]

One more thing to consider would be: How would we show the docs?

• As HTML? Goodbye markdown docs
• As Markdown? We would have some trouble because not every platform shows Markdown in the same way. So we would have to create a whole new Markdown spec.

---

We could do both

[SiddharthShyniben]

I'm working on a syntax for a new "doc" (TODO: rename) file type (which will be read by a cli etc. and converted to docs), and here is my idea for how the docs would be made:

• Developer writes code,
• Developer creates the doc file,
  • Dev uses CLI which parses the module.exports and generates boilerplate doc files
    • Optional comments can be added in the source to help this CLI choose methods, etc. (I think comments are preserved when we stringify a function)
  • Dev creates file by hand
• Dev fills the doc file using the new syntax
• Dev uses CLI to create markdown docs (or an html project with the docs in the /docs folder to help github pages hosting)

So basically we would have some doc files (.npdoc extension if you like) which would be parsed and rendered.

This is just a really basic incomplete version. If everyone likes the general idea we can elaborate

---

Waiting for everybody's opinion, mentioning: @sindresorhus @ruyadorno @Qix- @RangerMauve @dthree