Support use of autodoc with myst

[asmeurer]

Edit:

The implementation of eval-rst has now been closed (see this documentation section)

However, a new issue has also been opened, specifically for using Markdown in docstrings: #228

---

I can't figure out how to use autodoc with myst.

With recommonmark, I have something like

```eval_rst
.. autofunction:: functionname
```

or

```eval_rst
.. autoclass:: classname
   :members:
```

Based on on the myst docs. I can't figure out what to use to make it work in myst.

(by the way, the example at https://myst-parser.readthedocs.io/en/latest/using/syntax.html#extended-block-tokens, doesn't seem to agree with the example at https://myst-parser.readthedocs.io/en/latest/using/syntax.html#syntax-directives).

After many attempts at the syntax, I finally got something that at least produced some output. But all it seems to do is inline raw rst instead of rendering it.

[chrisjsewell]

Hey @asmeurer well in principle it would be the same as any other directive:

```{autofunction} functionname
```

```{autoclass} classname
:members:
```

These directives are a bit tricky though because (again as with every directive) myst should parse the content as Markdown NOT rST. The complications are then:

1. You will have written your docstrings as rST, but myst expects them as Markdown.
2. If you wanted to write your docstrings in MyST Markdown, there is currently no syntax equivalent to field lists, if you are using rST style parameter naming, i.e.

def func(arg1):
    """
    :param arg1: description
    :type arg1: type description
    :return: return description
    :rtype: the return type description
    """

I think it would be nice to add the syntax for (2), such that you could write docstrings in Markdown.
But, for existing code bases, I concede that its impractical that people should need to rewrite all their docstrings. In which case we should offer a similar "special case" eval_rst directive that just parses the contents as rST, i.e.

```{eval_rst}
.. autoclass:: classname
   :members:
```

the example at ..., doesn't seem to agree

Could you elaborate on this

[choldgraf]

I'm +1 on both of those ideas - I opened up #164 to track the eval_rst feature

[asmeurer]

I don't actually care about parameter lists, although I might want support for napoleon headers at some point.

But I think you've misunderstood the bug report. This is what I get from

### ndindex

```{autofunction} ndindex.ndindex
```

### Integer

```{autoclass} ndindex.Integer
:members:
```

Compared to what I get with recommonmark and eval_rst: https://quansight.github.io/ndindex/api.html#ndindex

(the docstrings are currently using rst markup, but that doesn't seem to be an issue yet here)

Could you elaborate on this

Like the one example doesn't show that you can put stuff after the {} part (at first I tried putting the function name on a separate line, because that's what the example seems to indicate). And the other example seems to indicate that you need at --- before and after the options list. This all goes back to #146. The docs for myst are often not written in a way that is clear to someone who doesn't already know how myst works under the hood.

By the way, this issue is currently a major blocker for me, since I have API docs that require autodoc, but also a Markdown document that can only be processed with myst. Is it possible to make one file be processed with recommonmark or even just regular Sphinx as a workaround?

[choldgraf]

this issue is currently a major blocker for me, since I have API docs that require autodoc

you can always write these pages in rST...hopefully your autodoc pages don't require too much other formatting so it wouldn't be a big deal?

[asmeurer]

you can always write these pages in rST...

Ah so if you just have an .rst file that works?

hopefully your autodoc pages don't require too much other formatting so it wouldn't be a big deal?

If autodoc worked in myst and I had to reformat my docstrings in Markdown, that would (currently) be fine, as I'm not using any napoleon/numpydoc specific stuff. But the issue for me is that it isn't working at all.

[chrisjsewell]

Is it possible to make one file be processed with recommonmark or even just regular Sphinx as a workaround?

Yes to clarify, if your file ends in .rst then it will be processed with the rST parser.

[asmeurer]

Right, just tested that and it is working. So this is no longer a hard blocker for my project.

[chrisjsewell]

Like the one example doesn't show that you can put stuff after the {} part

The simple example in the table doesn't

But then it does point you towards the actual section on directives, which starts with this table:

And the other example seems to indicate that you need at --- before and after the options list

If you read through, it then explains later:

How would you suggest improving this? Because TBH I don't see how this could be clearer, or why it requires any knowledge of how MyST works?

[asmeurer]

The simple example in the table doesn't
But then it does point you towards the actual section on directives, which starts with this table:

That's exactly my point. The one thing differs from the other. And it doesn't really state what "content" represents, so I assumed it should be

```autofunction
functionname
```

I suppose it would also help if the table showed the actual rendering of each of the examples.

Because TBH I don't see how this could be clearer, or why it requires any knowledge of how MyST works?

What are "directive", ""option", "value", "arguments", or "content"? If you are an expert at RST you might know this, but anyone else won't.

[choldgraf]

Ah - I think @asmeurer that we do have an assumption that readers already have a decent understanding of rST and Sphinx (our assumption is that MyST users would only be fairly well-versed in rST before considering MyST). Perhaps the syntax.md page (which is fairly overcomplete) could be preceded by a more introductory page that explains roles and directives specifically, and some stuff that is specific to Sphinx?

[choldgraf]

addressed some of this in #165, would love comments there 👍

[sbliven]

I'd be really interested in seeing option 2 implemented (markdown docs).

[sbliven]

I saw this solution using the autodoc-process-docstring hook. It gives access to the docs prior to running autodoc, and the solution uses m2r to rewrite markdown to reST for sphinx to process. I was hoping to do the same using MyST for a consistent markdown flavor across pages and docs. However, I don't see a reST writer in MyST, docutils, or sphinx. Am I missing something, or is this a dead end?

[chrisjsewell]

thanks for the link btw @sbliven I think that (eventually) we will be able to do something along the lines of (2), maybe using them hooks. But re-writing to rST would not be an ideal route IMO, since it kinda goes against the "aim" of this package; to translate directly from Markdown to AST

[sbliven]

@chrisjsewell Should that be an autodoc feature request? A hook that replaces the AST generation step?

[chrisjsewell]

Should that be an autodoc feature request? A hook that replaces the AST generation step?

Yep quite possibly. You may know better than me, since I haven't had time to look into this in too much detail. But it appears that autodoc is quite strongly coupled to rST, and so anything to loosen this coupling would be welcome

[sbliven]

I asked on the sphinx repo, and it seems like writing docstrings in MyST will to take a large amount of refactoring in autodoc.

[choldgraf]

I think the shortest path here is to support something like eval_rst in MyST...I bet that'll happen before major refactorings in autodoc

[sbliven]

Yeah, eval_rst seems like a good start, and will probably be handy for other situations in the future. It seems like we are quite far from my dream of markdown docstrings.

[chrisjsewell]

Closed in v0.12.2 😄 See https://myst-parser.readthedocs.io/en/latest/using/howto.html#use-sphinx-ext-autodoc-in-markdown-files

[asmeurer]

There's also the question of writing the docstrings themselves in markdown.

[chrisjsewell]

Yep opened a separate issue for that 👍 : #228

[asmeurer]

OK. I did point this issue to a few people for the general support of autodoc, so you may want to add a note about that to the top of the issue here.

[chrisjsewell]

so you may want to add a note about that to the top of the issue here.

Done 👍

[dhermes]

@chrisjsewell RE:

there is currently no syntax equivalent to field lists, if you are using rST style parameter naming, i.e.

Are there plans to support field lists? I'm happy to file a placeholder issue if none exists already (I looked and haven't found one yet).

[chrisjsewell]

#153. However, this is not how auto doc would be natively supported.
This requires sphinx-doc/sphinx#8018, whereby auto doc should be rewritten to not write rST then send it to the parser, but instead directly construct the AST nodes

[dhermes]

@chrisjsewell I'm actually interested in it from another direction: I want to revisit the Go domain and just target MyST as an intermediate representation. I'm toying around with the right way to represent function / method signatures and field lists seems like a good choice.

[chrisjsewell]

Ok cool, do you want to leave a note on #153, about your exact use case and maybe some example pieces of text.

[chrisjsewell]

Note there is also now https://myst-parser.readthedocs.io/en/latest/using/syntax-optional.html#definition-lists which could be considered