Move sphinx and docutils to optional dependencies

[choldgraf]

If I'm not mistaken, the myst parser can be used independently of sphinx and docutils. Is that right? If so, perhaps we should move those dependencies into an extra install. This would make the parser more lightweight and potentially encourage others to use it in other ways. Wdyt @chrisjsewell ?

[chrisjsewell]

No, this is a sphinx extension, if they want to use just the parser, then that's what markdown-it-py is for.
You can add extra documentation about how to do the parsing with markdown-it-py if you want, i.e. what extensions and options to use

[choldgraf]

Maybe another way of phrasing this question - I think that the Python library could be useful for others outside of the Sphinx use case. Is there a way that we could offer this to others without requiring them to rewrite it from scratch with their own markdown it py config? If not, i agree with your assessment that documenting how to recreate a myst parser with markdown it py would be helpful

[chrisjsewell]

I think that the Python library could be useful for others outside of the Sphinx use case.

I disagree: this is literally all the code that would be made available:

from markdown_it import MarkdownIt
from mdit_py_plugins.dollarmath import dollarmath_plugin
from mdit_py_plugins.footnote import footnote_plugin
from mdit_py_plugins.front_matter import front_matter_plugin
from mdit_py_plugins.myst_blocks import myst_block_plugin
from mdit_py_plugins.myst_role import myst_role_plugin
md = (
        MarkdownIt("commonmark")
        .enable("table")
        .use(front_matter_plugin)
        .use(myst_block_plugin)
        .use(myst_role_plugin)
        .use(footnote_plugin)
        .disable("footnote_inline")
        .disable("footnote_tail")
        .use(dollarmath_plugin)  # potentially optional
    )

Also, sphinx was previously an extra and changed to a core requirement because it added extra complexity and the chance for broken dependency versions, for the core use-case as a sphinx extension,
i.e. this has already been litigated and decided

[chrisjsewell]

If you didn't want this to be the case, then you should have objected here: #170 (comment) 😜

[choldgraf]

Then I think we should document this pattern for others to follow, since the myst parser repo is also effectively the reference implementation this could be a helpful way for others to apply the myst spec in other contexts. I'll close this and re open another issue

[chrisjsewell]

👍

[astrojuanlu]

(Coming from here https://twitter.com/juanluisback/status/1379101793653776387)

Hi @chrisjsewell! I was surprised to see docutils 0.17 add experimental Markdown support using Recommonmark, so I found this thread on docutils-users https://sourceforge.net/p/docutils/mailman/docutils-users/thread/rkv4nb%24139g%241%40ciao.gmane.io/#msg37118232 (in which you participated I believe) and that's what sparked my original question to @choldgraf about the optional dependencies of myst-parser. It sounds like docutils could potentially use https://github.com/executablebooks/markdown-it-py for its Markdown support then?

[choldgraf]

Yep, i think that is the answer.

[chrisjsewell]

I was surprised to see docutils 0.17 add experimental Markdown support using Recommonmark

Grr you and me both, and I completely missed the response on the email thread.

Ah well that's a different proposition: I literally have myst_parser/docutils_renderer.py and myst_parser/sphinx_renderer.py as a sub-class, so that it is certainly possible to use myst-parser with "docutils only" functionality.

If I had known previously about such an intention to ship Markdown in docutils, I would certainly have considered splitting the "docutils only" aspects in to a separate "myst-docutils" package (and have that as a dependency here), which I think would probably be a better solution than just moving dependencies to extras.
Indeed, if there was an agreement in principle with the docutils guys to include something like myst-docutils in docutils I would look into this.

@choldgraf you've been promising to talk to the recommonmark guys about potentially combining our efforts (indeed myst-parser evolved from the recommonmark code), get on it! 😉

[astrojuanlu]

I now work on Read the Docs, so although it's still a bit bold to call me one of the "recommonmark guys", I'd love to stay in the loop of such conversations. If the release cadence of docutils is ~once a year, I'd say we have enough time to give this a try :)

[astrojuanlu]

cc @ericholscher

[chrisjsewell]

I now work on Read the Docs

Ah fair lol

If the release cadence of docutils is ~once a year, I'd say we have enough time to give this a try

yep absolutely 👍

[ericholscher]

I'm 👍 on deprecating recommonmark. I will add it to our internal roadmap call this week, and then see if @astrojuanlu can move forward on it as a priority item on our side 👍

[chrisjsewell]

Ok cool, yeh feel free to get in touch about any question/issues, and I will think about how best to approach the docutils integration.
I guess we can open up a new issue or meta discussion on this.

[astrojuanlu]

xref readthedocs/recommonmark#221