Using jupytext with nbsphinx

[mgeier]

Sorry for abusing your issue tracker, this is no bug report but rather an advertisement for my Sphinx extension nbsphinx that allows using Jupyter notebooks as Sphinx source files.

And the reason why I'm mentioning this here, is that I've just added a feature (spatialaudio/nbsphinx#234) that allows using jupytext in this process! This should allow using any files supported by jupytext as Sphinx source files.

See https://nbsphinx.readthedocs.io/en/latest/contents-manager.html for documentation.

Note that this feature is currently only available in the master branch, it will be available on PyPI with the next release. It would be great if some people could test it before the next release.

For now, you can install it directly from the Git repository (as described in CONTRIBUTING.rst) or you can use pip like this:

python3 -m pip install git+https://github.com/spatialaudio/nbsphinx.git@master

It would be great if you could try this out and give some feedback if it works and open some issues or pull requests if some things don't work!

Feel free to close this issue any time.

[mwouts]

Thanks @mgeier for reaching out! I am curious to learn more about your project. For now I've only looked a bit at the documentation, but I liked what I read, and I will certainly give it a try. At this time I have a few questions, to which I will probably find the answer when I try nbsphinx:

• Does nbsphinx translates the markdown content to rST?
• Will nbsphinx implement the opposite conversion (rST to Jupyter notebook)? Would you like to contribute a new format for notebooks as text? Or do you think the markdown to rST conversion will make it too difficult to achieve safe round trip conversions?
• I am not sure to see why you use contents manager, and not simply jupytext.readf. Maybe you're doing this as contents manager have a common interface?

Thanks again for letting me know about your project - I am looking forward to learning a bit more about it!

[mgeier]

Does nbsphinx translates the markdown content to rST?

Currently yes. But this should be an implementation detail and at some point I hope to be able to avoid this step, see spatialaudio/nbsphinx#36. I think it would be better to convert the notebook representation directly to a docutils document. But this change may not be happening for a long time ...

Will nbsphinx implement the opposite conversion (rST to Jupyter notebook)?

No, I'm not planning to do that.

I'm not interested in the translation of ipynb -> reST -> ipynb. But if somebody is interested in such a thing, the nbsphinx source code might be interesting to read.

There is even a branch called save-rst, which allows to save the intermediate reST string to a file. But this uses several Sphinx/docutils extensions, so it will not be compatible with a plain reST renderer.

I don't think it makes sense to work directly with such a reST file, but others might have a different opinion.

I am not sure to see why you use contents manager, and not simply jupytext.readf. Maybe you're doing this as contents manager have a common interface?

That's a good point! The reason was indeed that the contents manager is a common interface. But it is also a bit strange for my use case. Within a Sphinx extension, you normally get the contents of a source file as a single string, and I was hacking around this by opening the source file from disk (even though Sphinx had just done that for me).

Based on your suggestion, I tried providing an interface for specifying conversion functions that take the file contents as a string: spatialaudio/nbsphinx#237

This not only makes more sense from a Sphinx API point of view, but I think it actually makes the usage simpler. You would have to use a setting such as this:

import jupytext

nbsphinx_custom_formats = {
    '.Rmd': lambda s: jupytext.reads(s, '.Rmd'),
}

See the documentation here: https://github.com/spatialaudio/nbsphinx/blob/559e2b532c26efebe4861fecf06d1ea007d35e3c/doc/custom-formats.ipynb

It has the additional benefit that it would allow several different custom formats (from different libraries) at the same time.

What do you think about this alternative method?

[mgeier]

I've merged spatialaudio/nbsphinx#237, because I think it is easier to use. Thanks for your suggestion!

If you have any further suggestions, please tell me!

I'm closing this issue because it really isn't an issue, it has never been one ...

[mwouts]

Hello @mgeier , well I still wanted to have a more detailed look at nbsphinx... I am interested as Jupytext is already somehow connected with sphinx-gallery.

I was so busy with the preparation of my first talk on Jupytext, and for that reason I was not as reactive as I had wished... Do you mind if I reopen until I can find so time to read more on nbsphinx? Thanks.

[mgeier]

Sure, I just didn't want to pollute your issue tracker with a non-issue.

There doesn't really have to be an explicit connection to nbsphinx from your side. There just needs to be a function that converts a string in whatever format you support to a Jupyter notebook object.

But of course you could mention nbsphinx in your documentation, or even write your documentation using nbsphinx.

If you didn't look into it, just as a teaser: The idea would be that you commit a .Rmd or .py or whatever supported file to a Github repository which is connected to a documentation project on https://readthedocs.org. Whenever you push to your repository, your files are automatically executed on the RTD server and are displayed together with all their fancy outputs, even though you didn't commit the outputs to your repository.

I think this could probably make paired notebooks to some extent obsolete.

And BTW, you'll get a PDF with all your content for free.

I don't know how sphinx-gallery interacts with all this, but I hope there aren't any conflicts. And if there are problems, please open an issue at https://github.com/spatialaudio/nbsphinx/issues!

[mgeier]

Will nbsphinx implement the opposite conversion (rST to Jupyter notebook)?

I've just stumbled upon those two projects:

https://github.com/nthiery/rst-to-ipynb

https://github.com/QuantEcon/sphinxcontrib-jupyter

I didn't look into them, but I thought you might be interested.

[jamesmyatt]

Can I add my vote for synchronization between rst and ipynb? It would be very helpful to convert a pile of rst to ipynb so that you can convert a docs directory into executable notebooks, and vice versa once the code snippets are updated.

[mwouts]

Hello @jamesmyatt , sure you can ! But tell me a bit more about your expectations. Is it fine for you if by default the rST content is not converted to Markdown (otherwise the round trip conversion would not be safe) ? Also, do you expect that outputs are inserted back into the rST document (that would be a big change for Jupytext, as for now we do not support output in any format)?

[akhmerov]

I like the idea of converting rst to a notebook, especially because this would allow to edit rst files in jupyter environment. However rst is a richer format, and allows more than markdown. The notebook format has markdown postulated in it, and therefore I'm not sure how such a conversion would work.

[mwouts]

I like the idea of converting rst to a notebook, especially because this would allow to edit rst files in jupyter environment

Yes, I think it would help many people if we could offer that. However, as you note, it's not clear how the rST document would look like in Jupyter - should the rST content be converted to Markdown, or not? I think it's very important to preserve the round trip (i.e. you don't want your document to change just because you opened it in Jupyter), so the default should probably be to not converting the text, and only optionally allow the conversion (and there would be no coming back to rST after the conversion...). What do you think?

Also, I'd be interested to know how one could store the metadata in rST document, would you have any suggestion for that?

[akhmerov]

My (rather utopic) preference would be if jupyter notebook format was supporting multiple markup languages, jupyter/nbformat#80 (comment). I don't think there are tools for converting markdown to rst and back with decent round trip consistency, so I indeed agree that not converting the text is the best.

Also, I'd be interested to know how one could store the metadata in rST document, would you have any suggestion for that?

Looking at the docutils spec, there seems to be a meta directive, which is used for document-level key-value storage, but which seems to be ignored if it occurs in the middle of a document. That seems like a good option.

[lukelbd]

Surprised no one has mentioned this, but for the time being (i.e. until jupyter supports multiple markup languages) jupytext could convert reST files by treating the "text" cells as unformatted "Raw" cells belonging to the reST format. Wouldn't this be pretty easy to implement?

I already use "raw" reST cells for the notebooks in my package documentation so that I can populate them with reST links to different parts of the documentation (e.g. `~package.module.function`, :ref:`Section title`).

Although maybe my use case is unusual and this behavior would be confusing for new jupytext users.

[mwouts]

Hello @lukelbd , that's interesting. Does this work when your notebook is saved in one of Jupytext's format?

In theory, the raw cells are supported in Jupytext, see for instance the notebook named The flavors of raw cells in the test folder, and its mirror representations like e.g. this one. Or, if you want a few example, a raw cell with reST format with content :math:1+1 can be encoded as

# + raw_mimetype="text/restructuredtext" active=""
# :math:`1+1`

in the light format, as

# %% [raw] raw_mimetype="text/restructuredtext"
# :math:`1+1`

in the percent format, and as

<!-- #raw raw_mimetype="text/restructuredtext" -->
:math:`1+1`
<!-- #endraw -->

in the markdown format.

[mgeier]

Just a little update from nbsphinx: I've found a problem with the current way to use jupytext in nbsphinx and I'm suggesting a possible solution: spatialaudio/nbsphinx#408

Please try it out and leave your comments/suggestions over there!

[lukelbd]

@mwouts Finally had the chance to try this out. It works perfectly! This really simplifies my workflow -- I can now just run a python linter on the jupytext files to test my documentation examples while keeping the sphinx RST links intact.

Might be worth highlighting this potential use case on the jupytext and/or nbsphinx websites.

[mwouts]

Great news @lukelbd !

Well, could you tell us a bit more? Maybe you have an example that you could share? Especially I'd be curious to know how you structure your documentation - do you maintain the main documentation and the notebooks in two different types of files (.rst and .py maybe?)?

Also, have you seen the new myst format, by the Executable Book Project (cc. @chrisjsewell ) which intends to provide a good rendering of notebooks-like documents with rST content?

Personally I do like very much the idea of testing the documentation. The two examples I have for this are

• test_jupytext_commands_in_the_documentation_work, which tests the jupytext commands in the documentation, once I get tired of getting the documentation wrong...
• The Jupytext/Papermill blog post where I test... the example notebook (Python), and also the blog post itself (using Bash), see the .travis.yml file.

Clearly, instead of testing the doc files, we could also easily lint them with e.g. jupytext --pipe black.

And to conclude, there are two sujects that I'd like to write on when time permits - these are:

• How to write a documentation site, including notebooks How to build a documentation site with Jupytext? #247
• What is a good pre-commit workflow for notebooks How to clean/tests notebooks? #432

[nehalecky]

@mwouts , I've been subscribed to this tread for quite some time: nice to see the progress, and I am going to test out a few workflows here soon. Just FYI the link you have above to the Executable Book Project is broken (possibly https://github.com/executablebooks). Keeping an eye on #247 and #432 also! Thanks so much!

[lukelbd]

@mwouts Yep I now use a mix of .rst files and .py:percent files for my documentation. Previously, I used .ipynb files with nbstripout to avoid ballooning my repo size due to committing inline notebook images. But this is much simpler, especially for collaborators.

Here's a py:percent notebook from a yet-to-be-merged branch of my ProPlot project. It gets translated to this page of my documentation. I add the percent format to conf.py as follows:

# Make nbsphinx detect jupytext files
import jupytext
nbsphinx_custom_formats = {
    '.py': lambda s: jupytext.reads(s, 'py:percent'),
}

To test the notebooks, in my .travis.yaml file I run flake8 on both the repo and documentation folders:

script:
  - flake8 proplot docs --max-line-length=88 --ignore=W503,E402
  - pushd docs
  - make html
  - popd

I've also set up flake8 pre-commit hooks that will now work when committing notebook examples. Have not heard of myrst before but it looks pretty cool! I think I'll stick with this workflow for now though, for all the other benefits of jupytext.

It might still be useful if jupytext can eventually read from RsT files, even without RsT <--> markdown translation (i.e. with "raw" notebook cells). That way my documentation files with examples will look just like my other documentation files, and I can get syntax-highlighting help on the RsT code. Vim has excellent syntax highlighting for RsT files, including python syntax highlighting within embedded .. code-block:: python blocks, and I lose out on that by putting RsT instructions in python comments. Apparently there is a flake8-rst package that can be used to test python blocks in RsT files, so I could still run tests on .rst "notebooks" this way.

Though these are pretty marginal benefits. So far using py:percent is working great for me.

[mgeier]

@lukelbd Please note that your usage of nbsphinx_custom_formats inhibits Sphinx's caching mechanism. I have added a better way to do this: spatialaudio/nbsphinx#408.

You should use something like:

nbsphinx_custom_formats = {
    '.py': ['jupytext.reads', {'fmt': 'py:percent'}],
}

You don't need to import jupytext in this case.

See also https://nbsphinx.readthedocs.io/custom-formats.html.

[lukelbd]

@mgeier Thanks for the suggestion, this is way better. Was trying to figure out why nbshpinx was re-compiling the notebooks every time.

[jorgepiloto]

Hi all, it's Jorge from the poliastro dev team. We recently switched our Jupyter notebook based examples to Jupytext and both sphinx-gallery and Binder seem to keep running without any problem.

Migration process was very simple, thank you for this amazing tool. Please refer to migration to Jupyext pull request for more information 😄

[mgeier]

@jorgepiloto Cool stuff! You should also avoid using lambda in nbsphinx_custom_formats, see above: #119 (comment).

If you don't do that, Sphinx's caching mechanism will be disabled.

[jorgepiloto]

Thanks for the advice @mgeier! Hope to remove the lambda expression in a couple of minutes 🚀