Mistletoe bugs and questions

[choldgraf]

I'm just playing around with mistletoe, and since your fork doesn't allow issues (forks generally don't) I'll put code-specific comments here.

First thing I found: sometimes you want to have colons in arguments. For example in the default sphinx toctree directive it has :caption: Contents:. Doing this with myst results in this error:

yaml.scanner.ScannerError: mapping values are not allowed here

[choldgraf]

Another question - in our example rST file, we have this role:

:doc:`this lecture <heavy_tails>`

How would this be accomplished in myst? Something like

{doc}`[this lecture](heavy_tails)`

?

[choldgraf]

seems that directive nesting is a bit finnicky sometimes - the following code doesn't produce proper output for me:

```{note}
The next info should be nested
```{warning}
Here's my warning
```
```

This is also throwing the following warning:

Pygments lexer name '' is not known

so it sounds like it's treating it as a raw

```

block

[choldgraf]

Labels and other syntax that doesn't exist in rST?

Something else I wanted to do when trying out the roles syntax was using a ref role. However, normally I use it like so:

.. _myref:

Some heading
============

... later on in text

:ref:`myref`.

However, I don't believe anything like .. _myref: syntax exists in markdown, right?

[choldgraf]

Another piece of syntax that doesn't exist in markdown: comments. Perhaps we want some kind of comment syntax to tell the parser to skip things? Generally people use HTML comments to do this, but I feel like we don't want to encourage that...

[chrisjsewell]

First thing I found: sometimes you want to have colons in arguments. For example in the default sphinx toctree directive it has :caption: Contents:.

Ermm yeh it just has to be correct YAML lol, as in caption: "Contents:"

seems that directive nesting is a bit finnicky

You control it by the number of backslashes, each block has to have a different number, with the most on the outside, i.e.

`````{outer}
The next info should be nested
````{inner}
Here's my warning
```{more inner}
```
````
`````

But obviously the option to indent would be ideal.

However, I don't believe anything like .. _myref: syntax exists in markdown, right?

No it doesn't, I've put a few 'todos' for syntax extensions, primarily targets (i.e. labels), comments (as you say, preferably not just the HTML type ones), and field lists (e.g. so you can use the sphinx :orphan: option).

[chrisjsewell]

How would this be accomplished in myst?

Just {doc}`this lecture <heavy_tails>`. I tested and it worked fine

[choldgraf]

Nice - lemme note a few more points. I'm discovering these as I'm writing up some docs about this spec here:

https://github.com/ExecutableBookProject/markup_test_cases/

Multi-part roles

How would multi-part roles be handled here? E.g. :rst:dir:

Code nesting

Your point about outer-fences being longer than inner seemed to work! That said I am still getting a weird warning about WARNING: Pygments lexer name '' is not known

Non-kwarg parameters

It's also possible to pass non-kwarg parameters to a directive (e.g. https://www.sphinx-doc.org/en/master/usage/restructuredtext/directives.html#directive:option-code-block-linenos), which I believe currently doesn't work.

EDIT: Ah I see this is what you mean by field lists above, yeah?

roles link syntax

{doc}this lecture <heavy_tails>

Why would this work? Isn't that <> syntax rST and not markdown?

Error reporting

I'm noticing that when I get errors, the reporting chain breaks down and I get this error-error:

'LoggingReporter' object has no attribute 'get_source_and_line'

Newlines and paragraph breaks

I noticed that breaks in the markup will cause line breaks in the output...maybe this is default markdown behavior?

[chrisjsewell]

multi part roles

Firstly, make sure you update your fork, because I did some updates to ensure tests pass, and also added table parsing.

At the bottom of test_docutils_renderer you'll see that I run a test for all of the docutils/sphinx roles/directives (against sphinx_directives.json and sphinx_roles.json). In these tests you can see that I skip a few roles/directives currently that for whatever reason aren't working, must most pass, including multi-part roles.

Code nesting

If you can give me the exact example of what is failing, I can check

Non-kwarg parameters

You would just use non-kwarg: True, but yes potenetially we would add in the docutils field list syntax, which I didn't actually realise was part of the spec. Then maybe use that directly in the directive blocks:

```{name} arguments
:option: a
:non-kwarg:

Content
```

roles link syntax, Why would this work?

Becuse you don't have to have RST/Markdown syntax within the role content. The role function interprets that string however it wants. Just now if it calls a 'nested parse' on all or part of the string, it would be evaluated as Markdown, rather than RST.

In the case of the doc role, you can see how it interprets it here

Error reporting

That is now fixed, see here

I noticed that breaks in the markup will cause line breaks in the output...maybe this is default markdown behavior?

No I don't think it should, I'll look into it

[chrisjsewell]

From the commonmark discussions, it looks like comment blocks would be best to start with %?

% This is a comment

[AakashGfude]

Another piece of syntax that doesn't exist in markdown: comments. Perhaps we want some kind of comment syntax to tell the parser to skip things? Generally, people use HTML comments to do this, but I feel like we don't want to encourage that...

HTML comments can still be seen in view source though. There seems to be an indirect way of writing comments in markdown specification where it does not appear in the target file, by leveraging the syntax of the link labels. something like:- [//]: <> (This is a comment.) or [//]: # "This is also a comment." . A bit hacky.

[choldgraf]

@chrisjsewell I think that if there is anything close to consensus elsewhere in markdownland, we should just use that 👍

[chrisjsewell]

We can probably move this discussion to ExecutableBookProject/myst_parser now.

For paragraph breaks, the markdown spec is actually that there are soft breaks (no space after) and hard breaks (2+ spaces or \):

Hello  (<-- two spaces)
World

I recall @jstac saying about this before.

Before I was treating these equally:

    def render_line_break(self, token):
       self.current_node.append(nodes.raw("", "<br />", format="html"))

I've now improved that with:

    def render_line_break(self, token):
        if token.soft:
            self.current_node.append(nodes.Text('\n'))
        else:
            self.current_node.append(nodes.raw("", "<br />", format="html"))

[chrisjsewell]

Also you can indent nested directives, but the indentation (for each nesting level) has to be less than 4, otherwise it is treated as a standard code block:

a standard code block

e.g.

  ````{note}
  Here's my note

    ```{warning}
    Here's my warning
    ```

  ````

[choldgraf]

@chrisjsewell I agree - really fantastic work on spot-checking these.

I will go through any outstanding points here and open a new issue in the new repo, and will then close this issue

[chrisjsewell]

LineComment block token now added, so this:

% a comment

goes to this:

<comment xml:space="preserve">
    a comment

and is omitted from HTML

[chrisjsewell]

Target token now added, so you can do things like this:

(header_target)= 
# Header

{ref}`header_target`

Open to syntax suggestions, but I though the = was intuitive in showing it is being assigned to the next element

[choldgraf]

Nice - I just added it in the docs to #7

We can take a pass through syntax maybe tomorrow or something. This seems reasonable but I guess the main question is whether something like this exists in pandoc already.

[chrisjsewell]

Yeh sure ting, to my knowledge there's no 'target' like syntax/functionality in Commonmark/pandoc, it is a purely RST construct