link text containing backtick code doesn't work for []::()

[ericseppanen]

With Markdown 3.3.6, the following markdown link doesn't render properly (no output is generated):

[Hello `[T]::world()` link](https://example.com)

If I remove the inner right-bracket ] or the following colon :, or the open parenthesis (, then it does render something.

commonmark 0.9.1 is able to render this link as expected:

<a href="https://example.com">Hello <code>[T]::world()</code> link</a>

[facelessuser]

What is interesting about this is that it only fails when it is the only thing in a paragraph. Adding any other text, before or after the link, it then parses fine.

https://johnmacfarlane.net/babelmark2/?text=%5BHello+%60%5BT%5D%3A%3Aworld()%60+link%5D(https%3A%2F%2Fexample.com)%0A%0A%5BHello+%60%5BT%5D%3A%3Aworld()%60+link%5D(https%3A%2F%2Fexample.com)+test%0A%0Atest+%5BHello+%60%5BT%5D%3A%3Aworld()%60+link%5D(https%3A%2F%2Fexample.com)

[ericseppanen]

I found a bunch of examples of this bug in the "This Week in Rust" repository, which has a lot of unordered lists containing only links, so the link is the only thing in the enclosing tag (an <li>).

[ericseppanen]

I noticed another example, where there are no parentheses:

• broken: [Hello `[T]::world`](https://example.com)
• works: [Hello `[T]::world `](https://example.com) (note space before closing backtick)

[facelessuser]

This is a weird issue for sure, and I doubt it is something as simple as the link pattern needs improvement because it does work, depending on unrelated context. I suspect that something else is causing the content to get dropped, but it will require a deep dive to debug what is going on.

[facelessuser]

Well, I know what is causing the issue now, I'm just not sure how to fix it yet. The block processor called ReferenceProcessor is responsible for swallowing this content before the link processor even has a chance.

I'll have to try and figure out how we can work around this. I probably won't have a solution this weekend, but this is a good starting point as we now have a root cause.

[waylan]

This is a very specific edge case which requires the internal square brackets, the colon and (optionally) the internal parentheses. Without any one of those specific characters in that specific order, this would not be an issue. For the record, the example ([Hello `[T]::world()` link](https://example.com)) gets parsed into these parts:

Reference: [Hello `[T]:
URL: :world
Title: ()` link](https://example.com)

And the second example ([Hello `[T]::world`](https://example.com)) into these parts:

Reference: [Hello `[T]:
URL: :world`](https://example.com)

I should note that between the square brackets of the reference we do not allow a closing square bracket (regex: \[([^\]]*)\]:). If this is correct (needs verification), then we should simply not allow either opening or closing square brackets (\[([^\[\]]*)\]:), which would be an easy fix.

However, if it is wrong to exclude nested square brackets from references, then I see a few possible workarounds:

1. Make it backtick aware. It should ignore anything between backtick pairs within the reference.
2. Make it nested bracket aware. It should recognize that a closing bracket is closing the child opening bracket, not the parent bracket.
3. Only match valid URLs as URLs (we currently only match any non-whitespace characters).

Any of those solutions would drastically complicate the regex we use and/or require us to not use a single regex for the entire match.

[waylan]

Whether square brackets are allowed within references seems to be mixed (see Babelmark).

I find the strangest result to be the reference implementation (markdown.pl) which does not recognize the link as pointing to the reference, but it does recognize the reference and remove it from the document. It is clear that there is a bug there, but is the bug with the link or the reference? It is impossible to say for sure.

Also of interest is the large number of implementations which error out with this very simple input. It seems that for the most part it is not expected that square brackets be nested inside references. The few implementations which do support it are the ones which already have the reputation of being more forgiving that average. Therefore, I am comfortable excluding the opening square bracket. We already disallow the closing square bracket anyway, so this is consistent with our existing behavior.

[ericseppanen]

I'm not sure if This Week in Rust is an extreme outlier, but we use code backticks inside link text quite a lot (which often includes square brackets and other interesting characters), and it looks like we've been using them (mostly) successfully since about 2013.

Today, it works about 99% of the time (because ]: is somewhat rare.) Removing support for [] entirely would have a pretty significant effect on our website & newsletter workflow.

If we're going to be collateral damage, then I guess that's the way it goes, but I hope you understand that this would be a significant breaking change for us.

[facelessuser]

Today, it works about 99% of the time (because ]: is somewhat rare.) Removing support for [] entirely would have a pretty significant effect on our website & newsletter workflow.

I think you are misunderstanding. The change would apply to the reference pattern, not the link pattern.

• References: [id]: link
• LInk: [content](link)

We would restrict brackets in the id within references.

[ericseppanen]

I think you are misunderstanding. The change would apply to the reference pattern, not the link pattern.

Ah, thank you for the clarification! Sorry for my misunderstanding.

[waylan]

The change would apply to the reference pattern, not the link pattern.

Yes, that is correct. Of note is the fact that reference ids should generally be rather plain. While we do support the shortcut where the id is the link label ([label]) we also support the full syntax ([label][id]), which can be used when a label is more complex. Therefore, we don't loose any functionality with this restriction. In case it's not clear, I'm speaking of the syntax of the link in the document body which would point to a reference.

[ericseppanen]

I verified that the committed fix works on all of the problematic examples in the TWiR repository.

Thank you!