[Markdown] Fix ligature tokanization #2151
Conversation
| @@ -319,6 +320,9 @@ contexts: | |||
| Markdown will convert this for us. We match it so that the | |||
| HTML grammar will not mark it up as invalid. | |||
| scope: meta.other.valid-ampersand.markdown | |||
| ligatures: | |||
| - match: '<=>' # <=> | |||
| - match: '<[-=<]' # <- <= << | |||
There was a problem hiding this comment.
So, Sublime Text internally passes sequences of symbols that are part of the same token to the text shaper. That is to say, every ascii code point > 32 and < 127 that is not alphanumeric. If we want to form as many ligatures as possible, we'd ideally want to tokenize those. However, it will get complicated since we don't want to consume any symbols that are part of valid Markdown syntax.
There was a problem hiding this comment.
This context doesn't intend to match all possible ligatures. It instead tries to handle the edge case caused by the html tag matching or the valid-bracket rule. I could also try to add some lookaheads or things like that to the corresponding rules to avoid matching the tokens, but I found it a bit more complicated due to the several levels of lookaheads etc.
I found this the most secure way to fix the issue - maybe not the most elegant one.
Another edge case is the **= operator which is not recognized as ligature, but as the ** is used to open bold regions, I am worried about breaking things by matching it.
There was a problem hiding this comment.
Okay, that sounds like a good approach for now, let's try it.
Fixes #1453
This PR is an attempt to fix the issue #1453.