Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Markdown] Fix ligature tokanization #2151

Merged
merged 1 commit into from
Oct 29, 2019
Merged

[Markdown] Fix ligature tokanization #2151

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions Markdown/Markdown.sublime-syntax
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -194,6 +194,7 @@ contexts:
pattern will only match stuff matched by the sub-patterns.
push:
- meta_scope: meta.block-level.markdown
- include: ligatures
- include: block-quote
- include: indented-code-block
- include: atx-heading
Expand Down Expand Up @@ -319,6 +320,9 @@ contexts:
Markdown will convert this for us. We match it so that the
HTML grammar will not mark it up as invalid.
scope: meta.other.valid-ampersand.markdown
ligatures:
- match: '<=>' # <=>
- match: '<[-=<]' # <- <= <<
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, Sublime Text internally passes sequences of symbols that are part of the same token to the text shaper. That is to say, every ascii code point > 32 and < 127 that is not alphanumeric. If we want to form as many ligatures as possible, we'd ideally want to tokenize those. However, it will get complicated since we don't want to consume any symbols that are part of valid Markdown syntax.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This context doesn't intend to match all possible ligatures. It instead tries to handle the edge case caused by the html tag matching or the valid-bracket rule. I could also try to add some lookaheads or things like that to the corresponding rules to avoid matching the tokens, but I found it a bit more complicated due to the several levels of lookaheads etc.

I found this the most secure way to fix the issue - maybe not the most elegant one.

Another edge case is the **= operator which is not recognized as ligature, but as the ** is used to open bold regions, I am worried about breaking things by matching it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, that sounds like a good approach for now, let's try it.

block-quote:
- match: '[ ]{,3}(>)[ ]?'
comment: |
Expand Down Expand Up @@ -388,6 +392,7 @@ contexts:
- match: '{{indented_code_block}}.*$\n?'
scope: markup.raw.block.markdown
bold:
- include: ligatures
- match: '(\*\*)(\*)(?=\S)(?!\*)'
captures:
1: punctuation.definition.bold.begin.markdown
Expand Down Expand Up @@ -662,6 +667,7 @@ contexts:
inline:
- include: escape
- include: ampersand
- include: ligatures
- include: bracket
- include: raw
- include: image-inline
Expand Down
19 changes: 19 additions & 0 deletions Markdown/syntax_test_markdown.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -488,6 +488,12 @@ [email protected].
this is a raw ampersand & does not require HTML escaping
| ^ meta.other.valid-ampersand

this is a raw bracket < <= <- << does not require HTML escaping
| ^ meta.other.valid-bracket
| ^^ - meta.other-valid-bracket - meta.tag
| ^^ - meta.other-valid-bracket - meta.tag
| ^^ - meta.other-valid-bracket - meta.tag

[2]: https://github.com/sublimehq/Packages "Packages Repo"
| <- meta.link.reference.def
|^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ meta.link.reference.def
Expand Down Expand Up @@ -2020,3 +2026,16 @@ end

\~/.bashrc
|^ constant.character.escape

-= += /= %= -- ++ ** !~ =~ ~~ <= >= => <=> // && == !=
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - meta - constant - keyword - variable - punctuation

-= += /= %= -- ++ ** !~ =~ ~~ <= >= => <=> // && == !=
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ meta.block-level markup.raw - constant - keyword - variable - punctuation

> -= += /= %= -- ++ ** !~ =~ ~~ <= >= => <=> // && == !=
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ meta.block-level - constant - keyword - variable

> > -= += /= %= -- ++ ** !~ =~ ~~ <= >= => <=> // && == !=
| ^ meta.block-level.markdown markup.quote.markdown markup.quote.markdown punctuation.definition.blockquote.markdown
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ meta.block-level - constant - keyword - variable