Maint: Restrict empty-matching regexes #1548
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 3 commits
June 19, 2020 12:41
* an empty-matching regex may not be used without a predicate (such as :pop!, :push, or a block) * an empty-matching regex with no lookahead/lookbehind or anchoring "closes" the state - it is invalid to add more rules, as they would never be reached.
pyrmont
reviewed
Jun 19, 2020
pyrmont
added
the
author-action
pyrmont
added
needs-review
pyrmont
reviewed
Jun 19, 2020
pyrmont
approved these changes
Jun 19, 2020
|
I don't quite remember what prompted this, but I think I had seen it come up somewhere. It's a really difficult bug to track down if you're not super familiar with the internals of Rouge. |
|
But yes, I think this is ready to merge. |
This was referenced Nov 9, 2020
pyrmont
added a commit
that referenced
this pull request
Nov 10, 2020
The maintenance changes in #1548 resulted in updates to two rules in the Ruby lexer, one regarding regular expression flags and the other regarding heredoc strings. However, these patterns should not have been changed. The change in #1548 was to prevent rules that matched an empty string unless they contained a 'predicate' of some sort. Both rules in the Ruby lexer contained such a predicate and so should not have been changed. This commit fixes that error.
pyrmont
added a commit
that referenced
this pull request
Nov 10, 2020
The changes made in #1548 to avoid empty regular expression patterns, broke the template string rules in the JavaScript lexer (and lexers such as the TypeScript lexer that inherit from it). This commit adds the rules necessary to fix this lexing.
mattt
pushed a commit
to NSHipster/rouge
that referenced
this pull request
May 19, 2021
This commit makes several changes to the way ‘empty’ regular expressions work in Rouge. 1. The use of ‘empty’ regular expressions is restricted. An empty-matching regex may not be used without a predicate (such as `:pop!`, `:push`, or a block). 2. The use of an empty-matching regular expression with no lookahead/lookbehind or anchoring "closes" the state. It is invalid to add more rules, as they would never be reached. 3. Lexers that were using improper empty-matching regular expressions have been fixed.
mattt
pushed a commit
to NSHipster/rouge
that referenced
this pull request
May 19, 2021
The merging of rouge-ruby#1548 caused issues with two lexers that had been added after the PR was originally submitted. This commit fixes those errors. It also reduces the number of calls to `String#upcase` and `String#downcase` in the Apex lexer (this lexer was updated in rouge-ruby#1548).
mattt
pushed a commit
to NSHipster/rouge
that referenced
this pull request
May 19, 2021
The maintenance changes in rouge-ruby#1548 resulted in updates to two rules in the Ruby lexer, one regarding regular expression flags and the other regarding heredoc strings. However, these patterns should not have been changed. The change in rouge-ruby#1548 was to prevent rules that matched an empty string unless they contained a 'predicate' of some sort. Both rules in the Ruby lexer contained such a predicate and so should not have been changed. This commit fixes that error.
mattt
pushed a commit
to NSHipster/rouge
that referenced
this pull request
May 19, 2021
The changes made in rouge-ruby#1548 to avoid empty regular expression patterns, broke the template string rules in the JavaScript lexer (and lexers such as the TypeScript lexer that inherit from it). This commit adds the rules necessary to fix this lexing.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
It is very common for a lexer developer to accidentally include a regex that matches the empty string, which can cause a number of issues. This patch restricts the use of empty-matching regexes in two ways:
A regex that matches the empty string must have a predicate - either pushing a new state,
:pop!, or a custom block. Otherwise it will infinite-loop.Any empty-matching regex that does not contain lookahead/lookbehind causes the state to be "closed" - no more rules can be added. This is because these regexes must serve as "default" rules, and it's guaranteed that the processing won't continue past them.