Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix lexing of jsl messages and nested block comments #1638

Merged
merged 5 commits into from
Dec 5, 2020
Merged

Fix lexing of jsl messages and nested block comments #1638

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
18bc9c8
fix for messages that don't end with ( or ;
Nov 27, 2020
2393085
handle nested block comments
Nov 27, 2020
5c37a45
more correct lexing of messages
Nov 27, 2020
dd1644b
add missing operators
Dec 4, 2020
0add3f6
parse decimals that start with . as nums
Dec 4, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 19 additions & 7 deletions lib/rouge/lexers/jsl.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,22 +14,20 @@ class JSL < RegexLexer
rule %r/\s+/m, Text::Whitespace

rule %r(//.*?$), Comment::Single
rule %r(/\*.*?\*/)m, Comment::Multiline
rule %r'/[*].*', Comment::Multiline, :comment

# messages
rule %r/(<<)(.*?)(\(|;)/ do |m|
groups Operator, Name::Function, Punctuation
end
rule %r/<</, Operator, :message

# covers built-in and custom functions
rule %r/([a-z_][\w\s'%.\\]*)(\()/i do |m|
groups Keyword, Punctuation
end

rule %r/\b[+-]?(?:[0-9]+(?:\.[0-9]+)?|\.[0-9]+|\.)(?:e[+-]?[0-9]+)?i?\b/i, Num

rule %r/\d{2}(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\d{2}(\d{2})?(:\d{2}:\d{2}(:\d{2}(\.\d*)?)?)?/i, Literal::Date

rule %r/-?(?:[0-9]+(?:[.][0-9]+)?|[.][0-9]*)(?:e[+-]?[0-9]+)?i?/i, Num

rule %r/::[a-z_][\w\s'%.\\]*/i, Name::Variable
rule %r/:\w+/, Name
rule %r/[a-z_][\w\s'%.\\]*/i, Name::Variable
Expand All @@ -40,16 +38,30 @@ class JSL < RegexLexer
end
rule %r/"/, Str::Double, :dq

rule %r/[-+*\/!%&<>\|=:]/, Operator
rule %r/[-+*\/!%&<>\|=:`^]/, Operator
rule %r/[\[\](){},;]/, Punctuation
end

state :message do
rule %r/\s+/m, Text::Whitespace
rule %r/[a-z_][\w\s'%.\\]*/i, Name::Function
rule %r/[(),;]/, Punctuation, :pop!
rule %r/[&|!=<>]/, Operator, :pop!
end

state :dq do
rule %r/\\![btrnNf0\\"]/, Str::Escape
rule %r/\\/, Str::Double
rule %r/"/, Str::Double, :pop!
rule %r/[^\\"]+/m, Str::Double
end

state :comment do
rule %r'/[*]', Comment::Multiline, :comment
rule %r'[*]/', Comment::Multiline, :pop!
rule %r'[^/*]+', Comment::Multiline
rule %r'[/*]', Comment::Multiline
end
end
end
end
13 changes: 12 additions & 1 deletion spec/visual/samples/jsl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,14 @@ dt << Distribution( Column( :age ), Histograms Only( 1 ) );
Multi-line comment
*/

/*
Nested
/*
Comments
*/
Work
*/

escapeSequence = "This is an \!b escaped sequence";
escapeQuote = "This is a \!" quotation mark";
escapeStr = "\[This is """"""" an escaped string]\"
Expand All @@ -16,9 +24,12 @@ a name with spaces = 5;

scientificNotation = 5e9;
decimal = 1.234;
missing = .;
date = 01jan00;
dateTime = 12dec1999:12:30:00.45;

New Window( "Rouge Test",
Text Box( "Syntax highlighting is great!" )
tb = Text Box( "Syntax highlighting is great!" )
);

If(tb << Get Text != "", "I'm still formatted correctly!");