Antlr Preview Freezes Intellij but no crash

[AlexCouch]

I am getting a weird behavior with this plugin and I have looked everywhere for a solution. Every time I try using the grammar preview tool, it freezes intellij. I don't know why. I have prepared a snippet of the log that shows what intellij was doing when it froze, and the grammar I am using that causes it to crash. Besides that, antlr works just fine. This is the only problem that I have. I've tried alternative means of testing a rule but it's always freezing. This happened in 2019.2 and 2019.3 with the latest antlr plugin.

2020-02-20 23:17:08,132 [   7460]   INFO - j.ide.script.IdeStartupScripts - 0 startup script(s) found 
2020-02-20 23:17:08,316 [   7644]   INFO - rojectCodeStyleSettingsManager - Initialized from default code style settings. 
2020-02-20 23:17:08,779 [   8107]   INFO -        ANTLRv4PluginController - ANTLR 4 Plugin version 1.13.1, Java version 11.0.4 
2020-02-20 23:17:08,779 [   8107]   INFO -        ANTLRv4PluginController - createToolWindows kotlinx-llvm 
2020-02-20 23:17:08,790 [   8118]   INFO -             ANTLR PreviewPanel - createEditorPanel kotlinx-llvm 
2020-02-20 23:17:08,868 [   8196]   INFO -             ANTLR PreviewPanel - createParseTreePanel kotlinx-llvm 
2020-02-20 23:17:09,446 [   8774]   INFO -        ANTLRv4PluginController - installListeners kotlinx-llvm 
2020-02-20 23:17:10,469 [   9797]   INFO -        ANTLRv4PluginController - currentEditorFileChangedEvent none -> C:/projects/kotlinx-llvm/toylang/src/main/kotlin/parsing/ASTToLLVM.kt kotlinx-llvm 
2020-02-20 23:17:10,592 [   9920]   INFO - .diagnostic.PerformanceWatcher - Pushing properties took 908ms; general responsiveness: ok; EDT responsiveness: 1/1 sluggish 
2020-02-20 23:17:11,075 [  10403]   INFO -        ANTLRv4PluginController - currentEditorFileChangedEvent C:/projects/kotlinx-llvm/toylang/src/main/kotlin/parsing/ASTToLLVM.kt -> C:/projects/kotlinx-llvm/toylang/src/main/antlr/ToylangParser.g4 kotlinx-llvm 
2020-02-20 23:17:11,500 [  10828]   INFO -        ANTLRv4PluginController - loadGrammars C:/projects/kotlinx-llvm/toylang/src/main/antlr/ToylangParser.g4 kotlinx-llvm 
2020-02-20 23:17:11,618 [  10946]   INFO -        ANTLRv4PluginController - loadGrammars parser ToylangParser 
2020-02-20 23:17:11,618 [  10946]   INFO -             ANTLR PreviewPanel - switchToGrammar C:/projects/kotlinx-llvm/toylang/src/main/antlr/ToylangParser.g4 kotlinx-llvm 
2020-02-20 23:17:11,618 [  10946]   INFO -               ANTLR InputPanel - switchToGrammar C:/projects/kotlinx-llvm/toylang/src/main/antlr/ToylangParser.g4 kotlinx-llvm 
2020-02-20 23:17:11,619 [  10947]   INFO -               ANTLR InputPanel - createEditor: create new editor for C:/projects/kotlinx-llvm/toylang/src/main/antlr/ToylangParser.g4 kotlinx-llvm 
2020-02-20 23:17:13,441 [  12769]   WARN - com.intellij.util.xmlb.Binding - no accessors for class org.jetbrains.kotlin.idea.highlighter.KotlinDefaultHighlightingSettingsProvider 
2020-02-20 23:17:13,705 [  13033]   INFO - j.ide.ui.OptionsTopHitProvider - 1251 ms spent to cache options in project 
2020-02-20 23:17:14,209 [  13537]   INFO - ge.ExternalProjectsDataStorage - Load external projects data in 693 millis (read time: 682) 
2020-02-20 23:17:14,344 [  13672]   INFO - .diagnostic.PerformanceWatcher - Post-startup activities under progress took 4830ms; general responsiveness: ok; EDT responsiveness: 2/4 sluggish 
2020-02-20 23:17:14,403 [  13731]   INFO - indexing.UnindexedFilesUpdater - Unindexed files update canceled 
2020-02-20 23:17:14,418 [  13746]   INFO - pl.ProjectRootManagerComponent - project roots have changed 
2020-02-20 23:17:14,677 [  14005]   INFO - tartup.impl.StartupManagerImpl - C:/projects/kotlinx-llvm/.idea case-sensitivity: expected=false actual=false 
2020-02-20 23:17:14,746 [  14074]   INFO -  #git4idea.commands.GitHandler - [.] git version 
2020-02-20 23:17:14,777 [  14105]   INFO - tellij.diagnostic.LoadingPhase - Reached PROJECT_OPENED loading phase 
2020-02-20 23:17:14,850 [  14178]   INFO - .diagnostic.PerformanceWatcher - Pushing properties took 97ms; general responsiveness: ok; EDT responsiveness: ok 
2020-02-20 23:17:14,867 [  14195]   INFO -  #git4idea.commands.GitHandler - git version 2.23.0.windows.1 
2020-02-20 23:17:14,930 [  14258]   INFO - pl.projectlevelman.NewMappings - Mapped Roots: 1 
2020-02-20 23:17:15,000 [  14328]   INFO - ea.config.GitExecutableManager - Git version for C:\tools\git\cmd\git.exe : 2.23.0 
2020-02-20 23:17:15,236 [  14564]   INFO - tor.impl.FileEditorManagerImpl - Project opening took 10356 ms 
2020-02-20 23:17:18,654 [  17982]   INFO - .diagnostic.PerformanceWatcher - Indexable file iteration took 3803ms; general responsiveness: ok; EDT responsiveness: 1/4 sluggish 
2020-02-20 23:17:18,657 [  17985]   INFO - tellij.diagnostic.LoadingPhase - Reached INDEXING_FINISHED loading phase 
2020-02-20 23:17:19,839 [  19167]   INFO - ings.impl.UpdateCheckerService - channel: release 
2020-02-20 23:17:21,525 [  20853]   INFO - idation.GrammarIssuesCollector - doAnnotate C:/projects/kotlinx-llvm/toylang/src/main/antlr/ToylangParser.g4 
2020-02-20 23:17:21,538 [  20866]   INFO - idation.GrammarIssuesCollector - token vocab file ToylangLexer 
2020-02-20 23:17:50,639 [  49967]   INFO - idation.GrammarIssuesCollector - doAnnotate C:/projects/kotlinx-llvm/toylang/src/main/antlr/ToylangParser.g4 
2020-02-20 23:17:50,647 [  49975]   INFO - idation.GrammarIssuesCollector - token vocab file ToylangLexer 
2020-02-20 23:18:01,044 [  60372]   WARN - ConfigurableExtensionPointUtil - ignore deprecated groupId: language for id: preferences.language.Kotlin.scripting 
2020-02-20 23:18:52,015 [ 111343]   INFO - rationStore.ComponentStoreImpl - Saving Module: 'kotlinx-llvm.test'ExternalFacetManager took 30 ms 
2020-02-20 23:18:52,377 [ 111705]   INFO - mponents.impl.stores.StoreUtil - saveProjectsAndApp took 548 ms 
2020-02-20 23:19:22,572 [ 141900]   INFO -           ANTLR TestRuleAction - actionPerformed file://C:/projects/kotlinx-llvm/toylang/src/main/antlr/ToylangParser.g4 
2020-02-20 23:19:22,595 [ 141923]   INFO -        ANTLRv4PluginController - setStartRuleNameEvent toylangFile kotlinx-llvm 
2020-02-20 23:19:22,595 [ 141923]   INFO -             ANTLR PreviewPanel - updateParseTreeFromDoc file://C:/projects/kotlinx-llvm/toylang/src/main/antlr/ToylangParser.g4 rule toylangFile 
2020-02-20 23:19:37,388 [ 156716]   WARN - pl.local.NativeFileWatcherImpl - Watcher terminated with exit code 1 
2020-02-20 23:19:37,391 [ 156719]   INFO - pl.local.NativeFileWatcherImpl - Starting file watcher: C:\tools\intellij\bin\fsnotifier64.exe 

Toylang Lexer

lexer grammar ToylangLexer;

channels { WHITESPACE }

NEWLINE             : ('\r\n' | '\r' | '\n') -> channel(WHITESPACE);
WS                  : [\t ]+ -> channel(WHITESPACE);

LET                 : 'let';
MUT                 : 'mut';
FN                  : 'fn';
RETURN              : 'return';

IDENT               : [a-zA-Z][A-Za-z0-9_]*;

INTLITERAL          : [0-9]+;
DECIMALLITERAL      : ([0-9]+ 'f') | [0-9]+ '.' [0-9]+;


PLUS                : '+';
MINUS               : '-';
ASTERISK            : '*';
DIVISION            : '/';
ASSIGN              : '=';
LPAREN              : '(';
RPAREN              : ')';
COLON               : ':';
SEMICOLON           : ';';
COMMA               : ',';
LCURLBRACE          : '{';
RCURLBRACE          : '}';


STRING_OPEN         : '"' -> pushMode(MODE_IN_STRING);

UNMATCHED           : . ;

mode MODE_IN_STRING;

ESCAPE_STRING_DELIMITER : '\\"';
ESCAPE_SLASH            : '\\\\';
ESCAPE_NEWLINE          : '\\n';
STRING_CLOSE            : '"' -> popMode;
INTERPOLATION_OPEN      : '\\$\\{' -> pushMode(MODE_IN_INTERPOLATION);
STRING_CONTENT          : ~["\n\r\t]+;

STR_UNMATCHED           : . -> type(UNMATCHED);

mode MODE_IN_INTERPOLATION;

INTERPOLATION_CLOSE     : '}' -> popMode;
INTERP_WS               : [\t ]+ -> channel(WHITESPACE), type(WS);
INTERP_LET              : 'let'->type(LET);
INTERP_MUT              : 'mut'->type(MUT);

INTERP_INTLIT           : [0-9]+ -> type(INTLITERAL);
INTERP_DECI_LIT         : ([0-9]+ 'f' | [0-9]+) '.' [0-9]+ -> type(DECIMALLITERAL);

INTERP_PLUS             : '+' -> type(PLUS);
INTERP_MINUS            : '-' -> type(MINUS);
INTERP_ASTERISK         : '*' -> type(ASTERISK);
INTERP_DIVISION         : '/' -> type(DIVISION);
INTERP_LPAREN           : '(' -> type(LPAREN);
INTERP_RPAREN           : ')' -> type(PLUS);

INTERP_IDENT            : [_]*[a-z][A-Za-z0-9_]* -> type(IDENT);
INTERP_STRING_OPEN      : '"' -> type(STRING_OPEN), pushMode(MODE_IN_STRING);
INTERP_UNMATCHED        : . -> type(UNMATCHED);

Toylang Grammar

parser grammar ToylangParser;

options { tokenVocab=ToylangLexer; }

toylangFile             : lines=line+ ;

line                    : statement(NEWLINE | EOF)*;

statement               : letDeclaration | fnDeclaration;

letDeclaration          : LET (MUT?) IDENT assignment SEMICOLON;

fnDeclaration           : FN IDENT fnParams fnType? codeBlock;

fnParam                 : IDENT COLON type;
fnParams                : LPAREN ((fnParam) | (fnParam COMMA fnParam))* RPAREN;
fnType                  : COLON type;
codeBlockStatements     : statement+;
codeBlock               : LCURLBRACE codeBlockStatements returnStatement? RCURLBRACE;
returnStatement         : RETURN expression SEMICOLON;

assignment              : ASSIGN expression;
expression              : left=expression operator=(DIVISION|ASTERISK) right=expression # binaryOperation
                        | left=expression operator=(PLUS|MINUS) right=expression        # binaryOperation
                        | LPAREN expression RPAREN                                      # parenExpression
                        | MINUS expression                                              # minusExpression
                        | STRING_OPEN (parts+=stringLiteralContent)* STRING_CLOSE       # stringLiteral
                        | INTLITERAL                                                    # intLiteral
                        | DECIMALLITERAL                                                # decimalLiteral
                        | IDENT                                                         # valueReference;


stringLiteralContent    : STRING_CONTENT
                        | INTERPOLATION_OPEN expression INTERPOLATION_CLOSE;

type                    : IDENT;

[bjansen]

Hi, usually this is because the grammar is very inefficient at parsing the input, which causes the IDE to freeze because parsing is currently done on the EDT. See #380 for a similar case and #381 for a future 'fix'.

To confirm that it's a perf problem, you could either:

• wait a couple seconds/minutes until the parsing is done, then have a look at the Profiler tab
• parse the same input using grun and see if it's also slow

As to how you can make your grammar faster, you should ask StackOverflow or the mailing list I guess.

[AlexCouch]

I totally forgot to mention that this happens with or without any example code to parse, input and file. I've tried waiting half an hour. The generator visitor and listener work just fine. I have the actual generator code working flawlessly. It's just the previewer

[bjansen]

OK I was able to reproduce the bug. The code is stuck in a loop producing errors like line 1:0 extraneous input '<EOF>' expecting {<EOF>, NEWLINE}, eventually leading to an OOME.

As a workaround, you can modify your start rule like this:

toylangFile             : lines=line+ | EOF;

[bjansen]

Actually, a better fix would be:

toylangFile             : lines=line+;

line                    : statement (NEWLINE* | EOF);

(NEWLINE | EOF)* doesn't really make sense because you have exactly one EOF per file. This causes the interpreter to get stuck in a loop. It doesn't happen in the generated parser because it's not using the interpreter like the IDE does.

[AlexCouch]

Beautiful. It works. Thank you my friend!

[AlexCouch]

Just ran into this issue a bit ago. Not sure what caused it but I have a feeling its similar to this issue with the interpreter. It happened after I tried adding comments.

...
channels { WHITESPACE, COMMENT }
...
mode MODE_IN_COMMENT;

COMMENT_CONTENT             : .*? -> channel(COMMENT);
COMMENT_NEWLINE             : ('\r' | '\n' | '\r\n') -> popMode;

When I switch from the lexer grammar to the parser grammar, it suddenly freezes and presents me with the following error message.

[bjansen]

This is another infinite loop, but this time it's happening both in the interpreter and the real parser. It's caused by a warning that is reported by the IDE:

The parser gets stuck matching an empty string over and over again.

To fix this, simply change your rule to:

COMMENT_CONTENT             : .+? -> channel(COMMENT);

[bjansen]

Related: antlr/antlr4#2444

[bjansen]

I added a "watchdog" that will detect such cases and kill the interpreter if it's stuck because of empty string issues.

In addition, the parser now runs in a separate thread, which means that rogue grammars shouldn't completely freeze the UI anymore. This thread can be killed using a dedicated button:

[AlexCouch]

Beautiful, thank you!