UI freeze after parse on SQL grammars with large inputs

[Endeavourl]

I'm trying to profile my SQL grammar on a large input with diverse statements, which used to go fine in previous versions some years ago.
I managed to consistently reproduce a UI freeze that seems to happen after parsing of the input is finished.
Looking at the CPU load, there's a ~15s multithreaded parser spike and then a single thread becomes pinned at 100% and UI freezes.
Considering this and #381 it seems that the issue is with displaying parsing/profiling results. And it happens even when disabling every option on the left of the preview panel (auto-update, parse-tree etc).

Repro:
Use example PostgreSQL grammar or my PostgreSQL grammar.
Test root rule (sql rule in my grammar).
Concat every example SQL into a single file, i used this command: find *.sql | xargs -I{} sh -c "cat {}; echo ''" > _concat.sql.
Use the new, ~2.9MB file as ANTLR Preview input.
Note UI freeze after a short delay (parse stage). UI doesn't unfreeze for at least 10 minutes, process has to be killed.

[bjansen]

Hi, thanks for reporting the problem and for the reproducible case. Not sure how to handle this one. There might be ways to improve the situation, like parsing in a (possibly cancellable) background thread instead of the UI thread.
Have you tried parsing this rather huge file with grun directly? Depending on how the grammar is written, if the parsing itself is slow there's not much we can do to make it faster. Avoiding UI freezes should be possible, but there's no guarantee that the background thread won't still take 10min to process the file.
Also I'm not sure how the IDE will react when it has to render the parsing results in the preview window. This has to be done in the UI thread anyway.

[KvanTTT]

There might be ways to improve the situation, like parsing in a (possibly cancellable) background thread instead of the UI thread.

Unfortunately, currently ANTLR doesn't support cancellation: antlr/antlr4#2274

[Endeavourl]

The issue is not cancelability, obviously such large input is expected to produce some temporary UI stutter. Right now though it completely hangs in what seems to be an infinite loop. I let the process run for an hour and it didn't recover. It didn't run out of memory either.
In earlier (ca. 2019) plugin versions this kind of test worked ok, completing UI render in about a minute or so.

I tried running my grammar with grun method, it required removing java package lines from the @header sections, but otherwise worked fine. Parser took several seconds, and it even rendered the graph UI, obviously laggy but somewhat responsive.

To me this seems like an infinite loop somewhere in ANTLR Preview UI code, when it fills the parse result data.

[parrt]

I'm pretty sure that the tree visualization is the issue; it's either nonlinear or mildly so. I wonder if we could simply put the tree construction in a separate thread and kill it if it takes too long? I ran into this issue myself recently.

[bjansen]

@KvanTTT we could do something like override org.antlr.v4.runtime.ParserInterpreter#visitState and throw an exception when needed ;)

[Thracky]

Would like to add my +1 to this issue and note that I am seeing it on a grammar of similar complexity and a much smaller input, only about 310 KB.

[parrt]

hi. I totally agree. I'm going to try to solve this hopefully this weekend. The basic problem is that the tree layout mechanism is nonlinear and I stupidly build the tree picture using the GUI thread I think. Upon keystroke, I need to initiate thread worker to build the Picture but it must kill any Picture construction that is going on already. I don't think this has anything to do with parsing speed canceling the parse. @bjansen does that sound reasonable?

[parrt]

Hi. @Thracky @Endeavourl Can you guys try a snapshot of 1.19? antlr-intellij-plugin-v4-1.19-SNAPSHOT.zip

[Endeavourl]

Hangs exactly like the stable version for me.

[parrt]

Dang. i tried it on java grammar and big input. had no problem. Well, it's slow parsing like 8s but doesn't freeze. You sure it installed correctly?

[parrt]

I'm happy to get your grammar and input if public...ah i see the SQL grammars. i'll try it out. wow. ok, 76k lines? Hmm...not sure that's gonna ever work in interp mode rather than gen'd code executing.

[parrt]

Confirmed. 76k lines kills it. Hmm...should work...investigating.

Wow. It's the hierarchy view update that is super slow. And I don't think it's the initial view...it's the update for new input.

Nope. It's caused by the views but isn't the views. Might be memory pressure / thrashing? Still digging. Damn thing freezes after displaying tree. 5000 lines of SQL makes it jump 2G in RAM. Also I note it's using intel for the 2020.2.1. Trying 2021.3.3 now.

[parrt]

Heh @KvanTTT other than kotlin compiler do you work on plugins? hint hint haha I can't trace this gui thread blocking thing.

[KvanTTT]

No, at least not yet :)

[parrt]

Ok, I have confirmed that not even the simple treeview from a main program works for displaying the tree from 76k lines of SQL. The GUI immediately freezes after I tried to draw:

$ grun SQL sql -gui < /tmp/all.sql 

That is with 76k lines. It seems to work "OK" with 5000 lines of sql.

Anyway, I just don't think the GUI freezing is our problem. Displaying millions of no it's just not some thing that swing with set up for.

I'm going to try to smooth out how we handle displaying the tree but Java swing just can't handle this and put @Endeavourl sorry!

[parrt]

Ok, i give up. Even with tree building totally turned off, the GUI thread spins for 5k lines. Tried all sorts of stuff. :(

START selectFileEvent AWT-EventQueue-0
START updateParseTreeFromDoc AWT-EventQueue-0
PARSE START ApplicationImpl pooled thread 19
STOP updateParseTreeFromDoc AWT-EventQueue-0
STOP selectFileEvent AWT-EventQueue-0
PARSE STOP ApplicationImpl pooled thread 19 619.272541ms
START onParsingCompleted AWT-EventQueue-0
ATTEMPT updateTreeViewer AWT-EventQueue-0
IGNORING updateTreeViewer AWT-EventQueue-0
ATTEMPT updateHierarchyViewer AWT-EventQueue-0
IGNORING updateHierarchyViewer AWT-EventQueue-0
STOP onParsingCompleted AWT-EventQueue-0

[parrt]

Holy crap! It looks like it's the token view text field, which has, maybe, 800k lines given 76k input lines of SQL. @bjansen it's TokenStreamViewer I think. If I comment that out, it takes all 76k SQL lines. A bit sluggish but works, even in intel emulation mode for M1 mac.

1. Do we need this view when we can use ctrl+mouse hover to get token info? Looks like it was added here Token Visualisation #417 but I'm not sure it's needed.

2. If so, maybe we just need a limit to, say, 5000 tokens?
3. If so, maybe we use a text box rather than JBList?

Also, I don't think selecting the token in that view does anything but I see addParsingResultSelectionListener() so maybe we have a bug?

I think I'd vote for removing the view.

[parrt]

Fixed by #551

[parrt]

Ok, I believe I have now fixed it! @bjansen @Thracky @Endeavourl Can you guys try a snapshot of 1.19? Input should be much more responsive. Delay is about 1s if you stop typing in 76k sql input and it updates tree. (I posted one about an hour ago that was wrong. This is the correct one.)

antlr-intellij-plugin-v4-1.19-SNAPSHOT.zip

[parrt]

Still not perfect for huge input but great for a few thousand :)

[Endeavourl]

Thanks, works as expected now!