Token Visualisation

[gkjpettet]

Hi,

Many thanks for this extension - it’s proving really useful when learning ANTLR.

Does the extension provide any way to see the tokens generated during lexing? I have figured out how to view the parse tree but I would really like to see the tokens generated before parsing. I'm having issues with certain tokens and I'd like to just throw the extension a large file and view the generated tokens.

Is this possible?

[bjansen]

Hi,

You can see lexer tokens in the Hierarchy view:

But it's not a detailed view of the whole token stream, if that's what you are looking for.

[gkjpettet]

Ah I see - you can only view the tokens matching a rule? OK, thanks.

[bjansen]

Here's a simple implementation:

As you can see, it shows tokens that are not matched by any rule, like ERR.

Things that could be improved, depending on what would be useful for you:

• show the token name instead of its int value (at the moment it's just a Token.toString())
• highlight the corresponding text in the preview editor when a token is selected in the list
• highlight the corresponding token in the list when moving the cursor in the preview editor
• show/hide tokens based on their channel? (for example if you're not interested in what's in the HIDDEN channel, but that might be too far-fetched).

[bjansen]

Updated version:

It shows token names instead of token ids, and I added global "scroll from source/scroll to source" buttons on the left toolbar, which are also used by the Hierarchy tab:

WDYT @gkjpettet and @nmz787?

[bjansen]

Will be in 1.17

[parrt]

What about just looking in input stream at token of interest and just using ctrl + mouse hover? It pops up info on the token:

@gkjpettet would this work for you?