Shared token types #2100
Shared token types #2100
Conversation
|
| Shared token types enable comparisons between languages | ||
|
|
||
| Examples are given in a pseudo language similar to Java or Cpp | ||
|
|
||
| # The Imperative paradigm | ||
|
|
||
| ## Variables |
There was a problem hiding this comment.
This might be a bit too detailed for the wiki, as the wiki is currently mostly user-focused and has some high-level info for developers. However, we do not even explain tokenization in detail. Maybe we should move this documentation elsewhere, e.g. to the language API.
| METHOD("METHOD"), | ||
| METHOD_END("METHOD_END"), |
There was a problem hiding this comment.
Are methods really that object-oriented? I would say no, as this token would correspond to any function/routine.
|
|
||
| import de.jplag.TokenType; | ||
|
|
||
| public enum ObjectOrientationTokens implements TokenType { |
There was a problem hiding this comment.
I would expect the tokens for classes, interfaces, and enums here or maybe in a separate enum for types.
|
|
||
| import de.jplag.TokenType; | ||
|
|
||
| public enum AnnotationTokenTypes implements TokenType { |
There was a problem hiding this comment.
I would try avoid token types with single tokens.
| IF("IF"), | ||
| ELSE("ELSE"), | ||
| IF_END("IF_END"), | ||
| SWITCH("SWITCH"), | ||
| SWITCH_END("SWITCH_END"), | ||
| CASE("CASE"), | ||
| CASE_END("CASE_END"), | ||
| LOOP("LOOP"), | ||
| LOOP_END("LOOP_END"), | ||
| GOTO("GOTO"), // TODO different context? | ||
| BREAK("BREAK"), | ||
| CONTINUE("CONTINUE"), |
There was a problem hiding this comment.
I would move those into a token type for control flow tokens.
| FUNCTION_DEFINITION("FUNCTION_DEF"), | ||
| FUNCTION_END("FUNCTION_END"), |
There was a problem hiding this comment.
Why differentiate from methods here? Do we have languages that distinguish methods and functions during tokenization?
Added shared token types and description for how they should be used