Shared token types

docs/8.-Shared-Token-Types.md

@@ -0,0 +1,320 @@
+Shared token types enable comparisons between languages
+
+Examples are given in a pseudo language similar to Java or Cpp
+
+# The Imperative paradigm
+
+## Variables
+
+Variable definition:
+```
+def x: Int
+| VAR_DEF
+```
+
+Variable definition with assignment:
+```
+def x: Int = 2
+| VAR_DEF
+           | ASSIGN
+```
+
+Assignment:
+```
+x = 3
+  | ASSIGN
+```
+
+## Types
+
+Structure definition:
+```
+type Name {
+| STRUCT_DEF
+    firstName: String
+    | STRUCT_MEMBER
+    lastName: String
+    | STRUCT_MEMBER
+}
+| STRUCT_END
+```
+
+## Control Structures
+
+If else:
+```
+if (...) {
+| IF
+...
+} else {
+  | ELSE
+...
+}
+| IF_END
+```
+
+If without else:
+```
+if(...) {
+| IF
+}
+| IF_END
+```
+
+Switch-case:
+```
+switch(...) {
+| SWITCH
+    case a: {
+    | CASE
+        ...
+    }
+    | CASE_END
+    case b: {
+    | CASE
+        ...
+    }
+    | CASE_END
+}
+```
+
+Switch-case with joined-cases (like in Java):
+```
+switch(...) {
+|SWITCH
+    case a:
+    | CASE
+    case b: {
+    | CASE
+        ...
+    }
+    | CASE_END
+}
+| SWITCH_END
+```
+
+Languages that require an explicit break to end the case clause should extract the CASE_END token on the break.
+
+
+
+While-loop:
+```
+while(...) {
+| LOOP
+      | <potential tokens in condition>
+    ....
+}
+| LOOP_END
+```
+
+For-loop:
+```
+for(def x: Int = 0; x < y; x++) {
+| LOOP
+                    | <potential tokens in condition>
+        | VAR_DEF
+               | ASSIGN
+        ...
+                           | <potential tokens in increment>
+}
+| LOOP_END
+```
+The order of the tokens is the order of the lines here. It is intentionally different from the appearance in the code to be compatible to the while loop
+
+For-each:
+```
+for i in <list> {
+| LOOP
+    | VAR_DEF
+          | <potential tokens in list creation>        
+    ....
+}
+| LOOP_END
+```
+
+Goto:
+```
+goto <label>
+|GOTO
+```
+
+Break:
+```
+break
+| BREAK
+```
+
+Continue:
+```
+continue
+| CONTINUE
+```
+
+Functions:
+```
+<modifier> function doSomething(parameter: Int): Int {
+           | FUNCTION           | VAR_DEF
+    ...
+    return 1
+    | RETURN
+}
+| FUNCTION_END
+```
+Modifiers and other features should be ignored here. All different kinds of functions (procedure, methods, functions, ...) Should be treated the same
+
+Function call:
+```
+doSomething(1)
+| CALL
+
+myObject.doSomething(1)
+         | CALL
+```
+
+## Exception Handling
+
+Throw:
+```
+throw <exception value>
+|THROW
+      | <potential tokens for value>
+```
+
+
+Try-Catch:
+```
+try {
+| TRY
+    ...
+}
+catch(e: Error1) {
+| CATCH
+    ...
+}
+catch(e: Error2) {
+| CATCH
+    ...
+}
+finally {
+| FINALLY
+}
+|TRY_END
+```
+There is no token for the error variable declaration, since it always has to be there and to be as language independent as possible
+
+# Object Orientation
+
+## Class / Interface definitions
+
+To maintain compatibility with other languages, all object-oriented types (enum, class, interface, abstract class, record, etc.) should be treated the same:
+
+```
+class MyClass extends MyInterface {
+| STRUCT_DEF
+    def attribute: Int
+    | STRUCT_MEMBER
+
+    function MyClass() {
+    | CONST
+        ...
+    }
+    | CONT_END
+
+    function memberFunction(): Int {
+    | METHOD
+        ...
+    }
+    | METHOD_END
+}
+| STRUCT_END
+```
+
+## Object creation
+
+```
+def object: MyClass = new MyClass()
+|VAR_DEF
+                    | ASSIGN
+                      | NEW
+```
+
+# Code structure
+
+```
+package my_package
+| CONTEXT
+
+import other_package
+| IMPORT
+```
+
+```
+namespace {
+| CONTEXT
+}
+```
+
+Since JPlag doesn't care about the contents of a namespace or package, it's end isn't actually relevant. The only relevant thing is that it is declared.
+
+# Assert
+
+```
+assert 1 < 2
+| ASSERT
+```
+
+# Annotations
+
+```
+@MyAnnotation
+| ANNOTATION
+function myFunction(@NotNull param: Int) {
+| FUNCTION
+                    | ANNOTATION
+                             | VAR_DEF
+    ...
+}
+| FUNCTION_END
+```
+
+The annotation token should always appear before the annotated structure
+
+# Arrays
+
+```
+def x: Int[] = new Int[1]
+               | ARRAY_NEW
+
+def x: Int[] = new Int[]{1}
+               | ARRAY_NEW
+                        | ARRAY_INIT_START
+                          | ARRAY_INIT_END
+```
+
+# Changes with attributes
+
+```
+class X {
+| (STRUCT_DEF, CLASS_DEF)
+}
+| (STRUCT_END, CLASS_END)
+```
+
+Other types (enum, union, record) follow suit
+
+```
+assert 1 < 2
+| (CALL, ASSERT)
+```
+
+```
+def x: Int[] = new Int[]{1}
+               | (ARRAY_NEW, NEW)
+                        | ARRAY_INIT_START
+                          | ARRAY_INIT_END
+```
+
+# TODO
+
+Synchronized-Syntax
+Modules

language-api/src/main/java/de/jplag/tokentypes/AnnotationTokenTypes.java

@@ -0,0 +1,18 @@
+package de.jplag.tokentypes;
+
+import de.jplag.TokenType;
+
+public enum AnnotationTokenTypes implements TokenType {
+    ANNOTATION("ANNO");
+
+    private String description;
+
+    AnnotationTokenTypes(String description) {
+        this.description = description;
+    }
+
+    @Override
+    public String getDescription() {
+        return this.description;
+    }
+}

language-api/src/main/java/de/jplag/tokentypes/ArraySyntaxTokenTypes.java

@@ -0,0 +1,20 @@
+package de.jplag.tokentypes;
+
+import de.jplag.TokenType;
+
+public enum ArraySyntaxTokenTypes implements TokenType {
+    NEW_ARRAY("ARRAY_NEW"),
+    ARRAY_INITIALIZER_START("ARRAY_INIT_START"),
+    ARRAY_INITIALIZER_END("ARRAY_INIT_END"),;
+
+    private String description;
+
+    ArraySyntaxTokenTypes(String description) {
+        this.description = description;
+    }
+
+    @Override
+    public String getDescription() {
+        return this.description;
+    }
+}

language-api/src/main/java/de/jplag/tokentypes/AssertTokenTypes.java

@@ -0,0 +1,18 @@
+package de.jplag.tokentypes;
+
+import de.jplag.TokenType;
+
+public enum AssertTokenTypes implements TokenType {
+    ASSERT("ASSERT"),;
+
+    private String description;
+
+    AssertTokenTypes(String description) {
+        this.description = description;
+    }
+
+    @Override
+    public String getDescription() {
+        return this.description;
+    }
+}