update to 4.10 ANTLR, update grammar (#524)

* add options to lexer rules per antlr/antlr4#3437

* Fixes #525, clean up calls to antlr tool. upgrade lib versions.

Seems to work with 2021.3.3

* Fixes #518

* update doc

Signed-off-by: Terence Parr <[email protected]>

* Allow earlier versions of intellij

* add noteAbout how to remove a deprecated function in the future; I couldn't figure out how to make it work at the moment.

* update doc

Signed-off-by: Terence Parr <[email protected]>

* remove dead code

* let gradle see antlr snapshots

* test with latest intellij

.github/workflows/gradle.yml

@@ -15,15 +15,11 @@ jobs:
         env:
           # see https://www.jetbrains.com/idea/download/previous.html
           # and https://www.jetbrains.com/intellij-repository/snapshots/
-          - IDEA_VERSION: IC-2018.3.6
-          - IDEA_VERSION: IC-2019.1.4
-          - IDEA_VERSION: IC-2019.2.4
-          - IDEA_VERSION: IC-2019.3.3
           - IDEA_VERSION: IC-2020.1
           - IDEA_VERSION: IC-2020.2
           - IDEA_VERSION: IC-2020.3
           - IDEA_VERSION: IC-2021.2.4
-          - IDEA_VERSION: IC-2021.3
+          - IDEA_VERSION: IC-2021.3.3
           - IDEA_VERSION: IU-LATEST-EAP-SNAPSHOT
 
     steps:

README.md

@@ -1,10 +1,10 @@
 # IntelliJ Idea Plugin for ANTLR v4 ![Java CI](https://github.com/antlr/intellij-plugin-v4/workflows/Java%20CI/badge.svg?branch=master) [![Latest version](https://img.shields.io/jetbrains/plugin/v/7358.svg?label=latest%20version)](https://plugins.jetbrains.com/plugin/7358) ![Downloads](https://img.shields.io/jetbrains/plugin/d/7358.svg)
 
-An [IntelliJ](https://www.jetbrains.com/idea/) 2018.3 .. 2021.x plugin for ANTLR v4 ([plugin source at github](https://github.com/antlr/intellij-plugin-v4)).
+An [IntelliJ](https://www.jetbrains.com/idea/) 2020.3+ plugin for ANTLR v4 ([plugin source at github](https://github.com/antlr/intellij-plugin-v4)). (Technically, the plugin will load in 2019.2+ but you might not have much luck earlier than 2020.3)
 
 [Plugin page at intellij](http://plugins.jetbrains.com/plugin/7358?pr=idea)
 
-This plugin is for ANTLR v4 grammars and includes ANTLR 4.9.1. Works with IntelliJ IDEA and other IntelliJ-based IDEs.
+This plugin is for ANTLR v4 grammars and includes ANTLR 4.10. Works with IntelliJ IDEA and other IntelliJ-based IDEs.
 
 ## Features:
 
@@ -131,7 +131,7 @@ If you see ambiguities highlighted, those you should definitely take a look
  consistent with the speed of a generated and compiled parser but it does use
  single-stage full LL parsing which can be slower.  It needs to do that so
  that it gets full and complete profiling information. For those in the know,
- it uses PredictionMode.LL_EXACT_AMBIG_DETECTION. For really big files and
+ it uses `PredictionMode.LL_EXACT_AMBIG_DETECTION`. For really big files and
  slow grammars, there is an appreciable delay when displaying the parse tree or profiling information.
 
 ![parse-region.png](images/profiler.png)
@@ -173,7 +173,7 @@ Even when a grammar is getting a single interpretation of the input, we often wo
 
 # Contributing
 
-We [moved to Gradle](https://github.com/antlr/intellij-plugin-v4/pull/295) for 1.9. To contribute to the project, 
+We [use Gradle](https://github.com/antlr/intellij-plugin-v4/pull/295) to build. To contribute to the project, 
 you need a recent version of IntelliJ IDEA (either Community or Ultimate) with the `Gradle` and `Plugin DevKit` 
 plugins enabled. 
 
@@ -189,6 +189,21 @@ To launch unit tests, run `./gradlew check`.
 
 To build a zipped version of the plugin and its dependencies, run `./gradlew buildPlugin`.
 
-You can tweak the version of the IntelliJ platform used to build/test the plugin in `gradle.properties`. Versions prior to 2018.1.5 require a JDK 1.8, but later versions also support JDK 9+.
+You can tweak the version of the IntelliJ platform used to build/test the plugin in `gradle.properties`, such as:
+
+```
+ideaVersion=IC-2020.2.2
+```
+
+As of 1.18, java 11 is assumed but you might get away with earlier java.
 
 A high level description of how the plugin works can be found in `ARCHITECTURE.md`.
+
+# Releasing
+
+* Search for 4.9, 4.10 etc... of ANTLR.
+* Update release notes in plugin.xml
+* Test with multiple IDE versions
+* Run unit tests
+* Upload to https://plugins.jetbrains.com/plugin/add#intellij
+* Write release note in github

build.gradle

@@ -5,11 +5,11 @@ buildscript {
 }
 
 plugins {
-    id "org.jetbrains.intellij" version "1.4.0"
+    id "org.jetbrains.intellij" version "1.5.2"
 }
 
 wrapper {
-    gradleVersion = '6.8'
+    gradleVersion = '7.4.2'
 }
 
 group 'antlr'
@@ -34,6 +34,10 @@ intellij {
 
 repositories {
     mavenCentral()
+    mavenLocal()
+    maven { // Gets snapshots of antlr if needed
+        url "https://oss.sonatype.org/content/repositories/snapshots/"
+    }
 }
 
 dependencies {
@@ -42,7 +46,7 @@ dependencies {
     }
     implementation "org.antlr:antlr4-intellij-adaptor:0.1"
     testImplementation group: 'junit', name: 'junit', version: '4.13.2'
-    testImplementation group: 'org.mockito', name: 'mockito-core', version: '4.3.1'
+    testImplementation group: 'org.mockito', name: 'mockito-core', version: '4.4.0'
 }
 
 generateGrammarSource {

gradle.properties

@@ -1,4 +1,4 @@
-pluginVersion=1.17
+pluginVersion=1.18
 
 # e.g. IC-2016.3.3, IU-2018.2.5 etc
 # For a list of possible values, refer to the section 'com.jetbrains.intellij.idea' at
@@ -7,12 +7,16 @@ pluginVersion=1.17
 # Old version we use to test code
 #ideaVersion=IC-2019.1.3
 
+#ideaVersion=IC-2020.1 might work on non mac
+
 # Fixes OS X Monterey error about missing a .dylib
 #ideaVersion=IC-2020.2.1
 
 #ideaVersion=IC-2020.2.2
 
-ideaVersion=IC-2021.2
+#ideaVersion=IC-2021.2
+
+ideaVersion=IC-2021.3.3
 
 # The version of ANTLR v4 that will be used to generate the parser
-antlr4Version=4.9.3
+antlr4Version=4.10-SNAPSHOT

src/main/antlr/org/antlr/intellij/plugin/parser/ANTLRv4Parser.g4

@@ -207,7 +207,7 @@ labeledAlt
 
 lexerRule
 	:	DOC_COMMENT* FRAGMENT?
-		TOKEN_REF COLON lexerRuleBlock SEMI
+		TOKEN_REF optionsSpec? COLON lexerRuleBlock SEMI
 	;
 
 lexerRuleBlock

src/main/java/org/antlr/intellij/plugin/ANTLRv4ExternalAnnotator.java

@@ -122,6 +122,9 @@ private Optional<Annotation> annotate(@NotNull AnnotationHolder holder, GrammarI
 
 		case WARNING_ONE_OFF:
 		case INFO:
+			/* When trying to remove the deprecation warning, you will need something like this:
+			AnnotationBuilder builder = holder.newAnnotation(HighlightSeverity.WEAK_WARNING, issue.getAnnotation()).range(range);
+			 */
 			return Optional.of(holder.createWeakWarningAnnotation(range, issue.getAnnotation()));
 
 		default:

src/main/java/org/antlr/intellij/plugin/parsing/PreviewParser.java

@@ -1,16 +1,8 @@
 package org.antlr.intellij.plugin.parsing;
 
 import com.intellij.openapi.progress.ProgressManager;
-import org.antlr.v4.runtime.InterpreterRuleContext;
-import org.antlr.v4.runtime.ParserRuleContext;
-import org.antlr.v4.runtime.RecognitionException;
-import org.antlr.v4.runtime.Token;
-import org.antlr.v4.runtime.TokenStream;
-import org.antlr.v4.runtime.atn.ATN;
-import org.antlr.v4.runtime.atn.ATNDeserializer;
-import org.antlr.v4.runtime.atn.ATNSerializer;
-import org.antlr.v4.runtime.atn.ATNState;
-import org.antlr.v4.runtime.atn.DecisionState;
+import org.antlr.v4.runtime.*;
+import org.antlr.v4.runtime.atn.*;
 import org.antlr.v4.tool.Grammar;
 import org.antlr.v4.tool.GrammarParserInterpreter;
 
@@ -33,7 +25,7 @@ public PreviewParser(Grammar g, ATN atn, TokenStream input) {
 	}
 
 	public PreviewParser(Grammar g, TokenStream input) {
-		this(g, new ATNDeserializer().deserialize(ATNSerializer.getSerializedAsChars(g.getATN())), input);
+		this(g, new ATNDeserializer().deserialize(ATNSerializer.getSerialized(g.getATN()).toArray()), input);
 	}
 
 	@Override

src/main/java/org/antlr/intellij/plugin/parsing/RunANTLROnGrammarFile.java

@@ -86,7 +86,7 @@ public void run(@NotNull ProgressIndicator indicator) {
 				Grammar g = previewState.lg;
 				String language = g.getOptionString(ANTLRv4GrammarProperties.PROP_LANGUAGE);
 				Tool tool = ParsingUtils.createANTLRToolForLoadingGrammars(getGrammarProperties(project, grammarFile));
-				CodeGenerator gen = new CodeGenerator(tool, g, language);
+				CodeGenerator gen = CodeGenerator.create(tool, g, language);
 				gen.writeVocabFile();
 			}
 		}
@@ -106,7 +106,7 @@ private boolean isGrammarStale(ANTLRv4GrammarProperties grammarProperties) {
 		}
 
 		String language = g.getOptionString(ANTLRv4GrammarProperties.PROP_LANGUAGE);
-		CodeGenerator generator = new CodeGenerator(null, g, language);
+		CodeGenerator generator = CodeGenerator.create(null, g, language);
 		String recognizerFileName = generator.getRecognizerFileName();
 
 		VirtualFile contentRoot = getContentRoot(project, grammarFile);

src/main/java/org/antlr/intellij/plugin/validation/GrammarIssuesCollector.java

@@ -8,6 +8,8 @@
 import org.antlr.runtime.ANTLRReaderStream;
 import org.antlr.runtime.Token;
 import org.antlr.v4.Tool;
+import org.antlr.v4.codegen.CodeGenerator;
+import org.antlr.v4.codegen.Target;
 import org.antlr.v4.parse.ANTLRParser;
 import org.antlr.v4.runtime.misc.IntervalSet;
 import org.antlr.v4.tool.*;
@@ -19,9 +21,9 @@
 
 import java.io.File;
 import java.io.StringReader;
+import java.lang.reflect.Constructor;
 import java.util.*;
 
-import static org.antlr.v4.codegen.CodeGenerator.targetExists;
 
 public class GrammarIssuesCollector {
     private static final String LANGUAGE_ARG_PREFIX = "-Dlanguage=";
@@ -189,4 +191,16 @@ else if ( x instanceof RuleRefAST) {
         ruleDefs.keySet().removeAll(ruleRefs);
         return ruleDefs;
     }
+
+    public static boolean targetExists(String language) {
+        String targetName = "org.antlr.v4.codegen.target."+language+"Target";
+        try {
+            Class<? extends Target> c = Class.forName(targetName).asSubclass(Target.class);
+            Constructor<? extends Target> ctor = c.getConstructor(CodeGenerator.class);
+            return true;
+        }
+        catch (Exception e) { // ignore errors; we're detecting presence only
+        }
+        return false;
+    }
 }

src/main/resources/META-INF/plugin.xml

@@ -6,8 +6,8 @@
 
   <description><![CDATA[
   <p>
-      This plugin is for ANTLR v4 grammars and includes ANTLR 4.9.1. It works with
-      2018.3-2021.x.  It should work in other JetBrains IDEs.
+      This plugin is for ANTLR v4 grammars and includes ANTLR 4.10. It works with
+      2020.2.1 and above.  It should work in other JetBrains IDEs.
      </p>
 
      <ul>
@@ -73,16 +73,14 @@ For really big files and slow grammars, there is an appreciable delay when displ
 
   <change-notes><![CDATA[
 	<ul>
-	    <li>Added a new 'Tokens' tab in the preview (<a href="https://github.com/antlr/intellij-plugin-v4/issues/417">#417</a>)</li>
-	    <li>The profiler results now show rule names (<a href="https://github.com/antlr/intellij-plugin-v4/issues/466">#466</a>)</li>
-	    <li>Updated to ANTLR 4.9.2 (<a href="https://github.com/antlr/intellij-plugin-v4/issues/477">#477</a>)</li>
+	    <li>Updated to ANTLR 4.10</li>
     </ul>
-	 See the complete list of <a href="https://github.com/antlr/intellij-plugin-v4/issues?q=milestone%3A1.17+is%3Aclosed">fixed issues</a>.
+	 See the complete list of <a href="https://github.com/antlr/intellij-plugin-v4/issues?q=milestone%3A1.18+is%3Aclosed">fixed issues</a>.
       ]]>
   </change-notes>
 
-  <!-- please see http://confluence.jetbrains.net/display/IDEADEV/Build+Number+Ranges for description -->
-  <idea-version since-build="183"/> <!-- requires IntelliJ 2018.3+ -->
+  <!-- please see https://plugins.jetbrains.com/docs/intellij/build-number-ranges.html for description -->
+  <idea-version since-build="192"/> <!-- requires IntelliJ 2020.3+ due to java 11 issue really but allowing 2019.2 just in case it works for somebody -->
 
   <!-- please see http://confluence.jetbrains.net/display/IDEADEV/Plugin+Compatibility+with+IntelliJ+Platform+Products
        on how to target different products -->