Bump antlr4 from 4.9.3 to 4.11.1 #4546

reta · 2022-09-19T13:09:00Z

Signed-off-by: Andriy Redko [email protected]

Description

Manually bumping antlr4 version and regenerate parsers

Issues Resolved

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2022-09-19T13:12:11Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/3199/
CommitID: c4220a1

github-actions · 2022-09-19T13:31:55Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/3200/
CommitID: 5081a56

github-actions · 2022-09-19T13:46:13Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/3207/
CommitID: 5081a56

reta · 2022-09-19T14:51:12Z

Ah, bad news, the Apache Lucene still uses some Antlr 3.x bits (serialized ATN, see here [1] please), but the compatibility with Antlr 3.x was removed [2] recently. As such, Apache Lucene is not compatible with Antlr 4.10+ :(

[1] https://github.com/apache/lucene/blob/main/lucene/expressions/src/java/org/apache/lucene/expressions/js/JavascriptLexer.java#L189
[2] antlr/antlr4@c68e127

github-actions · 2022-12-09T21:33:37Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/7867/
CommitID: 8397dd7
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

reta · 2022-12-16T13:51:11Z

Getting there :) apache/lucene#11788 is merged, 🤞 it will be backported to 9.5

Signed-off-by: Andriy Redko <[email protected]>

reta · 2022-12-16T16:08:43Z

buildSrc/version.properties

@@ -1,5 +1,5 @@
 opensearch        = 3.0.0
-lucene            = 9.5.0-snapshot-d5cef1c
+lucene            = 9.5.0-snapshot-6700b7e


The latest snapshots are needed to bring apache/lucene#12016 in

github-actions · 2022-12-16T16:18:40Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/8191/
CommitID: e5fdf9d

codecov-commenter · 2022-12-16T16:22:14Z

Codecov Report

Merging #4546 (e5fdf9d) into main (cb26035) will decrease coverage by 0.04%.
The diff coverage is 10.52%.

@@             Coverage Diff              @@
##               main    #4546      +/-   ##
============================================
- Coverage     70.95%   70.91%   -0.05%     
+ Complexity    58304    58296       -8     
============================================
  Files          4733     4733              
  Lines        278256   278256              
  Branches      40249    40249              
============================================
- Hits         197441   197319     -122     
- Misses        64628    64771     +143     
+ Partials      16187    16166      -21

Impacted Files	Coverage Δ
...arch/painless/antlr/PainlessParserBaseVisitor.java	`1.26% <ø> (ø)`
.../org/opensearch/painless/antlr/PainlessParser.java	`68.43% <5.55%> (ø)`
...a/org/opensearch/painless/antlr/PainlessLexer.java	`68.18% <100.00%> (ø)`
...adonly/AddIndexBlockClusterStateUpdateRequest.java	`0.00% <0.00%> (-75.00%)`	⬇️
...readonly/TransportVerifyShardIndexBlockAction.java	`9.75% <0.00%> (-73.18%)`	⬇️
.../index/shard/IndexShardNotRecoveringException.java	`0.00% <0.00%> (-50.00%)`	⬇️
.../opensearch/test/store/MockFSDirectoryFactory.java	`22.66% <0.00%> (-46.67%)`	⬇️
...rch/client/transport/NoNodeAvailableException.java	`28.57% <0.00%> (-42.86%)`	⬇️
...h/action/ingest/SimulateDocumentVerboseResult.java	`60.71% <0.00%> (-39.29%)`	⬇️
...indices/readonly/TransportAddIndexBlockAction.java	`20.68% <0.00%> (-37.94%)`	⬇️
... and 476 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

reta · 2022-12-16T16:33:49Z

@dblock @nknize I think we are good to go! (no backport since hard dependency on Apache Lucene 9.5), thanks guys!

acarbonetto · 2022-12-22T00:10:10Z

I know this is merged, but it seems to be causing fatal errors on the darwin-arm64-tar builds when running on m1 macs.

» ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [runTask-0] fatal error in thread [main], exiting
»  java.lang.NoClassDefFoundError: org/opensearch/index/rankeval/RankEvalAction
»       at java.lang.Class.getDeclaredConstructors0(Native Method) ~[?:?]
»       at java.lang.Class.privateGetDeclaredConstructors(Class.java:3373) ~[?:?]
»       at java.lang.Class.getConstructors(Class.java:2060) ~[?:?]
»       at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:765) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»       at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:731) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»       at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:533) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»       at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:195) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»       at org.opensearch.node.Node.<init>(Node.java:427) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»       at org.opensearch.node.Node.<init>(Node.java:354) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»       at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»       at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»       at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»       at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»       at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»       at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»       at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»       at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»       at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»       at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»  Caused by: java.lang.ClassNotFoundException: org.opensearch.index.rankeval.RankEvalAction
»       at java.net.URLClassLoader.findClass(URLClassLoader.java:445) ~[?:?]
»       at java.lang.ClassLoader.loadClass(ClassLoader.java:587) ~[?:?]
»       at java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:872) ~[?:?]
»       at java.lang.ClassLoader.loadClass(ClassLoader.java:520) ~[?:?]
»       ... 19 more

When I checkout the commit prior git checkout ff43d768b10f97bac35a2638c2c6bc2b642cd10f then my ./gradlew run runs fine.

reta · 2022-12-22T00:22:10Z

When I checkout the commit prior git checkout ff43d768b10f97bac35a2638c2c6bc2b642cd10f then my ./gradlew run runs fine.

I have difficulties understanding how this specific change impacts rank-eval plugin (especially on M1), @acarbonetto do you build the complete distribution locally?

acarbonetto · 2022-12-22T17:06:07Z

I don't understand it either. I've been trying to work through the changes and I don't see the relevance.

do you build the complete distribution locally?

Yes. I'm using :distribution:archives:darwin-arm64-tar:assemble and :distribution:archives:darwin-arm64-tar:build to build the install and running it from ./distribution/archives/darwin-arm64-tar/build/install/opensearch-3.0.0-SNAPSHOT/bin/opensearch

acarbonetto · 2022-12-22T21:59:54Z

Workaround is to work with Java 11.

If I build with Java 17 I get the above error, but not with Java 11. Something is missing in java 17 on arm64. The intel Mac seem to compile and build fine.

reta · 2022-12-22T22:08:28Z

Workaround is to work with Java 11.

If I build with Java 17 I get the above error, but not with Java 11. Something is missing in java 17 on arm64. The intel Mac seem to compile and build fine.

Ah ... this is very interesting, I sadly don't have M1 at my disposal but I will try to troubleshoot the issue on first occasion. What JDK distribution you use on M1?

acarbonetto · 2022-12-27T06:16:05Z

What JDK distribution you use on M1?

I'm using Corretto-17.0.3.6.1.

If I found out more information, I'll let you know.

reta requested a review from a team as a code owner September 19, 2022 13:09

reta added the backport 2.x Backport to 2.x branch label Sep 19, 2022

reta force-pushed the antl4.4.11.1 branch from c4220a1 to 5081a56 Compare September 19, 2022 13:10

reta force-pushed the antl4.4.11.1 branch from 5081a56 to 8397dd7 Compare December 9, 2022 21:10

reta requested review from anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, setiah, kartg, kotwanikunal, mch2, nknize, owaiskazi19, adnapibar, Rishikesh1159, ryanbogan, saratvemulapalli, shwetathareja, dreamer-89 and tlfeng as code owners December 9, 2022 21:10

reta requested review from VachaShah and xuezhou25 as code owners December 9, 2022 21:10

reta mentioned this pull request Dec 13, 2022

Upgrade ANTLR to version 4.11.1 apache/lucene#12016

Merged

Bump antlr4 from 4.9.3 to 4.11.1

e5fdf9d

Signed-off-by: Andriy Redko <[email protected]>

reta force-pushed the antl4.4.11.1 branch from 8397dd7 to e5fdf9d Compare December 16, 2022 15:49

reta commented Dec 16, 2022

View reviewed changes

dblock approved these changes Dec 19, 2022

View reviewed changes

dblock merged commit 306b199 into opensearch-project:main Dec 19, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bump antlr4 from 4.9.3 to 4.11.1 #4546

Bump antlr4 from 4.9.3 to 4.11.1 #4546

reta commented Sep 19, 2022

github-actions bot commented Sep 19, 2022

github-actions bot commented Sep 19, 2022

github-actions bot commented Sep 19, 2022

reta commented Sep 19, 2022 •

edited

Loading

github-actions bot commented Dec 9, 2022

reta commented Dec 16, 2022

reta Dec 16, 2022

github-actions bot commented Dec 16, 2022

codecov-commenter commented Dec 16, 2022

reta commented Dec 16, 2022

acarbonetto commented Dec 22, 2022

reta commented Dec 22, 2022

acarbonetto commented Dec 22, 2022

acarbonetto commented Dec 22, 2022

reta commented Dec 22, 2022

acarbonetto commented Dec 27, 2022

Bump antlr4 from 4.9.3 to 4.11.1 #4546

Bump antlr4 from 4.9.3 to 4.11.1 #4546

Conversation

reta commented Sep 19, 2022

Description

Issues Resolved

Check List

github-actions bot commented Sep 19, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 19, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 19, 2022

Gradle Check (Jenkins) Run Completed with:

reta commented Sep 19, 2022 • edited Loading

github-actions bot commented Dec 9, 2022

Gradle Check (Jenkins) Run Completed with:

reta commented Dec 16, 2022

reta Dec 16, 2022

Choose a reason for hiding this comment

github-actions bot commented Dec 16, 2022

Gradle Check (Jenkins) Run Completed with:

codecov-commenter commented Dec 16, 2022

Codecov Report

reta commented Dec 16, 2022

acarbonetto commented Dec 22, 2022

reta commented Dec 22, 2022

acarbonetto commented Dec 22, 2022

acarbonetto commented Dec 22, 2022

reta commented Dec 22, 2022

acarbonetto commented Dec 27, 2022

reta commented Sep 19, 2022 •

edited

Loading