[1.3.0][BUG] ARM64 build failed due to could not find netty-tcnative-2.0.25.Final-linux-aarch_64.jar

[peterzhuamazon]

2022-02-25 17:16:59 INFO     Recording plugins artifact for k-NN: plugins/opensearch-knn-1.3.0.0-SNAPSHOT.zip (from /tmp/tmpnqdj8jaf/k-NN/builds/plugins/opensearch-knn-1.3.0.0-SNAPSHOT.zip)
2022-02-25 17:16:59 INFO     Checked /tmp/tmpnqdj8jaf/k-NN/builds/plugins/opensearch-knn-1.3.0.0-SNAPSHOT.zip (1.3.0.0-SNAPSHOT)
2022-02-25 17:16:59 INFO     Building security
2022-02-25 17:16:59 INFO     Executing "git init" in /tmp/tmpnqdj8jaf/security
2022-02-25 17:16:59 INFO     Executing "git remote add origin https://github.com/opensearch-project/security.git" in /tmp/tmpnqdj8jaf/security
2022-02-25 17:16:59 INFO     Executing "git fetch --depth 1 origin main" in /tmp/tmpnqdj8jaf/security
2022-02-25 17:17:00 INFO     Executing "git checkout FETCH_HEAD" in /tmp/tmpnqdj8jaf/security
2022-02-25 17:17:00 INFO     Executing "git rev-parse HEAD" in /tmp/tmpnqdj8jaf/security
2022-02-25 17:17:00 INFO     Checked out https://github.com/opensearch-project/security.git@main into /tmp/tmpnqdj8jaf/security at fc3842ca806a2b9449f506a669a895e94549f1f6
2022-02-25 17:17:00 INFO     Executing "bash /var/jenkins/workspace/distribution-build-opensearch/scripts/components/security/build.sh -v 1.3.0 -p linux -a arm64 -s true -o builds" in /tmp/tmpnqdj8jaf/security
+ getopts :h:v:s:o:p:a: arg
+ case $arg in
+ VERSION=1.3.0
+ getopts :h:v:s:o:p:a: arg
+ case $arg in
+ PLATFORM=linux
+ getopts :h:v:s:o:p:a: arg
+ case $arg in
+ ARCHITECTURE=arm64
+ getopts :h:v:s:o:p:a: arg
+ case $arg in
+ SNAPSHOT=true
+ getopts :h:v:s:o:p:a: arg
+ case $arg in
+ OUTPUT=builds
+ getopts :h:v:s:o:p:a: arg
+ '[' -z 1.3.0 ']'
+ OPENSEARCH_RELEASE=1.3.0
+ [[ true == \t\r\u\e ]]
+ VERSION=1.3.0-SNAPSHOT
+ '[' -z builds ']'
++ cat plugin-descriptor.properties
++ grep '^version='
++ cut -d= -f2
++ sed s/-SNAPSHOT//
cat: plugin-descriptor.properties: No such file or directory
+ PLUGIN_VERSION=
+ [[ true == \t\r\u\e ]]
+ PLUGIN_VERSION=-SNAPSHOT
+ ./gradlew assemble --no-daemon --refresh-dependencies -DskipTests=true -Dopensearch.version=1.3.0-SNAPSHOT -Dopensearch_version=1.3.0-SNAPSHOT -Dbuild.snapshot=true
Downloading https://services.gradle.org/distributions/gradle-7.3-bin.zip
...........10%...........20%...........30%...........40%...........50%...........60%...........70%...........80%...........90%...........100%

Welcome to Gradle 7.3!

Here are the highlights of this release:
 - Easily declare new test suites in Java projects
 - Support for Java 17
 - Support for Scala 3

For more details see https://docs.gradle.org/7.3/release-notes.html

To honour the JVM settings for this build a single-use Daemon process will be forked. See https://docs.gradle.org/7.3/userguide/gradle_daemon.html#sec:disabling_the_daemon.
Daemon will be stopped at the end of the build 
> Task :createPluginDescriptor UP-TO-DATE
> Task :compileJava FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':compileJava'.
> Could not resolve all files for configuration ':compileClasspath'.
   > Could not find netty-tcnative-2.0.25.Final-linux-aarch_64.jar (io.netty:netty-tcnative:2.0.25.Final).
     Searched in the following locations:
         https://repo.maven.apache.org/maven2/io/netty/netty-tcnative/2.0.25.Final/netty-tcnative-2.0.25.Final-linux-aarch_64.jar

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org/

Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

See https://docs.gradle.org/7.3/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 26s
1 actionable task: 1 executed
2022-02-25 17:17:28 ERROR    Error building security, retry with: ./build.sh manifests/1.3.0/opensearch-1.3.0.yml --component security --snapshot
Traceback (most recent call last):
  File "./src/run_build.py", line 79, in <module>
    sys.exit(main())
  File "./src/run_build.py", line 67, in main
    builder.build(build_recorder)
  File "/var/jenkins/workspace/distribution-build-opensearch/src/build_workflow/builder_from_source.py", line 54, in build
    self.git_repo.execute(build_command)
  File "/var/jenkins/workspace/distribution-build-opensearch/src/git/git_repository.py", line 83, in execute
    subprocess.check_call(command, cwd=cwd, shell=True)
  File "/usr/local/lib/python3.7/subprocess.py", line 363, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'bash /var/jenkins/workspace/distribution-build-opensearch/scripts/components/security/build.sh -v 1.3.0 -p linux -a arm64 -s true -o builds' returned non-zero exit status 1.

[peterzhuamazon]

@davidlago @cliu123 @peternied please take a look at this as it has blocked 1.3.0 build for almost a week now.

cc: @bbarani.

Thanks.

[cliu123]

@peterzhuamazon Was security plugin 1.2.4 successfully built on ARM64? AFAIK, security plugin cannot be built on ARM64 since some dependencies(might not be limited toio.netty:netty-tcnative) are missing on ARM64, which has been a known issue for a while. io.netty:netty-tcnative is a part of netty that is just one of dependencies of security plugin, so this is actually out of security plugin's control.

[peterzhuamazon]

Testing with @cliu123 for a bit and realize even in 1.2.4 prior, netty-tcnative is not downloaded by mvn, before switch to gradle as buildtool in security.

I then proceed to remove netty-tcnative from build.gradle and rebuild on arm64 host, and build succeed.

$  ./gradlew assemble --no-daemon --refresh-dependencies -DskipTests=true -Dopensearch.version=1.3.0 -Dopensearch_version=1.3.0 -Dbuild.snapshot=true
[0.002s][error][cds] Unable to map CDS archive -- os::vm_allocation_granularity() expected: 65536 actual: 4096
To honour the JVM settings for this build a single-use Daemon process will be forked. See https://docs.gradle.org/7.3/userguide/gradle_daemon.html#sec:disabling_the_daemon.
Daemon will be stopped at the end of the build

Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

See https://docs.gradle.org/7.3/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 35s
8 actionable tasks: 5 executed, 3 up-to-date

$ ll target/
total 2272
drwxrwxr-x  2 ec2-user ec2-user       6 Feb 28 19:26 archive-tmp
drwxrwxr-x  5 ec2-user ec2-user     112 Feb 28 19:26 classes
drwxrwxr-x  3 ec2-user ec2-user      25 Feb 28 19:26 generated-sources
drwxrwxr-x  3 ec2-user ec2-user      30 Feb 28 19:26 generated-test-sources
drwxrwxr-x  2 ec2-user ec2-user      28 Feb 28 19:26 maven-archiver
drwxrwxr-x  3 ec2-user ec2-user      35 Feb 28 19:26 maven-status
-rw-rw-r--  1 ec2-user ec2-user 1123221 Feb 28 19:26 opensearch-security-1.2.4.0-SNAPSHOT.jar
-rw-rw-r--  1 ec2-user ec2-user 1194468 Feb 28 19:26 opensearch-security-1.2.4.0-SNAPSHOT-tests.jar
drwxrwxr-x  2 ec2-user ec2-user     258 Feb 28 19:26 releases
drwxrwxr-x 19 ec2-user ec2-user    4096 Feb 28 19:26 test-classes

$ uname -a
Linux 4.14.256-197.484.amzn2.aarch64 #1 SMP Tue Nov 30 00:18:02 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux

[peterzhuamazon]

This seems like to me that netty-tcnative is never being used as a dependency in security, thus we can remove it from build.gradle now.

Thanks.