[1.3.0][Distribution][BUG] Error building alerting components #312

zelinh · 2022-02-21T21:44:09Z

Currently our distribution-build-opensearch is broken.
Here is the context of error.

A problem occurred evaluating root project 'opensearch-alerting'.
> A problem occurred configuring project ':alerting'.
  > Cannot mutate content repository descriptor 'MavenLocal(file:/usr/share/opensearch/.m2/repository/)' after repository has been used

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help [at https://help.gradle.](https://help.gradle.org/)org

Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
S[ee https://docs.gradle.org/6.6.1/userguide/command_line_interface.html#sec:command_line_warni](https://docs.gradle.org/6.6.1/userguide/command_line_interface.html#sec:command_line_warnings)ngs

BUILD FAILED in 16s
2022-02-21 21:11:15 ERROR    Error building alerting, retry with: ./build.sh manifests/1.3.0/opensearch-1.3.0.yml --component alerting

To reproduce

Run ./build.sh manifests/1.3.0/opensearch-1.3.0.yml --component alerting --snapshot.

The text was updated successfully, but these errors were encountered:

qreshi · 2022-02-22T00:32:21Z

I was able to reproduce this with ./build.sh manifests/1.3.0/opensearch-1.3.0.yml from opensearch-build but I also tracked down the problematic call to the ./gradlew assemble call in Alerting's scripts/build.sh

The call with the arguments all filled out is the following:

./gradlew assemble --no-daemon --refresh-dependencies -DskipTests=true -Dopensearch.version=1.3.0 -Dbuild.snapshot=false -x ktlint

Strangely enough, when you remove the -Dopensearch.version argument, the assemble passes.

Given that the default on main for the version (in build.gradle) is:

opensearch_version = System.getProperty("opensearch.version", "1.3.0-SNAPSHOT")

And we then do:

version = "${opensearch_version}" - "-SNAPSHOT" + ".0"
if (isSnapshot) {
    version += "-SNAPSHOT"
}

It should still be ending up with the same version internally (1.3.0). Knowing this, is anyone able to reproduce this with another plugin?

tianleh · 2022-02-22T01:04:30Z

@qreshi I just tried removing alerting from manifests/1.3.0/opensearch-1.3.0.yml and re-run ./ci.sh manifests/1.3.0/opensearch-1.3.0.yml --snapshot which now succeeds.

If the issue of alerting cannot be fixed soon, could we for now remove alerting from manifests/1.3.0/opensearch-1.3.0.yml (and the test manifest)? This can unblock others who are updating other manifest files. The PR checks are failing due to the alerting error.

qreshi · 2022-02-22T01:08:08Z

@tianleh Yeah you can remove it for now if it's blocking anything. Although, doesn't Alerting publish the Notification subproject locally before moving on to Index Management? I tried removing Alerting from the manifest locally and when I re-ran ./build.sh manifests/1.3.0/opensearch-1.3.0.yml, it failed on Index Management.

qreshi · 2022-02-22T01:12:10Z

Also, I'm not really a gradle expert but looking up the gradle code, I saw this was where that error message was being generated from.

We actually have a reference to the repos in a couple places. I tried removing all references to repositories.gradle in the Alerting project and then the error became:

* What went wrong:
A problem occurred evaluating root project 'opensearch-alerting'.
> A problem occurred configuring project ':alerting'.
   > Cannot mutate content repository descriptor 'MavenRepo' after repository has been used

Perhaps someone with Gradle expertise can identify if this is a configuration problem.

tianleh · 2022-02-22T02:16:56Z

@tianleh Yeah you can remove it for now if it's blocking anything. Although, doesn't Alerting publish the Notification subproject locally before moving on to Index Management? I tried removing Alerting from the manifest locally and when I re-ran ./build.sh manifests/1.3.0/opensearch-1.3.0.yml, it failed on Index Management.

I will wait for this issue to be resolved given the complicated situation.

tianleh · 2022-02-22T03:38:30Z

Based on my observation, the CI command./ci.sh manifests/1.3.0/opensearch-1.3.0.yml --snapshot was successful on 2/19/2022 12pm but failed on 2/21/2022 12pm. Seems something happened between 2/19/2022 12pm and 2/21/2022 12pm. Hope this can help narrow down the scope of research a bit.

peterzhuamazon · 2022-02-22T20:16:04Z

Transfer the issue to alerting repo as this is directly pointing to their script errors.
Thanks.

qreshi · 2022-02-22T21:51:16Z

@peterzhuamazon Running that Alerting script with snapshot set to true succeeds:

./scripts/build.sh -v 1.3.0 -p linux -a x64 -s true -o artifacts
BUILD SUCCESSFUL in 7s

@tianleh was also saying how this was working a few days ago. Is this possibly an issue introduced from OpenSearch. If it's a misconfiguration from our side that's incompatible with some OpenSearch change we can fix it but it would help if we knew what it was. Can anyone from the OpenSearch side lend a hand?

qreshi · 2022-02-22T22:41:28Z

Here are multiple command configurations and whether they succeed or pass in case it helps:

Passes

./gradlew assemble -Dbuild.snapshot=true
./gradlew assemble -Dbuild.snapshot=false

./scripts/build.sh -v 1.3.0 -p linux -a x64 -s true -o artifacts
(which is running ./gradlew assemble --no-daemon --refresh-dependencies -DskipTests=true -Dopensearch.version=1.3.0-SNAPSHOT -Dbuild.snapshot=true -x ktlint)

Fails

./gradlew assemble -Dopensearch.version=1.3.0 -Dbuild.snapshot=true
./gradlew assemble -Dopensearch.version=1.3.0 -Dbuild.snapshot=false

./scripts/build.sh -v 1.3.0 -p linux -a x64 -s false -o artifacts
(which is running ./gradlew assemble --no-daemon --refresh-dependencies -DskipTests=true -Dopensearch.version=1.3.0 -Dbuild.snapshot=false -x ktlint)

The main trend seems to be whether we're using 1.3.0-SNAPSHOT or 1.3.0 dependencies. ~~Although ./gradlew assemble -Dbuild.snapshot=false seems to be the outlier since it passes but it should be using 1.3.0~~ (actually scratch that, it looks like we use the default for the "org.opensearch.gradle:build-tools:${opensearch_version}" in gradle so it's also using the snapshot build-tools dependency).

peterzhuamazon · 2022-02-22T23:25:51Z

@qreshi are you able to identify what exactly is the issue related to non-snapshot dependencies?

I just build alerting on my devdesktop and it runs to success.

% ./build.sh manifests/1.3.0/opensearch-1.3.0.yml --component alerting --platform linux
Installing dependencies in . ...
Installing dependencies from Pipfile.lock (16e10a)...
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/0 — 00:00:00
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
Running ./src/run_build.py manifests/1.3.0/opensearch-1.3.0.yml --component alerting --platform linux ...
2022-02-22 23:22:07 INFO     Building in /tmp/tmp236zq0bc
2022-02-22 23:22:07 INFO     Building OpenSearch (x64) into /opensearch-build/builds/opensearch
2022-02-22 23:22:07 INFO     Skipping OpenSearch
2022-02-22 23:22:07 INFO     Skipping common-utils
2022-02-22 23:22:07 INFO     Skipping job-scheduler
2022-02-22 23:22:07 INFO     Building alerting
2022-02-22 23:22:07 INFO     Executing "git init" in /tmp/tmp236zq0bc/alerting
2022-02-22 23:22:07 INFO     Executing "git remote add origin https://github.com/opensearch-project/alerting.git" in /tmp/tmp236zq0bc/alerting
2022-02-22 23:22:07 INFO     Executing "git fetch --depth 1 origin main" in /tmp/tmp236zq0bc/alerting
2022-02-22 23:22:08 INFO     Executing "git checkout FETCH_HEAD" in /tmp/tmp236zq0bc/alerting
2022-02-22 23:22:08 INFO     Executing "git rev-parse HEAD" in /tmp/tmp236zq0bc/alerting
2022-02-22 23:22:08 INFO     Checked out https://github.com/opensearch-project/alerting.git@main into /tmp/tmp236zq0bc/alerting at 3632bd47e0f552b18ebe0a1f5dcaf38d7d13311b
2022-02-22 23:22:08 INFO     Executing "bash /opensearch-build/scripts/components/alerting/build.sh -v 1.3.0 -p linux -a x64 -s false -o builds" in /tmp/tmp236zq0bc/alerting
+ getopts :h:v:s:o:p:a: arg
+ case $arg in
+ VERSION=1.3.0
+ getopts :h:v:s:o:p:a: arg
+ case $arg in
+ PLATFORM=linux
+ getopts :h:v:s:o:p:a: arg
+ case $arg in
+ ARCHITECTURE=x64
+ getopts :h:v:s:o:p:a: arg
+ case $arg in
+ SNAPSHOT=false
+ getopts :h:v:s:o:p:a: arg
+ case $arg in
+ OUTPUT=builds
+ getopts :h:v:s:o:p:a: arg
+ '[' -z 1.3.0 ']'
+ [[ false == \t\r\u\e ]]
+ '[' -z builds ']'
+ mkdir -p builds/plugins
+ ./gradlew assemble --no-daemon --refresh-dependencies -DskipTests=true -Dopensearch.version=1.3.0 -Dbuild.snapshot=false -x ktlint
To honour the JVM settings for this build a new JVM will be forked. Please consider using the daemon: https://docs.gradle.org/6.6.1/userguide/gradle_daemon.html.
Daemon will be stopped at the end of the build stopping after processing
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 6.6.1
  OS Info               : Linux 5.4.176-103.347.amzn2int.x86_64 (amd64)
  JDK Version           : 14 (AdoptOpenJDK)
  JAVA_HOME             : /usr/lib/jvm/adoptopenjdk-14-hotspot
  Random Testing Seed   : 623C3B765F29D1FF
  In FIPS 140 mode      : false
=======================================

> Task :alerting-notification:compileJava
Note: /tmp/tmp236zq0bc/alerting/notification/src/main/java/org/opensearch/alerting/destination/Notification.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :alerting:compileKotlin
w: /tmp/tmp236zq0bc/alerting/alerting/src/main/kotlin/org/opensearch/alerting/model/action/ActionExecutionScope.kt: (141, 17): Parameter 'sin' is never used

Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.6.1/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 48s
16 actionable tasks: 16 executed
++ find . -path '*build/distributions/*.zip'
+ zipPath=./alerting/build/distributions/opensearch-alerting-1.3.0.0.zip
++ dirname ./alerting/build/distributions/opensearch-alerting-1.3.0.0.zip
+ distributions=./alerting/build/distributions
+ echo 'COPY ./alerting/build/distributions/*.zip'
COPY ./alerting/build/distributions/*.zip
+ cp ./alerting/build/distributions/opensearch-alerting-1.3.0.0.zip ./builds/plugins
+ ./gradlew publishShadowPublicationToMavenLocal -Dopensearch.version=1.3.0 -Dbuild.snapshot=false -x ktlint
Starting a Gradle Daemon (subsequent builds will be faster)
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 6.6.1
  OS Info               : Linux 5.4.176-103.347.amzn2int.x86_64 (amd64)
  JDK Version           : 14 (AdoptOpenJDK)
  JAVA_HOME             : /usr/lib/jvm/adoptopenjdk-14-hotspot
  Random Testing Seed   : B728BF9FF8D58B9E
  In FIPS 140 mode      : false
=======================================

Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.6.1/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 9s
6 actionable tasks: 5 executed, 1 up-to-date
+ ./gradlew publishShadowPublicationToStagingRepository -Dopensearch.version=1.3.0 -Dbuild.snapshot=false
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 6.6.1
  OS Info               : Linux 5.4.176-103.347.amzn2int.x86_64 (amd64)
  JDK Version           : 14 (AdoptOpenJDK)
  JAVA_HOME             : /usr/lib/jvm/adoptopenjdk-14-hotspot
  Random Testing Seed   : DA1636DB714D4B16
  In FIPS 140 mode      : false
=======================================

Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.6.1/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 1s
6 actionable tasks: 2 executed, 4 up-to-date
+ mkdir -p builds/maven/org/opensearch
+ cp -r ./build/local-staging-repo/org/opensearch/. builds/maven/org/opensearch
2022-02-22 23:23:08 INFO     Recording maven artifact for alerting: maven/org/opensearch/notification/maven-metadata.xml.sha1 (from /tmp/tmp236zq0bc/alerting/builds/maven/org/opensearch/notification/maven-metadata.xml.sha1)
2022-02-22 23:23:08 INFO     Recording maven artifact for alerting: maven/org/opensearch/notification/maven-metadata.xml.sha512 (from /tmp/tmp236zq0bc/alerting/builds/maven/org/opensearch/notification/maven-metadata.xml.sha512)
2022-02-22 23:23:08 INFO     Recording maven artifact for alerting: maven/org/opensearch/notification/maven-metadata.xml.md5 (from /tmp/tmp236zq0bc/alerting/builds/maven/org/opensearch/notification/maven-metadata.xml.md5)
2022-02-22 23:23:08 INFO     Recording maven artifact for alerting: maven/org/opensearch/notification/maven-metadata.xml.sha256 (from /tmp/tmp236zq0bc/alerting/builds/maven/org/opensearch/notification/maven-metadata.xml.sha256)
2022-02-22 23:23:08 INFO     Recording maven artifact for alerting: maven/org/opensearch/notification/maven-metadata.xml (from /tmp/tmp236zq0bc/alerting/builds/maven/org/opensearch/notification/maven-metadata.xml)
2022-02-22 23:23:08 INFO     Recording maven artifact for alerting: maven/org/opensearch/notification/1.3.0.0/notification-1.3.0.0.jar.sha1 (from /tmp/tmp236zq0bc/alerting/builds/maven/org/opensearch/notification/1.3.0.0/notification-1.3.0.0.jar.sha1)
2022-02-22 23:23:08 INFO     Recording maven artifact for alerting: maven/org/opensearch/notification/1.3.0.0/notification-1.3.0.0-javadoc.jar.sha1 (from /tmp/tmp236zq0bc/alerting/builds/maven/org/opensearch/notification/1.3.0.0/notification-1.3.0.0-javadoc.jar.sha1)
2022-02-22 23:23:08 INFO     Recording maven artifact for alerting: maven/org/opensearch/notification/1.3.0.0/notification-1.3.0.0.pom.sha512 (from /tmp/tmp236zq0bc/alerting/builds/maven/org/opensearch/notification/1.3.0.0/notification-1.3.0.0.pom.sha512)
2022-02-22 23:23:08 INFO     Recording maven artifact for alerting: maven/org/opensearch/notification/1.3.0.0/notification-1.3.0.0.pom (from /tmp/tmp236zq0bc/alerting/builds/maven/org/opensearch/notification/1.3.0.0/notification-1.3.0.0.pom)
2022-02-22 23:23:08 INFO     Recording maven artifact for alerting: maven/org/opensearch/notification/1.3.0.0/notification-1.3.0.0-sources.jar.sha512 (from /tmp/tmp236zq0bc/alerting/builds/maven/org/opensearch/notification/1.3.0.0/notification-1.3.0.0-sources.jar.sha512)
2022-02-22 23:23:08 INFO     Recording maven artifact for alerting: maven/org/opensearch/notification/1.3.0.0/notification-1.3.0.0-javadoc.jar.md5 (from /tmp/tmp236zq0bc/alerting/builds/maven/org/opensearch/notification/1.3.0.0/notification-1.3.0.0-javadoc.jar.md5)
2022-02-22 23:23:08 INFO     Recording maven artifact for alerting: maven/org/opensearch/notification/1.3.0.0/notification-1.3.0.0.jar.sha256 (from /tmp/tmp236zq0bc/alerting/builds/maven/org/opensearch/notification/1.3.0.0/notification-1.3.0.0.jar.sha256)
2022-02-22 23:23:08 INFO     Recording maven artifact for alerting: maven/org/opensearch/notification/1.3.0.0/notification-1.3.0.0-javadoc.jar.sha512 (from /tmp/tmp236zq0bc/alerting/builds/maven/org/opensearch/notification/1.3.0.0/notification-1.3.0.0-javadoc.jar.sha512)
2022-02-22 23:23:08 INFO     Recording maven artifact for alerting: maven/org/opensearch/notification/1.3.0.0/notification-1.3.0.0.jar.sha512 (from /tmp/tmp236zq0bc/alerting/builds/maven/org/opensearch/notification/1.3.0.0/notification-1.3.0.0.jar.sha512)
2022-02-22 23:23:08 INFO     Recording maven artifact for alerting: maven/org/opensearch/notification/1.3.0.0/notification-1.3.0.0-javadoc.jar (from /tmp/tmp236zq0bc/alerting/builds/maven/org/opensearch/notification/1.3.0.0/notification-1.3.0.0-javadoc.jar)
2022-02-22 23:23:08 INFO     Checked /tmp/tmp236zq0bc/alerting/builds/maven/org/opensearch/notification/1.3.0.0/notification-1.3.0.0-javadoc.jar (N/A)
2022-02-22 23:23:08 INFO     Recording maven artifact for alerting: maven/org/opensearch/notification/1.3.0.0/notification-1.3.0.0.pom.sha1 (from /tmp/tmp236zq0bc/alerting/builds/maven/org/opensearch/notification/1.3.0.0/notification-1.3.0.0.pom.sha1)
2022-02-22 23:23:08 INFO     Recording maven artifact for alerting: maven/org/opensearch/notification/1.3.0.0/notification-1.3.0.0.pom.md5 (from /tmp/tmp236zq0bc/alerting/builds/maven/org/opensearch/notification/1.3.0.0/notification-1.3.0.0.pom.md5)
2022-02-22 23:23:08 INFO     Recording maven artifact for alerting: maven/org/opensearch/notification/1.3.0.0/notification-1.3.0.0-sources.jar (from /tmp/tmp236zq0bc/alerting/builds/maven/org/opensearch/notification/1.3.0.0/notification-1.3.0.0-sources.jar)
2022-02-22 23:23:08 INFO     Checked /tmp/tmp236zq0bc/alerting/builds/maven/org/opensearch/notification/1.3.0.0/notification-1.3.0.0-sources.jar (N/A)
2022-02-22 23:23:08 INFO     Recording maven artifact for alerting: maven/org/opensearch/notification/1.3.0.0/notification-1.3.0.0-sources.jar.sha256 (from /tmp/tmp236zq0bc/alerting/builds/maven/org/opensearch/notification/1.3.0.0/notification-1.3.0.0-sources.jar.sha256)
2022-02-22 23:23:08 INFO     Recording maven artifact for alerting: maven/org/opensearch/notification/1.3.0.0/notification-1.3.0.0-javadoc.jar.sha256 (from /tmp/tmp236zq0bc/alerting/builds/maven/org/opensearch/notification/1.3.0.0/notification-1.3.0.0-javadoc.jar.sha256)
2022-02-22 23:23:08 INFO     Recording maven artifact for alerting: maven/org/opensearch/notification/1.3.0.0/notification-1.3.0.0-sources.jar.sha1 (from /tmp/tmp236zq0bc/alerting/builds/maven/org/opensearch/notification/1.3.0.0/notification-1.3.0.0-sources.jar.sha1)
2022-02-22 23:23:08 INFO     Recording maven artifact for alerting: maven/org/opensearch/notification/1.3.0.0/notification-1.3.0.0.pom.sha256 (from /tmp/tmp236zq0bc/alerting/builds/maven/org/opensearch/notification/1.3.0.0/notification-1.3.0.0.pom.sha256)
2022-02-22 23:23:08 INFO     Recording maven artifact for alerting: maven/org/opensearch/notification/1.3.0.0/notification-1.3.0.0-sources.jar.md5 (from /tmp/tmp236zq0bc/alerting/builds/maven/org/opensearch/notification/1.3.0.0/notification-1.3.0.0-sources.jar.md5)
2022-02-22 23:23:08 INFO     Recording maven artifact for alerting: maven/org/opensearch/notification/1.3.0.0/notification-1.3.0.0.jar (from /tmp/tmp236zq0bc/alerting/builds/maven/org/opensearch/notification/1.3.0.0/notification-1.3.0.0.jar)
2022-02-22 23:23:08 INFO     Checked /tmp/tmp236zq0bc/alerting/builds/maven/org/opensearch/notification/1.3.0.0/notification-1.3.0.0.jar (N/A)
2022-02-22 23:23:08 INFO     Recording maven artifact for alerting: maven/org/opensearch/notification/1.3.0.0/notification-1.3.0.0.jar.md5 (from /tmp/tmp236zq0bc/alerting/builds/maven/org/opensearch/notification/1.3.0.0/notification-1.3.0.0.jar.md5)
2022-02-22 23:23:08 INFO     Recording plugins artifact for alerting: plugins/opensearch-alerting-1.3.0.0.zip (from /tmp/tmp236zq0bc/alerting/builds/plugins/opensearch-alerting-1.3.0.0.zip)
2022-02-22 23:23:08 INFO     Checked /tmp/tmp236zq0bc/alerting/builds/plugins/opensearch-alerting-1.3.0.0.zip (1.3.0.0)
2022-02-22 23:23:08 INFO     Skipping asynchronous-search
2022-02-22 23:23:08 INFO     Skipping index-management
2022-02-22 23:23:08 INFO     Skipping k-NN
2022-02-22 23:23:08 INFO     Skipping security
2022-02-22 23:23:08 INFO     Skipping performance-analyzer
2022-02-22 23:23:08 INFO     Skipping anomaly-detection
2022-02-22 23:23:08 INFO     Skipping cross-cluster-replication
2022-02-22 23:23:08 INFO     Skipping sql
2022-02-22 23:23:08 INFO     Skipping dashboards-reports
2022-02-22 23:23:08 INFO     Skipping opensearch-observability
2022-02-22 23:23:08 INFO     Created build manifest /opensearch-build/builds/opensearch/manifest.yml
2022-02-22 23:23:08 INFO     Done.

peterzhuamazon · 2022-02-22T23:31:59Z

Seems like this is something changes in the core as @qreshi mentioned, my build is based on local .m2, thus pass.
The problem is, if the build is failing then no new snapshots publish to sonatype, but if we dont publish then alerting will keep failing since Jenkins build is separating local maven and plugin build to two different containers.

It is technically blocking itself at this point.

zelinh · 2022-02-22T23:46:29Z

Share my story on this issue:
I initially just run ./build.sh manifests/1.3.0/opensearch-1.3.0.yml --component alerting --snapshot in my local mac env and it failed with the same issue.
I then tried to run the same command in CI container where I have run build.sh before; but it passed.
I re-tried with ./build.sh manifests/1.3.0/opensearch-1.3.0.yml in container and it failed on alerting with this issue. After this, ./build.sh manifests/1.3.0/opensearch-1.3.0.yml --component alerting --snapshot will always fail on the same issue.
It seems like some changes on core side might causing the issue, but will need more investigations.

Hope this will provide more insights. Thanks.

qreshi · 2022-02-23T01:43:33Z

@peterzhuamazon No, I'm not really sure what the discrepancy is.

Would removing Alerting from the manifest to unblock the build and then reintroducing it fix the problem?

peterzhuamazon · 2022-02-23T16:23:52Z

@peterzhuamazon No, I'm not really sure what the discrepancy is.

Would removing Alerting from the manifest to unblock the build and then reintroducing it fix the problem?

Sure, please PR.
Thanks.

qreshi · 2022-02-23T20:30:53Z

@peterzhuamazon I tried to re-run ./build.sh manifests/1.3.0/opensearch-1.3.0.yml locally after the manifest change before creating a PR for it and k-NN fails after it gets passed Alerting:

2022-02-23 20:23:09 INFO     Executing "bash /tmp/tmpsg5sn_uf/k-NN/scripts/build.sh -v 1.3.0 -p linux -a x64 -s false -o builds" in /tmp/tmpsg5sn_uf/k-NN
+ getopts :h:v:s:o:p:a: arg
+ case $arg in
+ VERSION=1.3.0
+ getopts :h:v:s:o:p:a: arg
+ case $arg in
+ PLATFORM=linux
+ getopts :h:v:s:o:p:a: arg
+ case $arg in
+ ARCHITECTURE=x64
+ getopts :h:v:s:o:p:a: arg
+ case $arg in
+ SNAPSHOT=false
+ getopts :h:v:s:o:p:a: arg
+ case $arg in
+ OUTPUT=builds
+ getopts :h:v:s:o:p:a: arg
+ '[' -z 1.3.0 ']'
+ [[ false == \t\r\u\e ]]
+ '[' -z builds ']'
+ work_dir=/tmp/tmpsg5sn_uf/k-NN
+ git submodule update --init -- jni/external/nmslib
Submodule 'jni/external/nmslib' (https://github.com/nmslib/nmslib.git) registered for path 'jni/external/nmslib'
Cloning into '/tmp/tmpsg5sn_uf/k-NN/jni/external/nmslib'...
Submodule path 'jni/external/nmslib': checked out 'a2d6624e1315402662025debfdd614b505d9c3ef'
+ git submodule update --init -- jni/external/faiss
Submodule 'jni/external/faiss' (https://github.com/facebookresearch/faiss.git) registered for path 'jni/external/faiss'
Cloning into '/tmp/tmpsg5sn_uf/k-NN/jni/external/faiss'...
Submodule path 'jni/external/faiss': checked out '88eabe97f96d0c0964dfa075f74373c64d46da80'
+ cd jni
+ '[' x64 = x64 ']'
+ sed -i -e s/-march=native/-march=x86-64/g external/nmslib/similarity_search/CMakeLists.txt
+ '[' x64 = arm64 ']'
+ '[' /home/qreshi/workspace/jdk/src/OpenJDK14/build/jdk-14 = '' ']'
+ cmake .
/tmp/tmpsg5sn_uf/k-NN/scripts/build.sh: line 93: cmake: command not found
2022-02-23 20:23:18 ERROR    Error building k-NN, retry with: ./build.sh manifests/1.3.0/opensearch-1.3.0.yml --component k-NN
Traceback (most recent call last):
  File "./src/run_build.py", line 79, in <module>
    sys.exit(main())
  File "./src/run_build.py", line 67, in main
    builder.build(build_recorder)
  File "/local/home/qreshi/workspace/opensearch-build/src/build_workflow/builder_from_source.py", line 54, in build
    self.git_repo.execute(build_command)
  File "/local/home/qreshi/workspace/opensearch-build/src/git/git_repository.py", line 83, in execute
    subprocess.check_call(command, cwd=cwd, shell=True)
  File "/usr/lib64/python3.7/subprocess.py", line 363, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'bash /tmp/tmpsg5sn_uf/k-NN/scripts/build.sh -v 1.3.0 -p linux -a x64 -s false -o builds' returned non-zero exit status 127.

It can be reproduced with ./build.sh manifests/1.3.0/opensearch-1.3.0.yml --component k-NN. Is anyone else seeing this? If so maybe we can reach out to the k-NN team to debug before proceeding so the build doesn't get blocked on something else.

qreshi · 2022-02-23T20:32:29Z

Actually, taking a closer look, it looks like I'm just missing cmake where I'm running this so this can probably be disregarded.

qreshi · 2022-02-23T23:26:33Z

I tried to run locally again since Peter confirmed that the build ran.

The assembly command with snapshot=false is still failing:

+ ./gradlew assemble --no-daemon --refresh-dependencies -DskipTests=true -Dopensearch.version=1.3.0 -Dbuild.snapshot=false -x ktlint
To honour the JVM settings for this build a new JVM will be forked. Please consider using the daemon: https://docs.gradle.org/6.6.1/userguide/gradle_daemon.html.
Daemon will be stopped at the end of the build stopping after processing

FAILURE: Build failed with an exception.

* Where:
Build file '/local/home/qreshi/workspace/opensearch-alerting/build.gradle' line: 89

* What went wrong:
A problem occurred evaluating root project 'opensearch-alerting'.
> A problem occurred configuring project ':alerting'.
   > Cannot mutate content repository descriptor 'MavenLocal(file:/home/qreshi/.m2/repository/)' after repository has been used

And now, strangely enough, even with the snapshot dependency, it's failing:

% ./gradlew assemble --no-daemon --refresh-dependencies -DskipTests=true -Dbuild.snapshot=true -x ktlint
To honour the JVM settings for this build a new JVM will be forked. Please consider using the daemon: https://docs.gradle.org/6.6.1/userguide/gradle_daemon.html.
Daemon will be stopped at the end of the build stopping after processing

FAILURE: Build failed with an exception.

* Where:
Build file '/local/home/qreshi/workspace/opensearch-alerting/build.gradle' line: 89

* What went wrong:
A problem occurred evaluating root project 'opensearch-alerting'.
> A problem occurred configuring project ':alerting'.
   > Cannot mutate content repository descriptor 'MavenLocal(file:/home/qreshi/.m2/repository/)' after repository has been used

I just checked with ./gradlew build and that's failing now too.

@peterzhuamazon Can you check what the recent changes were to the OpenSearch snapshot repo? At this point even Alerting development is blocked.

peterzhuamazon · 2022-02-24T00:35:32Z

cc @CEHENKLE this seems like a major blocker at this point.
Tho interesting enough if you manually build in mavenlocal the run will pass.

Need some core engineers to help debug this.

peterzhuamazon · 2022-02-24T00:36:06Z

Also, recently I notice the run does not show error during builds and the upload complete, but the pipeline still show as failure. Will need to dig more into it to confirm what is happening.

Thanks.

CEHENKLE · 2022-02-24T00:52:31Z

Rabi is going to take a look.

adnapibar · 2022-02-24T19:47:05Z

It looks like this commit caused this issue

I tried building locally without this commit and the issue disappears.

Next step - need to look into the changes as part of the commit and find a fix if possible or revert this commit

adnapibar · 2022-02-24T23:59:01Z

@qreshi @zelinh @peterzhuamazon reverted the change in core that's probably causing the issue. Can you try building with alerting now with the latest snapshot when available?

@Rishikesh1159 will investigate the change to figure out a fix as he originally authored the PR and has more knowledge on it.

qreshi · 2022-02-25T02:12:50Z

@adnapibar I tried to build locally again and it looks like the snapshot version is working again. So it must have been that commit.

peterzhuamazon · 2022-02-25T19:07:14Z

I have figured out the issue specifically caused the failure is on security plugin in arm64 build.
opensearch-project/security#1647

peterzhuamazon · 2022-03-01T18:23:02Z

1.3.0 build success for a whole night, close this now.

peterzhuamazon · 2022-03-01T18:23:25Z

Thanks @qreshi @adnapibar @CEHENKLE

zelinh added bug Something isn't working v1.3.0 labels Feb 21, 2022

zelinh mentioned this issue Feb 22, 2022

[Bug]: ci.sh fails for manifests/1.3.0/opensearch-1.3.0.yml opensearch-project/opensearch-build#1661

Closed

peterzhuamazon transferred this issue from opensearch-project/opensearch-build Feb 22, 2022

peterzhuamazon changed the title ~~[Distribution-build-opensearch Breaks] Error building alerting components~~ [1.3.0][Distribution][BUG] Error building alerting components Feb 22, 2022

qreshi mentioned this issue Feb 23, 2022

Temporarily remove Alerting and Index Management from 1.3.0 manifest opensearch-project/opensearch-build#1669

Merged

1 task

adnapibar mentioned this issue Feb 24, 2022

Revert Backport 1.x PR #2191 opensearch-project/OpenSearch#2243

Merged

1 task

Rishikesh1159 mentioned this issue Feb 25, 2022

Revert "Override Default Distribution Download Url with Custom Distribution Url When User Passes a Url" opensearch-project/OpenSearch#2254

Closed

Rishikesh1159 mentioned this issue Feb 25, 2022

Revert "Override Default Distribution Download Url with Custom Distribution Url When User Passes a Url" opensearch-project/OpenSearch#2256

Merged

5 tasks

qreshi mentioned this issue Mar 1, 2022

Add back Alerting and Index Management to 1.3.0 manifest opensearch-project/opensearch-build#1677

Merged

1 task

Rishikesh1159 mentioned this issue Mar 1, 2022

Override Distribution Downloader to download opensearch-min artifact from a custom url opensearch-project/OpenSearch#1240

Closed

peterzhuamazon closed this as completed Mar 1, 2022

peterzhuamazon self-assigned this Mar 1, 2022

Rishikesh1159 mentioned this issue Mar 9, 2022

Override Default Distribution Download Url with Custom Distribution Url when it is passed from Plugin opensearch-project/OpenSearch#2420

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[1.3.0][Distribution][BUG] Error building alerting components #312

[1.3.0][Distribution][BUG] Error building alerting components #312

zelinh commented Feb 21, 2022 •

edited

Loading

qreshi commented Feb 22, 2022

tianleh commented Feb 22, 2022

qreshi commented Feb 22, 2022

qreshi commented Feb 22, 2022

tianleh commented Feb 22, 2022

tianleh commented Feb 22, 2022

peterzhuamazon commented Feb 22, 2022

qreshi commented Feb 22, 2022

qreshi commented Feb 22, 2022 •

edited

Loading

peterzhuamazon commented Feb 22, 2022 •

edited

Loading

peterzhuamazon commented Feb 22, 2022

zelinh commented Feb 22, 2022

qreshi commented Feb 23, 2022

peterzhuamazon commented Feb 23, 2022

qreshi commented Feb 23, 2022

qreshi commented Feb 23, 2022

qreshi commented Feb 23, 2022

peterzhuamazon commented Feb 24, 2022

peterzhuamazon commented Feb 24, 2022

CEHENKLE commented Feb 24, 2022

adnapibar commented Feb 24, 2022

adnapibar commented Feb 24, 2022

qreshi commented Feb 25, 2022

peterzhuamazon commented Feb 25, 2022

peterzhuamazon commented Mar 1, 2022

peterzhuamazon commented Mar 1, 2022

[1.3.0][Distribution][BUG] Error building alerting components #312

[1.3.0][Distribution][BUG] Error building alerting components #312

Comments

zelinh commented Feb 21, 2022 • edited Loading

To reproduce

qreshi commented Feb 22, 2022

tianleh commented Feb 22, 2022

qreshi commented Feb 22, 2022

qreshi commented Feb 22, 2022

tianleh commented Feb 22, 2022

tianleh commented Feb 22, 2022

peterzhuamazon commented Feb 22, 2022

qreshi commented Feb 22, 2022

qreshi commented Feb 22, 2022 • edited Loading

Passes

Fails

peterzhuamazon commented Feb 22, 2022 • edited Loading

peterzhuamazon commented Feb 22, 2022

zelinh commented Feb 22, 2022

qreshi commented Feb 23, 2022

peterzhuamazon commented Feb 23, 2022

qreshi commented Feb 23, 2022

qreshi commented Feb 23, 2022

qreshi commented Feb 23, 2022

peterzhuamazon commented Feb 24, 2022

peterzhuamazon commented Feb 24, 2022

CEHENKLE commented Feb 24, 2022

adnapibar commented Feb 24, 2022

adnapibar commented Feb 24, 2022

qreshi commented Feb 25, 2022

peterzhuamazon commented Feb 25, 2022

peterzhuamazon commented Mar 1, 2022

peterzhuamazon commented Mar 1, 2022

zelinh commented Feb 21, 2022 •

edited

Loading

qreshi commented Feb 22, 2022 •

edited

Loading

peterzhuamazon commented Feb 22, 2022 •

edited

Loading