[BUG] Flow-framework build failed because no opensearch-security dependency #410

zelinh · 2024-01-17T01:37:58Z

What is the bug?

Currently I noticed the distribution bundle build for 2.12.0 failed on flow-framework plugin because it can't find opensearch-security dependency after this introduced

flow-framework/build.gradle

Line 164 in f27f2d2

    
           zipArchive group: 'org.opensearch.plugin', name:'opensearch-security', version: "${opensearch_build}"

* Where:
Build file '/tmp/tmpda1j575p/flow-framework/build.gradle' line: 348

* What went wrong:
A problem occurred evaluating root project 'opensearch-flow-framework'.
> Could not resolve all files for configuration ':zipArchive'.
   > Could not find org.opensearch.plugin:opensearch-security:2.12.0.0.
     Searched in the following locations:
       - https://repo.maven.apache.org/maven2/org/opensearch/plugin/opensearch-security/2.12.0.0/opensearch-security-2.12.0.0.pom
       - file:/home/ci-runner/.m2/repository/org/opensearch/plugin/opensearch-security/2.12.0.0/opensearch-security-2.12.0.0.pom
       - https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/plugin/opensearch-security/2.12.0.0/opensearch-security-2.12.0.0.pom
       - https://plugins.gradle.org/m2/org/opensearch/plugin/opensearch-security/2.12.0.0/opensearch-security-2.12.0.0.pom
     Required by:
         project :

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
> Get more help at https://help.gradle.org.

BUILD FAILED in 11s

More error context can be found here.

There are two approaches that we think could resolve this:

Since it says there are test time dependency, when we run build, we should exclude these test dependencies.
You could create a customized build script just like how k-NN did here. https://github.com/opensearch-project/k-NN/blob/7900dbb339ab56a56a09cf07c50381aeba084cbb/scripts/build.sh#L121 and disable any test time dependency using -x integTest
You could copy our default build script from https://github.com/opensearch-project/opensearch-build/blob/main/scripts/default/opensearch/build.sh to your repo and made customization on that.
Ask security repo to publish to Maven Local when security plugin is built.
Currently security plugin never publishes Maven local and there isn't non-snapshot plugin available on Sonatype. https://github.com/opensearch-project/security/blob/4a9006840be3430bcb704a0e39507dc289e99a56/scripts/build.sh#L80

The text was updated successfully, but these errors were encountered:

owaiskazi19 · 2024-01-18T01:15:16Z

@peternied / @cwperks / @DarshitChanpura any idea here. We have already raised the PR based on the solution provided but still fails with the above error present in the description.

DarshitChanpura · 2024-01-18T19:04:15Z

Seems like the regular image is available in any of the mentioned locations. I confirmed by manually checking all locations except file:/home/ci-runner/.m2/repository/org/opensearch/plugin/opensearch-security/2.12.0.0/opensearch-security-2.12.0.0.pom.

sonatype has the snapshot image: https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/plugin/opensearch-security/2.12.0.0-SNAPSHOT/

A possible solution is to compile on the fly and publish it to maven local.

cwperks · 2024-01-18T19:20:13Z

@owaiskazi19 Based on 2) in the original post, I think the security plugin can add:

./gradlew publishToMavenLocal -Dopensearch.version=$VERSION -Dbuild.snapshot=$SNAPSHOT -Dbuild.version_qualifier=$QUALIFIER

to scripts/build.sh. I will raise a PR for this in the security repo.

peternied · 2024-01-18T19:27:46Z

@zelinh @owaiskazi19 ~~The maven artifact isn't found because security hasn't published a release build for 2.12.0, why isn't the workflow using snapshot builds?~~

NVM, it standard practice to public to maven local during builds in jenkins - makes sense to following that trend.

cwperks · 2024-01-19T15:54:58Z

@owaiskazi19 @vibrantvarun I'm not exactly sure how zipArchive resolution works at integTest time.

I'm looking at 2 examples of security-analytics and skills repo that have zipArchive dependencies on plugins other than job-scheduler:

The plugins that these repos have zipArchive dependencies on do not publish their zips to maven local for the build (same with security). Would Flow-Framework be able to use the example from security-analytics to resolve the zipArchive dependency on the security repo for the integ tests?

joshpalis · 2024-01-19T18:30:32Z

@cwperks I've modified how we install our plugins into our integration test cluster, similar to how the security analytics plugin's build.gradle integTest task is configured. Testing with a non-existent version, attempting to run with security disabled (in which we do not install the security zip into our integration test cluster) still fails with the same issue :

FAILURE: Build failed with an exception.

* Where:
Build file '/local/home/jpalis/repos/delete/flow-framework/build.gradle' line: 347

* What went wrong:
Could not determine the dependencies of task ':run'.
> Could not resolve all files for configuration ':zipArchive'.
   > Could not find org.opensearch.plugin:opensearch-security:3.0.0.1-SNAPSHOT.
     Searched in the following locations:
       - https://repo.maven.apache.org/maven2/org/opensearch/plugin/opensearch-security/3.0.0.1-SNAPSHOT/maven-metadata.xml
       - https://repo.maven.apache.org/maven2/org/opensearch/plugin/opensearch-security/3.0.0.1-SNAPSHOT/opensearch-security-3.0.0.1-SNAPSHOT.pom
       - file:/home/jpalis/.m2/repository/org/opensearch/plugin/opensearch-security/3.0.0.1-SNAPSHOT/maven-metadata.xml
       - file:/home/jpalis/.m2/repository/org/opensearch/plugin/opensearch-security/3.0.0.1-SNAPSHOT/opensearch-security-3.0.0.1-SNAPSHOT.pom
       - https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/plugin/opensearch-security/3.0.0.1-SNAPSHOT/maven-metadata.xml
       - https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/plugin/opensearch-security/3.0.0.1-SNAPSHOT/opensearch-security-3.0.0.1-SNAPSHOT.pom
       - https://plugins.gradle.org/m2/org/opensearch/plugin/opensearch-security/3.0.0.1-SNAPSHOT/maven-metadata.xml
       - https://plugins.gradle.org/m2/org/opensearch/plugin/opensearch-security/3.0.0.1-SNAPSHOT/opensearch-security-3.0.0.1-SNAPSHOT.pom
     Required by:
         project :

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
> Get more help at https://help.gradle.org.

joshpalis · 2024-01-19T18:38:54Z

@cwperks Interestingly, conditionally including the security zip archive dependency resolves this issue. For example :

if(System.getProperty('security.enabled') == 'true'){
        zipArchive group: 'org.opensearch.plugin', name:'opensearch-security', version: "${opensearch_build}"
    }

Testing with a dummy version for security 3.0.0.1-SNAPSHOT :
Running ./gradlew run (security disabled ) passes :

./gradlew run
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 8.5
  OS Info               : Linux 5.10.205-172.804.amzn2int.x86_64 (amd64)
  JDK Version           : 17 (OpenJDK)
  JAVA_HOME             : /opt/jdk-17
  Random Testing Seed   : 6D5DEDC6EFAAA0A6
  In FIPS 140 mode      : false
=======================================

> Task :run
[2024-01-19T18:31:46.620268821Z] [BUILD] Configuring custom cluster specific distro directory: /local/home/jpalis/repos/delete/flow-framework/build/testclusters/integTest-0/distro/3.0.0-ARCHIVE
[2024-01-19T18:31:46.660840999Z] [BUILD] Copying additional config files from distro [/local/home/jpalis/repos/delete/flow-framework/build/testclusters/integTest-0/distro/3.0.0-ARCHIVE/config/jvm.options.d, /local/home/jpalis/repos/delete/flow-framework/build/testclusters/integTest-0/distro/3.0.0-ARCHIVE/config/opensearch.yml, /local/home/jpalis/repos/delete/flow-framework/build/testclusters/integTest-0/distro/3.0.0-ARCHIVE/config/jvm.options, /local/home/jpalis/repos/delete/flow-framework/build/testclusters/integTest-0/distro/3.0.0-ARCHIVE/config/log4j2.properties]
[2024-01-19T18:31:46.661784366Z] [BUILD] installing 2 plugins in a single transaction
[2024-01-19T18:31:51.590765017Z] [BUILD] installed plugins
[2024-01-19T18:31:51.590878532Z] [BUILD] Creating opensearch keystore with password set to []
[2024-01-19T18:31:52.294406348Z] [BUILD] Starting OpenSearch process
<===========--> 88% EXECUTING [7s]
> :run
^C%

Running with security enabled (dummy version value), expectedly fails :

./gradlew run -Dsecurity.enabled=true 

> Configure project :
Download https://raw.githubusercontent.com/opensearch-project/security/main/bwc-test/src/test/resources/security/esnode.pem
Download https://raw.githubusercontent.com/opensearch-project/security/main/bwc-test/src/test/resources/security/esnode-key.pem
Download https://raw.githubusercontent.com/opensearch-project/security/main/bwc-test/src/test/resources/security/root-ca.pem

FAILURE: Build failed with an exception.

* Where:
Build file '/local/home/jpalis/repos/delete/flow-framework/build.gradle' line: 350

* What went wrong:
Could not determine the dependencies of task ':run'.
> Could not resolve all files for configuration ':zipArchive'.
   > Could not find org.opensearch.plugin:opensearch-security:3.0.0.1-SNAPSHOT.
     Searched in the following locations:
       - https://repo.maven.apache.org/maven2/org/opensearch/plugin/opensearch-security/3.0.0.1-SNAPSHOT/maven-metadata.xml
       - https://repo.maven.apache.org/maven2/org/opensearch/plugin/opensearch-security/3.0.0.1-SNAPSHOT/opensearch-security-3.0.0.1-SNAPSHOT.pom
       - file:/home/jpalis/.m2/repository/org/opensearch/plugin/opensearch-security/3.0.0.1-SNAPSHOT/maven-metadata.xml
       - file:/home/jpalis/.m2/repository/org/opensearch/plugin/opensearch-security/3.0.0.1-SNAPSHOT/opensearch-security-3.0.0.1-SNAPSHOT.pom
       - https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/plugin/opensearch-security/3.0.0.1-SNAPSHOT/maven-metadata.xml
       - https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/plugin/opensearch-security/3.0.0.1-SNAPSHOT/opensearch-security-3.0.0.1-SNAPSHOT.pom
       - https://plugins.gradle.org/m2/org/opensearch/plugin/opensearch-security/3.0.0.1-SNAPSHOT/maven-metadata.xml
       - https://plugins.gradle.org/m2/org/opensearch/plugin/opensearch-security/3.0.0.1-SNAPSHOT/opensearch-security-3.0.0.1-SNAPSHOT.pom
     Required by:
         project :

cwperks · 2024-01-19T19:06:35Z

@joshpalis to resolve that error you would need to publish a zip of the security plugin with the respective version to maven local. Since you are using a dummy version you would also need to modify security's build.gradle to the version you are expecting. To publish a zip to maven local you can use:

./gradlew publishPluginZipPublicationToMavenLocal -Dbuild.snapshot=true

I don't currently understand how security-analytics is able to pull the required zipArchive dependencies for its integration test run

joshpalis · 2024-01-19T19:15:28Z

to resolve that error you would need to publish a zip of the security plugin with the respective version to maven local

This error isnt the issue, my use of the dummy version was to demonstrate that the build error we're seeing can be avoided if we just conditionally include the security zipArchive dependency if security is disabled (by default). So now with this change, if we attempt to assemble the the flow framework plugin and the security plugin isn't available, the build should pass.

joshpalis · 2024-01-19T19:22:39Z

I don't currently understand how security-analytics is able to pull the required zipArchive dependencies for its integration test run

Seems the security analytics plugin relies on the OpenSearch docker image here in this security test workflow github action which has the security plugin.

The reason we decided to use the security plugin zip archive dependency directly for the flow framework plugin is due to the flaky nature of the docker image, in several cases that I've seen, most security test github actions either fail outright or skip security integration tests since the cluster fails to start.

For example, this security analytics CI run here fails due to :

Run cluster_running=`curl -XGET https://localhost:9200/_cat/plugins -u admin:admin --insecure`
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (7) Failed to connect to localhost port 9200 after 0 ms: Connection refused
Error: Process completed with exit code 7.

cwperks · 2024-01-19T21:40:27Z

@zelinh If Flow-Framework wants to run their integ tests at distribution build time with a zip archive of the security plugin, would they need to add it as a build dependency here?

If security is added as a build dependency, does that also mean that opensearch-build would need to be extended to support it as well?

According to the opensearch-build README:

In order to get around this issue, the instrumentation logic in test-workflow provides these dependencies into maven local repo, before kicking off the integration test for plugins.

How does opensearch-build provide these dependencies to the maven local repo before kicking off integ tests?

zelinh added bug Something isn't working untriaged v2.12.0 labels Jan 17, 2024

zelinh mentioned this issue Jan 17, 2024

[RELEASE] Release version 2.12.0 opensearch-project/opensearch-build#4115

Closed

88 tasks

dbwiddis assigned joshpalis Jan 17, 2024

owaiskazi19 assigned owaiskazi19 and unassigned joshpalis Jan 17, 2024

owaiskazi19 mentioned this issue Jan 17, 2024

Excluded integTest from build script to handle the security plugin dependency #418

Merged

dbwiddis removed the untriaged label Jan 17, 2024

cwperks mentioned this issue Jan 18, 2024

Add step to publishPluginZipPublicationToMavenLocal in the build script opensearch-project/security#3959

Closed

3 tasks

joshpalis mentioned this issue Jan 20, 2024

Only pull security plugin zip if security enabled #423

Closed

owaiskazi19 closed this as completed in #418 Jan 20, 2024

owaiskazi19 mentioned this issue Jan 22, 2024

[v2.12.0] Ensure CI/documentation reflect changes to default admin credentials #310

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] Flow-framework build failed because no opensearch-security dependency #410

[BUG] Flow-framework build failed because no opensearch-security dependency #410

zelinh commented Jan 17, 2024

owaiskazi19 commented Jan 18, 2024 •

edited

Loading

DarshitChanpura commented Jan 18, 2024 •

edited

Loading

cwperks commented Jan 18, 2024

peternied commented Jan 18, 2024 •

edited

Loading

cwperks commented Jan 19, 2024 •

edited

Loading

joshpalis commented Jan 19, 2024

joshpalis commented Jan 19, 2024 •

edited

Loading

cwperks commented Jan 19, 2024

joshpalis commented Jan 19, 2024

joshpalis commented Jan 19, 2024

cwperks commented Jan 19, 2024 •

edited

Loading

[BUG] Flow-framework build failed because no opensearch-security dependency #410

[BUG] Flow-framework build failed because no opensearch-security dependency #410

Comments

zelinh commented Jan 17, 2024

What is the bug?

owaiskazi19 commented Jan 18, 2024 • edited Loading

DarshitChanpura commented Jan 18, 2024 • edited Loading

cwperks commented Jan 18, 2024

peternied commented Jan 18, 2024 • edited Loading

cwperks commented Jan 19, 2024 • edited Loading

joshpalis commented Jan 19, 2024

joshpalis commented Jan 19, 2024 • edited Loading

cwperks commented Jan 19, 2024

joshpalis commented Jan 19, 2024

joshpalis commented Jan 19, 2024

cwperks commented Jan 19, 2024 • edited Loading

owaiskazi19 commented Jan 18, 2024 •

edited

Loading

DarshitChanpura commented Jan 18, 2024 •

edited

Loading

peternied commented Jan 18, 2024 •

edited

Loading

cwperks commented Jan 19, 2024 •

edited

Loading

joshpalis commented Jan 19, 2024 •

edited

Loading

cwperks commented Jan 19, 2024 •

edited

Loading