[Bug]Separate OpenSearch Min Version and Bundle Version in Input Manifest

[peterzhuamazon]

Describe the bug

When building 1.1.1 based on 1.1.0 artifacts for opensearch (BE), we take every plugin from 1.1.0 zips except reports-scheduler, which requires a build from source for 1.1.1.0 version patches.

The problem is, when building with gradle, build-tools is pointing to 1.1.1 which does not exist since we do not change min version.

+ ./gradlew assemble --no-daemon --refresh-dependencies -DskipTests=true -Dopensearch.version=1.1.1 -Dbuild.snapshot=false

The param opensearch.version is the min version, not the plugin version and 1.1.1 is passed to the build.sh, thus cause the build process trying to find 1.1.1 version of build-tools/min in order to build.

details logs

$ ./build.sh manifests/1.1.1/opensearch-1.1.1.yml --platform linux
Installing dependencies in . ...
Installing dependencies from Pipfile.lock (c3492c)...
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/0 — 00:00:00
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
Running ./src/run_build.py manifests/1.1.1/opensearch-1.1.1.yml --platform linux ...
2021-12-07 13:43:03 INFO     Building in /var/folders/d6/19_cf3qd6jb715ckzv_2jsvdmr559j/T/tmpclzr8zsn
2021-12-07 13:43:03 INFO     Building OpenSearch (x64) into /Users/zhujiaxi/amazonworks/sourcecodes/peterzhuamazon/opensearch-build/builds/opensearch
2021-12-07 13:43:03 INFO     Building dashboards-reports
2021-12-07 13:43:03 INFO     Executing "git init" in /var/folders/d6/19_cf3qd6jb715ckzv_2jsvdmr559j/T/tmpclzr8zsn/dashboards-reports
2021-12-07 13:43:03 INFO     Executing "git remote add origin https://github.com/opensearch-project/dashboards-reports.git" in /var/folders/d6/19_cf3qd6jb715ckzv_2jsvdmr559j/T/tmpclzr8zsn/dashboards-reports
2021-12-07 13:43:03 INFO     Executing "git fetch --depth 1 origin 1.1" in /var/folders/d6/19_cf3qd6jb715ckzv_2jsvdmr559j/T/tmpclzr8zsn/dashboards-reports
2021-12-07 13:43:12 INFO     Executing "git checkout FETCH_HEAD" in /var/folders/d6/19_cf3qd6jb715ckzv_2jsvdmr559j/T/tmpclzr8zsn/dashboards-reports
2021-12-07 13:43:12 INFO     Executing "git rev-parse HEAD" in /var/folders/d6/19_cf3qd6jb715ckzv_2jsvdmr559j/T/tmpclzr8zsn/dashboards-reports
2021-12-07 13:43:12 INFO     Checked out https://github.com/opensearch-project/[email protected] into /var/folders/d6/19_cf3qd6jb715ckzv_2jsvdmr559j/T/tmpclzr8zsn/dashboards-reports at aea039a2c68be84767ca19c09b17f69c28a153fa
2021-12-07 13:43:12 INFO     Executing "bash /Users/zhujiaxi/amazonworks/sourcecodes/peterzhuamazon/opensearch-build/scripts/default/opensearch/build.sh -v 1.1.1 -p linux -a x64 -s false -o builds" in /var/folders/d6/19_cf3qd6jb715ckzv_2jsvdmr559j/T/tmpclzr8zsn/dashboards-reports/reports-scheduler
+ getopts :h:v:s:o:p:a: arg
+ case $arg in
+ VERSION=1.1.1
+ getopts :h:v:s:o:p:a: arg
+ case $arg in
+ PLATFORM=linux
+ getopts :h:v:s:o:p:a: arg
+ case $arg in
+ ARCHITECTURE=x64
+ getopts :h:v:s:o:p:a: arg
+ case $arg in
+ SNAPSHOT=false
+ getopts :h:v:s:o:p:a: arg
+ case $arg in
+ OUTPUT=builds
+ getopts :h:v:s:o:p:a: arg
+ '[' -z 1.1.1 ']'
+ [[ false == \t\r\u\e ]]
+ '[' -z builds ']'
+ mkdir -p builds
+ ./gradlew assemble --no-daemon --refresh-dependencies -DskipTests=true -Dopensearch.version=1.1.1 -Dbuild.snapshot=false
To honour the JVM settings for this build a new JVM will be forked. Please consider using the daemon: https://docs.gradle.org/6.6.1/userguide/gradle_daemon.html.
Daemon will be stopped at the end of the build stopping after processing

FAILURE: Build failed with an exception.

* What went wrong:
A problem occurred configuring root project 'opensearch-reports-scheduler'.
> Could not resolve all artifacts for configuration ':classpath'.
   > Could not find org.opensearch.gradle:build-tools:1.1.1.
     Searched in the following locations:
       - file:/Users/zhujiaxi/.m2/repository/org/opensearch/gradle/build-tools/1.1.1/build-tools-1.1.1.pom
       - https://repo.maven.apache.org/maven2/org/opensearch/gradle/build-tools/1.1.1/build-tools-1.1.1.pom
       - https://jcenter.bintray.com/org/opensearch/gradle/build-tools/1.1.1/build-tools-1.1.1.pom
       - https://plugins.gradle.org/m2/org/opensearch/gradle/build-tools/1.1.1/build-tools-1.1.1.pom
     Required by:
         project :

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 10s
2021-12-07 13:43:24 ERROR    Error building dashboards-reports, retry with: ./build.sh manifests/1.1.1/opensearch-1.1.1.yml --component dashboards-reports
Traceback (most recent call last):
  File "./src/run_build.py", line 83, in <module>
    sys.exit(main())
  File "./src/run_build.py", line 71, in main
    builder.build(build_recorder)
  File "/Users/zhujiaxi/amazonworks/sourcecodes/peterzhuamazon/opensearch-build/src/build_workflow/builder_from_source.py", line 49, in build
    self.git_repo.execute(build_command)
  File "/Users/zhujiaxi/amazonworks/sourcecodes/peterzhuamazon/opensearch-build/src/git/git_repository.py", line 83, in execute
    subprocess.check_call(command, cwd=cwd, shell=True)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 363, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'bash /Users/zhujiaxi/amazonworks/sourcecodes/peterzhuamazon/opensearch-build/scripts/default/opensearch/build.sh -v 1.1.1 -p linux -a x64 -s false -o builds' returned non-zero exit status 1.

To reproduce

See above.

Expected behavior

The patch version should be passed to build.sh for opensearch plugins.

Screenshots

If applicable, add screenshots to help explain your problem.

Host / Environment

No response

Additional context

No response

Relevant log output

No response

[dblock]

It's not necessarily the patch version that needs to be passed, it depends on whether OpenSearch is being built or consumed as a distribution. The fix is to "pass the version of OpenSearch being built".

Local workaround probably of hard-coding -Dopensearch.version=1.1.0 in component build.sh should work.

[peterzhuamazon]

It's not necessarily the patch version that needs to be passed, it depends on whether OpenSearch is being built or consumed as a distribution. The fix is to "pass the version of OpenSearch being built".

Thanks @dblock I am currently just passing the patch version as a temp fix.
What would you suggest to add to fix this?
Probably a min version to the manifest?

Thanks.

[dblock]

Thanks @dblock I am currently just passing the patch version as a temp fix. What would you suggest to add to fix this? Probably a min version to the manifest?

I think that's probably the easiest fix. but semantically we pass -v as "the OpenSearch version", maybe we should be able to override that parameter more locally or try to be future-proof for some other hacks?

  - name: component
    repository: https://github.com/opensearch-project/component.git
    ref: "1.1"
    args:
        v: "1.1.0"

  - name: component
    repository: https://github.com/opensearch-project/component.git
    ref: "1.1"
    versions:
        opensearch: "1.1.0"

Open to suggestions.

[dblock]

I think my suggestion is over-engineering. Propose we hard-code the version of the dependency for dashboards-reports 1.1.1 as in opensearch-project/reporting#248 or opensearch-project/reporting#249. Which one do we prefer?

[peterzhuamazon]

After talking to @dblock I personally agree with opensearch-project/reporting#248 as it is the simplest way to get this issue resolved for patches.

The opensearch-project/reporting#249 approach also runs but would add up confusion on why there is an additional build.sh just for 1.1 branch.

We have agreed to open 248 and close 249 as a result.

Thanks.

[peterzhuamazon]

Resolved now.
Thanks.