Distribution bundle bwc tests

[vibrantvarun]

Description

There is a framework for BWC tests on distribution level as well which will set up the test cluster with the bundle distribution including all plugins engineering effectiveness team builds.

It will be beneficial during the release cycle since we will be testing the exact distribution that we would ship to the customers.
This PR consist configuration to run bwc tests on distribution bundle.

The bwc distribution will fail on neural search for opensearch versions <2.11 because of the bug from the ml-commons.
opensearch-project/ml-commons#1838

rg.opensearch.neuralsearch.bwc.SemanticSearchIT > testSemanticSearch_E2EFlow FAILED
    org.opensearch.client.ResponseException: method [POST], host [http://127.0.0.1:39449], URI [/neuralsearch-bwc-testsemanticsearch_e2eflow/_search?size=1], status line [HTTP/1.1 500 Internal Server Error]
    {"error":{"root_cause":[{"type":"illegal_state_exception","reason":"unexpected byte [0x48]"}],"type":"illegal_state_exception","reason":"unexpected byte [0x48]"},"status":500}
        at __randomizedtesting.SeedInfo.seed([9C3D08353B6D4C7A:2A2FA7AD9C7ED3BD]:0)
        at app//org.opensearch.client.RestClient.convertResponse(RestClient.java:376)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:346)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:321)
        at app//org.opensearch.neuralsearch.BaseNeuralSearchIT.search(BaseNeuralSearchIT.java:417)
        at app//org.opensearch.neuralsearch.BaseNeuralSearchIT.search(BaseNeuralSearchIT.java:378)
        at app//org.opensearch.neuralsearch.BaseNeuralSearchIT.search(BaseNeuralSearchIT.java:359)
        at app//org.opensearch.neuralsearch.bwc.SemanticSearchIT.validateTestIndexOnUpgrade(SemanticSearchIT.java:79)
        at app//org.opensearch.neuralsearch.bwc.SemanticSearchIT.testSemanticSearch_E2EFlow(SemanticSearchIT.java:46)

Issues Resolved

564

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed as per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (d298e2b) 84.39% compared to head (1b7973d) 84.39%.

Additional details and impacted files

@@            Coverage Diff            @@
##               main     #579   +/-   ##
=========================================
  Coverage     84.39%   84.39%           
  Complexity      535      535           
=========================================
  Files            40       40           
  Lines          1570     1570           
  Branches        245      245           
=========================================
  Hits           1325     1325           
  Misses          133      133           
  Partials        112      112           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[vibrantvarun]

./gradlew bwcTestSuite -Dtests.security.manager=false -PcustomDistributionDownloadType=bundle

> Configure project :
Download https://raw.githubusercontent.com/opensearch-project/security/main/bwc-test/src/test/resources/security/esnode.pem
Download https://raw.githubusercontent.com/opensearch-project/security/main/bwc-test/src/test/resources/security/esnode-key.pem
Download https://raw.githubusercontent.com/opensearch-project/security/main/bwc-test/src/test/resources/security/root-ca.pem
Download https://raw.githubusercontent.com/opensearch-project/security/main/bwc-test/src/test/resources/security/kirk.pem
Download https://raw.githubusercontent.com/opensearch-project/security/main/bwc-test/src/test/resources/security/kirk-key.pem
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 8.4
  OS Info               : Linux 5.10.205-172.807.amzn2int.x86_64 (amd64)
  JDK Version           : 21 (Amazon Corretto JDK)
  JAVA_HOME             : /local/home/varunudr/.sdkman/candidates/java/21.0.1-amzn
  Random Testing Seed   : 135916AFF3FBB0A6
  In FIPS 140 mode      : false
=======================================

> Task :generatePomFileForNebulaPublication
Maven publication 'nebula' pom metadata warnings (silence with 'suppressPomMetadataWarningsFor(variant)'):
  - Variant testFixturesApiElements:
      - Declares capability org.opensearch:neural-search-test-fixtures:3.0.0.0-SNAPSHOT which cannot be mapped to Maven
  - Variant testFixturesRuntimeElements:
      - Declares capability org.opensearch:neural-search-test-fixtures:3.0.0.0-SNAPSHOT which cannot be mapped to Maven
These issues indicate information that is lost in the published 'pom' metadata file, which may be an issue if the published library is consumed by an old Gradle version or Apache Maven.
The 'module' metadata file, which is used by Gradle 6+ is not affected.

> Task :qa:restart-upgrade:testAgainstOldCluster
Feb 01, 2024 1:30:57 PM sun.util.locale.provider.LocaleProviderAdapter <clinit>
WARNING: COMPAT locale provider will be removed in a future release

> Task :qa:restart-upgrade:testAgainstNewCluster
Feb 01, 2024 1:32:06 PM sun.util.locale.provider.LocaleProviderAdapter <clinit>
WARNING: COMPAT locale provider will be removed in a future release

> Task :qa:rolling-upgrade:testAgainstOldCluster
Feb 01, 2024 1:33:02 PM sun.util.locale.provider.LocaleProviderAdapter <clinit>
WARNING: COMPAT locale provider will be removed in a future release

> Task :qa:rolling-upgrade:testAgainstOneThirdUpgradedCluster
Feb 01, 2024 1:33:55 PM sun.util.locale.provider.LocaleProviderAdapter <clinit>
WARNING: COMPAT locale provider will be removed in a future release

> Task :qa:rolling-upgrade:testAgainstTwoThirdsUpgradedCluster
Feb 01, 2024 1:34:24 PM sun.util.locale.provider.LocaleProviderAdapter <clinit>
WARNING: COMPAT locale provider will be removed in a future release

> Task :qa:rolling-upgrade:testRollingUpgrade
Feb 01, 2024 1:34:53 PM sun.util.locale.provider.LocaleProviderAdapter <clinit>
WARNING: COMPAT locale provider will be removed in a future release

Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

For more on this, please refer to https://docs.gradle.org/8.4/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.

BUILD SUCCESSFUL in 5m 2s
38 actionable tasks: 13 executed, 25 up-to-date

[zelinh]

Had an offline discussion with @vibrantvarun. This PR looks good for me. Please make sure to change the BWC version when we backport to 2.x branch. Thanks.