[META] Automate backwards compatibility test support for OpenSearch project

[abhinavGupta16]

Following up on the previous META issue - #90

Is your feature request related to a problem? Please describe.

We do a lot of manual testing. Need visible and repeatable automation for end-to-end upgrade paths described in https://opensearch.org/faq/#c3, including rolling and restart upgrades. Plugins should be able to contribute setup steps in an older version (data) and integration tests that run in mixed and upgraded cluster states.

Proposal

We would like to see CI/CD that shows that a specific bundle build of OpenSearch has been tested on a list of upgrade paths.
Here is our proposal for BWC/Upgrade testing for OpenSearch project.

All BWC tests include:

1. Rolling upgrade
2. Mixed Cluster search
3. Snapshot upgrade
4. Restart upgrade

All components within opensearch project will define bwctests:

1. Run BWC tests in each component.
2. Run BWC tests in each component in PR push.
3. Run BWC tests in release CI against individual component.
4. Run BWC tests in release CI against the bundle.

Current Status

To run bwc-tests -

1. Deploy docker image - opensearchstaging/ci-runner:centos7-x64-arm64-jdkmulti-node10.24.1-cypress6.9.1-20211028
2. Clone opensearch-build repository
3. run build.sh to build the artifacts
4. run assemble.sh to install all the components
5. run test.sh with bwc-tests

To complete P0 we need the dependency installer to install each plugin and run the test for each plugin on a separate cluster.

Tasks

P0

• Adding bwc test framework support for plugins: Enable BWC tests for plugins OpenSearch#1002
• Adding BWC test framework automation and documentation: Add support for automated backwards-compatibility tests for plugins opensearch-plugins#53
• Adding support to run BWC tests in plugin CI: Run plugin BWC tests in CI #222

Release CI BWC test workflow: [Meta] #232

• Adding BWC test framework in release/nightly CI: Add support for running bwctest.sh #275
• Adding Jenkins job support: Add Jenkins job for hooking up BWC tests #492
• Add support for Dependency Installer for Opensearch #1897
• Add support to onboard to test manifest: Add support for BWC tests to onboard to Test Manifest #572
• Add support for TestRecorder/TestPublisher: Integrate BWC test result with test publisher framework(207) #277
• Add support BWC test onboarding for developers: Developer documentation for BWC Tests #223

P1

• BWC Tests against the bundle (Meta issue): Run bwc tests for the distribution (OpenSearch + plugins). #121
• Add support for Snapshot upgrades: Introduce snapshot upgrade bwc tests OpenSearch#1088
• Add support to run BWC tests in workflow via Docker Dockerize BWC test runs #278
• Add support to run BWC tests in parallel: Implement support for the bwc tests to run in parallel for multiple plugins #243
• Add support to download previous version artifacts predictably for plugins: Automate publishing plugins zips to artifacts repository #184

Here is the design proposal for #121 :

1. All repositories define their bwc tests for A and B.
2. Nightly builds will run A for repo level testing via bwctest.sh script defined in each repository.
   Ref: https://github.com/opensearch-project/anomaly-detection/blob/main/integtest.sh
   Eg: ./integtest.sh bwcTest
3. We will rely on [META] Test orchestration framework #122 and a job to the orchestrator to spin up various kinds of BWC test clusters.
4. Nightly builds will also run B for bundle level testing via bwctest.sh defined in each repository using the clusters setup by the orchestrator. The orchestrator job will make calls to the bwctest.sh script to perform various steps of testing during upgrade.
   Eg: For Rolling Upgrade, the orchestrator job will spin an old version (the versions should be defined via a manifest, TBD).
   a. Kick off ingestion via ./bwctest.sh RollingUpgradeIngestTest
   b. Upgrade a node to new version
   c. Kick off mixed cluster test via ./bwctest.sh MixedClusterSearchTest
   d. Upgrade all nodes to the latest version.
   e. Kick off upgraded cluster test via ./bwctest.sh UpgradedClusterTest
   f. Report test status and repeat for every repository.
5. Report bundle level test status.

[zelinh]

Closing in favor of #2870. Will update in that issue for progress.