[BUG] BWC tests are failing after upgrading to OpenSearch 3.0.0

[DarshitChanpura]

BWC tests encountered failure after the 3.0.0.0 version bump on the main branch. Seems like an issue with a cluster not coming-up in time.

2022-06-27T19:03:56.5156700Z FAILURE: Build failed with an exception.
2022-06-27T19:03:56.5157042Z 
2022-06-27T19:03:56.5157224Z * Where:
2022-06-27T19:03:56.5158743Z Build file '/home/runner/work/security/security/bwc-test/build.gradle' line: 200
2022-06-27T19:03:56.5159142Z 5 actionable tasks: 5 executed
2022-06-27T19:03:56.5159509Z 
2022-06-27T19:03:56.5159747Z * What went wrong:
2022-06-27T19:03:56.5160307Z Execution failed for task ':securityBwcCluster#mixedClusterTask'.
2022-06-27T19:03:56.5160891Z > `cluster{::securityBwcCluster0}` failed to wait for cluster health yellow after 40 SECONDS
2022-06-27T19:03:56.5161386Z     IO error while waiting cluster
2022-06-27T19:03:56.5161763Z     503 Service Unavailable

You can view the completed error log here

[DarshitChanpura]

@cliu123 Can you please follow-up with core regarding this issue?

[cliu123]

There's an open issue in core. Could that be related?

[DarshitChanpura]

[Triage] This is blocking security from moving to 3.0.0

[peternied]

[Triage] Don't see much movement on the issue in OpenSearch, @cliu123 could you check in with OpenSearch and see if they are actively working on this issue? (as it is blocking us upgrading 3.0.0 on main)

[cliu123]

Issue still perisists. I updated the latest status in the OpenSearch issue.

[peternied]

The BWC tests passed when we bumped to version 3, see this run https://github.com/opensearch-project/security/actions/runs/3151467687/jobs/5125449978

Reference issue: #1953

PR where version was bumped: #2051