[Proposal] Run gradle check tests in parallel

[owaiskazi19]

Is your feature request related to a problem? Please describe.

Currently, the gradle check on the PR runs on ubuntu x64 c518xlarge and takes around 40-45 mins to finish. As a result most of the flaky test failures are encountered because of the timeout or for the cluster state to be available.
Like:

org.opensearch.discovery.SnapshotDisruptionIT > testDisruptionAfterFinalization FAILED
    java.util.concurrent.TimeoutException: Timeout waiting for task.
        at org.opensearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:252)
        at org.opensearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:77)
        at org.opensearch.snapshots.AbstractSnapshotIntegTestCase.awaitClusterState(AbstractSnapshotIntegTestCase.java:552)
        at org.opensearch.snapshots.AbstractSnapshotIntegTestCase.awaitNoMoreRunningOperations(AbstractSnapshotIntegTestCase.java:519)
        at org.opensearch.discovery.SnapshotDisruptionIT.testDisruptionAfterFinalization(SnapshotDisruptionIT.java:137)

Describe the solution you'd like
To reduce the flaky tests, we can run the Verification tasks in a parallel manner. The tests can be run on either Jenkins or GHA (only if the time is less here as Github provides a 2 x86_64 CPU cores and a 7 GB RAM instance). The current tasks are

Verification tasks
------------------
branchConsistency - Ensures this branch is internally consistent. For example, that versions constants match released versions.
bwcTest - Runs backwards compatibility tests.
check - Runs all checks.
distroTest.deb - Runs deb tests within vagrant
distroTest.deb-arm64 - Runs deb-arm64 tests within vagrant
distroTest.deb-no-jdk - Runs deb-no-jdk tests within vagrant
distroTest.docker-arm64 - Runs docker-arm64 tests within vagrant
distroTest.linux-archive - Runs linux-archive tests within vagrant
distroTest.linux-archive-arm64 - Runs linux-archive-arm64 tests within vagrant
distroTest.linux-archive-no-jdk - Runs linux-archive-no-jdk tests within vagrant
distroTest.rpm - Runs rpm tests within vagrant
distroTest.rpm-arm64 - Runs rpm-arm64 tests within vagrant
distroTest.rpm-no-jdk - Runs rpm-no-jdk tests within vagrant
distroTest.windows-archive - Runs windows-archive tests within vagrant
distroTest.windows-archive-arm64 - Runs windows-archive-arm64 tests within vagrant
distroTest.windows-archive-no-jdk - Runs windows-archive-no-jdk tests within vagrant
distroUpgradeTest.v2.0.0.deb - Runs v2.0.0.deb tests within vagrant
distroUpgradeTest.v2.0.0.deb-arm64 - Runs v2.0.0.deb-arm64 tests within vagrant
distroUpgradeTest.v2.0.0.rpm - Runs v2.0.0.rpm tests within vagrant
distroUpgradeTest.v2.0.0.rpm-arm64 - Runs v2.0.0.rpm-arm64 tests within vagrant
distroUpgradeTest.v2.1.0.deb - Runs v2.1.0.deb tests within vagrant
distroUpgradeTest.v2.1.0.deb-arm64 - Runs v2.1.0.deb-arm64 tests within vagrant
distroUpgradeTest.v2.1.0.rpm - Runs v2.1.0.rpm tests within vagrant
distroUpgradeTest.v2.1.0.rpm-arm64 - Runs v2.1.0.rpm-arm64 tests within vagrant
integTest - Runs rest tests against an opensearch cluster.
internalClusterTest
jacocoTestCoverageVerification - Verifies code coverage metrics based on specified rules for the test task.
jacocoTestReport - Generates code coverage report for the test task.
javaRestTest - Runs the REST tests against an external cluster
precommit - Runs all non-test checks
run - Runs opensearch in the foreground
spotlessApply - Applies code formatting steps to sourcecode in-place.
spotlessCheck - Checks that sourcecode satisfies formatting steps.
spotlessDiagnose
spotlessJava
spotlessJavaApply
spotlessJavaCheck
spotlessJavaDiagnose
test - Runs the test suite.
testAggregateTestReport - Generates aggregated test report.
yamlRestTest - Runs the REST tests against an external cluster

As a POC when ran BWC test independently on GHA (smaller instance) there was an improvement in the time and a little less of timeout failures.
Sample run with BWC Test: https://github.com/owaiskazi19/OpenSearch/runs/4705950671?check_suite_focus=true
Sample run without BWC Test: https://github.com/owaiskazi19/OpenSearch/runs/4807896738?check_suite_focus=true

Integration tests are currently tightly coupled in the codebase so to separate them out will be a huge work.
Ref: https://rnorth.org/posts/faster-parallel-github-builds/

Describe alternatives you've considered
#2496

[dreamer-89]

Thanks @owaiskazi19 for creating this issue. I think this is duplicate of #1975. Hence, closing this one.

[owaiskazi19]

@dreamer-89 thanks for pointing it out. Can we copy the content from this issue and add it in #1975 for more clarity?