[BUG] Fix remote shards balancer when filtering throttled nodes

[bugmakerrrrrr]

Description

Today, RemoteShardsBalancer uses AllocationDecider#canAllocateAnyShardToNode to filter throttled or ineligible nodes during allocating unassigned shards. If all eligible nodes of an unassigned shard are filtered before trying to allocate this shard, the shard will be marked as ignored with UnassignedInfo.AllocationStatus.DECIDERS_NO status. As a result, the corresponding ShardRestoreStatus will be set to Failure (RestoreService.RestoreInProgressUpdater#unassignedInfoUpdated). This pull request takes throttled nodes into account and ensures that shards are marked with the appropriate status.

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)
• Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Compatibility status:

Checks if related components are compatible with change c8d000d

Incompatible components

Incompatible components: [https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/cross-cluster-replication.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/sql.git]

[github-actions]

❕ Gradle check result for 4c5408b: UNSTABLE

• TEST FAILURES:

      1 org.opensearch.repositories.azure.AzureBlobStoreRepositoryTests.testSnapshotAndRestore
      1 org.opensearch.repositories.azure.AzureBlobStoreRepositoryTests.classMethod
      1 org.opensearch.remotestore.RemoteIndexPrimaryRelocationIT.testPrimaryRelocationWhileIndexing

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[bugmakerrrrrr]

@kotwanikunal could you take a look at this additional one?

[bugmakerrrrrr]

@kotwanikunal can we merge this

[github-actions]

✅ Gradle check result for ec30c58: SUCCESS

[kotwanikunal]

@andrross / @linuxpi - Mind giving this one another review.

Question about tests

[andrross]

@bugmakerrrrrr Can we add a unit test for the bug being fixed here?

[bugmakerrrrrr]

@bugmakerrrrrr Can we add a unit test for the bug being fixed here?

@andrross I think that RemoteShardsAllocateUnassignedTests#testNoRemoteAllocation can cover this bug fix after modifying RemoteShardsBalancerBaseTestCase.

[github-actions]

❕ Gradle check result for c7ef861: UNSTABLE

• TEST FAILURES:

      1 org.opensearch.remotestore.RemoteIndexPrimaryRelocationIT.testPrimaryRelocationWhileIndexing

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions]

❌ Gradle check result for c8d000d:

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for c8d000d:

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❕ Gradle check result for c8d000d: UNSTABLE

• TEST FAILURES:

      3 org.opensearch.cluster.coordination.AwarenessAttributeDecommissionIT.testConcurrentDecommissionAction
      1 org.opensearch.search.SearchWeightedRoutingIT.testShardRoutingWithNetworkDisruption_FailOpenEnabled
      1 org.opensearch.remotestore.RemoteIndexPrimaryRelocationIT.testPrimaryRelocationWhileIndexing

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.