Fix assertion failure in updateGlobalCheckpointOnReplica() with remote translog

[sachinpkale]

Description

• When remote translog is enabled for an index, replication operation is limited to primary term validation and does not update local checkpoint at replica, so the local checkpoint at replica can be less than globalCheckpoint.
• This fails the assertion at following code:

OpenSearch/server/src/main/java/org/opensearch/index/shard/IndexShard.java

Lines 3071 to 3088 in 1856090

	if (globalCheckpoint > localCheckpoint) {
	/*
	* This can happen during recovery when the shard has started its engine but recovery is not finalized and is receiving global
	* checkpoint updates. However, since this shard is not yet contributing to calculating the global checkpoint, it can be the
	* case that the global checkpoint update from the primary is ahead of the local checkpoint on this shard. In this case, we
	* ignore the global checkpoint update. This can happen if we are in the translog stage of recovery. Prior to this, the engine
	* is not opened and this shard will not receive global checkpoint updates, and after this the shard will be contributing to
	* calculations of the global checkpoint. However, we can not assert that we are in the translog stage of recovery here as
	* while the global checkpoint update may have emanated from the primary when we were in that state, we could subsequently move
	* to recovery finalization, or even finished recovery before the update arrives here.
	*/
	assert state() != IndexShardState.POST_RECOVERY && state() != IndexShardState.STARTED
	: "supposedly in-sync shard copy received a global checkpoint ["
	+ globalCheckpoint
	+ "] "
	+ "that is higher than its local checkpoint ["
	+ localCheckpoint
	+ "]";

Issues Resolved

• [BUG] Assertion failure in IndexShard.updateGlobalCheckpointOnReplica() when remote translog is enabled #6193

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/13381/
• CommitID: 05d9df3

[codecov-commenter]

Codecov Report

Merging #6975 (05d9df3) into main (285b450) will increase coverage by 0.05%.
The diff coverage is 0.00%.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@             Coverage Diff              @@
##               main    #6975      +/-   ##
============================================
+ Coverage     70.67%   70.73%   +0.05%     
- Complexity    59206    59238      +32     
============================================
  Files          4812     4812              
  Lines        283748   283749       +1     
  Branches      40916    40917       +1     
============================================
+ Hits         200527   200697     +170     
+ Misses        66715    66527     -188     
- Partials      16506    16525      +19     

Impacted Files	Coverage Δ
...in/java/org/opensearch/index/shard/IndexShard.java	70.23% <0.00%> (+0.01%)	⬆️

... and 521 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.