[Test Failure]RecoveryIT IndexStatsIT Tests Failure

[Bukhtawar]

Describe the bug
org.opensearch.upgrades.RecoveryIT.testRelocationWithConcurrentIndexing
org.opensearch.upgrades.RecoveryIT.testRelocationWithConcurrentIndexing
org.opensearch.upgrades.RecoveryIT.testRelocationWithConcurrentIndexing
org.opensearch.upgrades.RecoveryIT.testRelocationWithConcurrentIndexing
org.opensearch.indices.replication.SegmentReplicationIT.testStartReplicaAfterPrimaryIndexesDocs
org.opensearch.indices.replication.SegmentReplicationRelocationIT.testIndexReopenClose
org.opensearch.indices.stats.IndexStatsIT.testFilterCacheStats

[Bukhtawar]

Test Result (19 failures / +15)
org.opensearch.backwards.IndexingIT.testUpdateSnapshotStatus
org.opensearch.backwards.IndexingIT.testSeqNoCheckpoints
org.opensearch.backwards.IndexingIT.testIndexVersionPropagation
org.opensearch.backwards.IndexingIT.classMethod
org.opensearch.backwards.IndexingIT.testUpdateSnapshotStatus
org.opensearch.backwards.IndexingIT.testSeqNoCheckpoints
org.opensearch.backwards.IndexingIT.testIndexVersionPropagation
org.opensearch.backwards.IndexingIT.classMethod
org.opensearch.backwards.IndexingIT.testUpdateSnapshotStatus
org.opensearch.backwards.IndexingIT.testSeqNoCheckpoints
org.opensearch.backwards.IndexingIT.testIndexVersionPropagation
org.opensearch.backwards.IndexingIT.classMethod
org.opensearch.backwards.IndexingIT.testSeqNoCheckpoints
org.opensearch.backwards.IndexingIT.testIndexVersionPropagation
org.opensearch.upgrades.RecoveryIT.testRelocationWithConcurrentIndexing
org.opensearch.upgrades.RecoveryIT.testRelocationWithConcurrentIndexing
org.opensearch.upgrades.RecoveryIT.testRelocationWithConcurrentIndexing
org.opensearch.upgrades.RecoveryIT.testRelocationWithConcurrentIndexing

[mch2]

Moving 2.5 specific tests to a separate issue.

[kotwanikunal]

Segment Replication tests have been muted and are being root caused separately.
We are seeing them post the feature flag merge: #4959

[dreamer-89]

Tried this org.opensearch.backwards.IndexingIT.testSeqNoCheckpoints on latest main. I think few BWC tests are broken with this commit #5282 which introduced replicated boolean flag in ReplicationTracker. This happens as this flag is always sent irrespective of checking on receiver version.

./gradlew ':qa:mixed-cluster:v2.6.0#mixedClusterTest' --tests "org.opensearch.backwards.IndexingIT.testSeqNoCheckpoints" -Dtests.seed=BE95B870ADDCC4FF -Dtests.security.manager=true -Dtests.jvm.arg
line="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=en-ZA -Dtests.timezone=Etc/GMT-7 -Druntime.java=17
...
»  Caused by: org.opensearch.transport.RemoteTransportException: [v2.6.0-3][127.0.0.1:57996][internal:index/shard/recovery/start_recovery]
»  Caused by: org.opensearch.transport.RemoteTransportException: [v2.6.0-1][127.0.0.1:59081][internal:index/shard/recovery/handoff_primary_context]
»  Caused by: java.lang.IllegalStateException: unexpected byte [0x6d]
»       at org.opensearch.common.io.stream.StreamInput.readBoolean(StreamInput.java:588) ~[opensearch-2.6.0-SNAPSHOT.jar:2.6.0-SNAPSHOT]
»       at org.opensearch.common.io.stream.StreamInput.readBoolean(StreamInput.java:578) ~[opensearch-2.6.0-SNAPSHOT.jar:2.6.0-SNAPSHOT]
»       at org.opensearch.index.seqno.ReplicationTracker$CheckpointState.<init>(ReplicationTracker.java:713) ~[opensearch-2.6.0-SNAPSHOT.jar:2.6.0-SNAPSHOT]

[anasalkouz]

Should be resolved. Closing the issue.

[saratvemulapalli]

Reopening as thte flaky tests show up again on main.
#7674 (comment)

[andrross]

@saratvemulapalli I'm going to resolve this as I believe there was a version check mismatch fixed in #7672 that caused the failure you linked. Please reopen if that's wrong.

[saratvemulapalli]

Thanks @andrross. Rebase fixed it for me.