[CI] SearchReplicaSelectionIT.testNodeSelection

[henningandersen]

This happened locally, but I can see it failing in CI too.

Build scan:
https://gradle-enterprise.elastic.co/s/ewsio3cffibpa

Repro line:

./gradlew ':server:internalClusterTest' --tests "org.elasticsearch.search.routing.SearchReplicaSelectionIT.testNodeSelection" -Dtests.seed=B189466086BCB35A -Dtests.security.manager=true -Dtests.locale=es-AR -Dtests.timezone=America/Whitehorse -Druntime.java=13

Reproduces locally?:
Happened locally, but repro line did not reproduce locally.

Applicable branches:
master (though likely also 7.x)

Failure history:
Has failed a few times per day since it was introduced in #70283.

Failure excerpt:

org.junit.ComparisonFailure: expected:<[oM8ifFUsRkG4AYoXVpxG8]g> but was:<[q6ccPdpJTyScxAYaTv88H]g>
        at __randomizedtesting.SeedInfo.seed([B189466086BCB35A:D85D541A5EDDFA1E]:0)
        at org.junit.Assert.assertEquals(Assert.java:115)
        at org.junit.Assert.assertEquals(Assert.java:144)
        at org.elasticsearch.search.routing.SearchReplicaSelectionIT.testNodeSelection(SearchReplicaSelectionIT.java:99)

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[henningandersen]

Muted in #70622

[henningandersen]

Got another failure on this doing ./gradlew check, looks like it is affected by concurrent activity (or prior tests).

[tlrx]

In case it helps to investigate the failure, here is another build scan (https://gradle-enterprise.elastic.co/s/3jolsfrjatrie) for a failure that occurred before the test was muted.

[jimczi]

The problem is related to the fact that some nodes are mocked with a search service that doesn't handle ARS:

elasticsearch/test/framework/src/main/java/org/elasticsearch/node/MockNode.java

Line 126 in a92a647

return new MockSearchService(clusterService, indicesService, threadPool, scriptService,

It's easy to change but I wonder if this IT can be trusted. Any query in the background can change the score of the nodes and make the test fail. It's unclear to me how clusters are reused in IT tests but the assumptions only hold in isolation.

[jtibshirani]

@jimczi thank you for debugging it. I opened a PR with this fix: #71507. I ran the test a bunch of times with the fix and didn't see failures.

I looked into ESIntegTestCase and confirmed that we do not share clusters between tests. However there is a concern around extra searches that affect ARS behavior: it's theoretically possible we could search a system index on start-up or in the background.

My suggestion is to fix and reenable the test, but if observe more flakiness we can consider removing it. I find the test valuable -- debugging the test actually surfaced another ARS bug (which I'll file soon)! But it doesn't add huge value beyond the existing unit tests, and we should avoid spending a lot of effort fixing a fragile test.