[CI] OldRepositoryAccessIT testOldRepoAccess failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-intake #11931 / part2
• elasticsearch-pull-request #38421 / part-2

Reproduction Line:

./gradlew ":x-pack:qa:repository-old-versions:javaRestTestBeforeRestart#5_0_0" --tests "org.elasticsearch.oldrepos.OldRepositoryAccessIT.testOldRepoAccess" -Dtests.seed=360E9472706BB500 -Dtests.locale=de-DE -Dtests.timezone=SystemV/PST8 -Druntime.java=22

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: expected:<1> but was:<0>

Issue Reasons:

• [main] 2 consecutive failures in test testOldRepoAccess
• [main] 2 failures in test testOldRepoAccess (0.2% fail rate in 1000 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 2 consecutive failures in test testOldRepoAccess
• [main] 2 failures in test testOldRepoAccess (0.2% fail rate in 1000 executions)

Build Scans:

• elasticsearch-intake #11931 / part2
• elasticsearch-pull-request #38421 / part-2

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)

[elasticsearchmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[henningandersen]

Looks like this line is the failure:

                assertEquals(numberOfShards, searchResponse.getSkippedShards());

marking as low risk since this only concerns shard skipping and it seems we should expect other tests to catch this.

Assigning search team as well, to get their eyes on this.

[andreidan]

++ thanks @henningandersen Apologies for the noise

[andreidan]

I'm unable to run this test:

./gradlew ":x-pack:qa:repository-old-versions:javaRestTestBeforeRestart#5_0_0" --tests "org.elasticsearch.oldrepos.OldRepositoryAccessIT.testOldRepoAccess" -Dtests.seed=360E9472706BB500 -Dtests.locale=de-DE -Dtests.timezone=SystemV/PST8 -Druntime.java=22

due to

  [log]
    Running [/Users/andrei/.gradle/caches/8.10.2/transforms/ccbbe6228fc0f642d5c0a44eb8738e9a/transformed/elasticsearch-5.0.0.zip/elasticsearch-5.0.0/bin/elasticsearch, -p, pid]
    port not found

@elastic/es-delivery is this something you can help with?

[mark-vieira]

@elastic/es-delivery is this something you can help with?

I have the same problem locally, and I believe the issue is that you cannot run these tests on an aarch64 system since old versions of Elasticsearch are x64 only.

[nik9000]

Another: https://gradle-enterprise.elastic.co/s/gedpv26kjtgks/tests/overview?outcome=FAILED

[elasticsearchmachine]

This has been muted on branch 8.x

Mute Reasons:

• [8.x] 3 failures in test testOldRepoAccess (0.3% fail rate in 1000 executions)
• [8.x] 2 failures in step part2 (8.7% fail rate in 23 executions)
• [8.x] 2 failures in pipeline elasticsearch-intake (8.7% fail rate in 23 executions)

Build Scans:

• elasticsearch-periodic #4961 / openjdk23_checkpart2_java-matrix
• elasticsearch-intake #13192 / part2
• elasticsearch-intake #13179 / part2

[breskeby]

I noticed yesterday that this hasn't been working for a while on osx due to an issue with the jdk download plugin and how it handles the Zulu jdk 8 download / artifact transform. This bit should be solved via #117120

[cbuescher]

At least in one of the build-scans above I can see

OldRepositoryAccessIT > testOldRepoAccess FAILED
    java.lang.AssertionError: expected:<1> but was:<0>
        at __randomizedtesting.SeedInfo.seed([A86299CB7C4BA6A0:5A15EC8D39D25895]:0)
[...]]
        at org.elasticsearch.oldrepos.OldRepositoryAccessIT.assertDocs(OldRepositoryAccessIT.java:487)
        at org.elasticsearch.oldrepos.OldRepositoryAccessIT.restoreMountAndVerify(OldRepositoryAccessIT.java:329)
        at org.elasticsearch.oldrepos.OldRepositoryAccessIT.beforeRestart(OldRepositoryAccessIT.java:253)
        at org.elasticsearch.oldrepos.OldRepositoryAccessIT.runTest(OldRepositoryAccessIT.java:109)
        at org.elasticsearch.oldrepos.OldRepositoryAccessIT.testOldRepoAccess(OldRepositoryAccessIT.java:77)

That line in OldRepositoryAccessIT was changed recently (25/10) by https://github.com/elastic/elasticsearch/pull/115314/files#diff-089e10b8f461c2e6a342b3fdb4d430eeea6284cf807472471244b532193a5678R487, so at least that would coincide with the first ocurrences of this failure.
I'll do some digging if this reproduces or if not, why...

[elasticsearchmachine]

This has been muted on branch 8.17

Mute Reasons:

• [8.17] 2 failures in test testOldRepoAccess (0.2% fail rate in 845 executions)

Build Scans:

• elasticsearch-periodic-platform-support #4980 / almalinux-8-aarch64_checkpart2_platform-support-arm
• elasticsearch-intake #13319 / part2

[cbuescher]

The test fails reproducibly when we only use one shard here. I'm not sure if that is an intended effect of the changes in #115314 as I don't see any aggregations in the request we are sending later, but I will open a PR that adjusts expectations conditionally and ping @piergm for his opinion when he returns.