[CI] TransformSurvivesUpgradeIT failure

[davidkyle]

Build scan:
https://gradle-enterprise.elastic.co/s/evsustaeeg5gs

Repro line:

 ./gradlew ':x-pack:qa:rolling-upgrade:v7.5.2#bwcTest' \
  -Dtests.class="org.elasticsearch.upgrades.TransformSurvivesUpgradeIT" \
  -Dtests.method="testTransformRollingUpgrade" \
  -Dtests.seed=6EE3BC23BF91038B \
  -Dtests.security.manager=true \
  -Dtests.bwc=true \
  -Dtests.locale=da \
  -Dtests.timezone=America/Matamoros \
  -Druntime.java=8

Reproduces locally?:
No

Applicable branches:
7.x

Failure history:
This is the only instance of this particular failure I could find.

Failure excerpt:

java.lang.AssertionError: expected:<1> but was:<2>
	at __randomizedtesting.SeedInfo.seed([6EE3BC23BF91038B:C19E899A14D7D04F]:0)
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.failNotEquals(Assert.java:834)
	at org.junit.Assert.assertEquals(Assert.java:118)
	at org.junit.Assert.assertEquals(Assert.java:144)
	at org.elasticsearch.upgrades.TransformSurvivesUpgradeIT.lambda$awaitWrittenIndexerState$9(TransformSurvivesUpgradeIT.java:293)
	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:1005)
	at org.elasticsearch.upgrades.TransformSurvivesUpgradeIT.awaitWrittenIndexerState(TransformSurvivesUpgradeIT.java:286)
	at org.elasticsearch.upgrades.TransformSurvivesUpgradeIT.verifyContinuousTransformHandlesData(TransformSurvivesUpgradeIT.java:251)
	at org.elasticsearch.upgrades.TransformSurvivesUpgradeIT.testTransformRollingUpgrade(TransformSurvivesUpgradeIT.java:157)

[elasticmachine]

Pinging @elastic/ml-core (Team:ML)

[hendrikmuhs]

I checked the logs and could not find anything fishy. This test basically tests the migration to a new transform version, which means the state document should be deleted from the old index. In the logs we have this line (after the upgrade):

[2021-02-07T09:58:46,545][TRACE][o.e.x.t.t.ClientTransformIndexer] [v7.5.2-0] [continuous-transform-upgrade-job] deleted old transform stats and state document

That means cleaning should have been executed.

The test is an assertBusy with a 60s timeout, this seems a lot, however the logs have tons of errors in it, maybe it was a bad build?

Unfortunately the logs and test feedback is insufficient, it would be good to:

• improve the assert with a proper message containing the search response
• log the number of deleted documents

As action I would improve the above and investigate further after it failed again. I checked build stats and this test rarely fails with this error.

[hendrikmuhs]

I checked buildstats and could not find this failure again. All failures that also reported TransformSurvivesUpgradeIT failed due to some cluster instability.

Closing for now. If it happens again the added logging hopefully helps.