DockerTests.test600Interrupt failing due to stdout being too long

[pgomulka]

CI Link

https://gradle-enterprise.elastic.co/s/kl67xjolc3cgc

Repro line

n/a

Does it reproduce?

Yes

Applicable branches

main

Failure history

No response

Failure excerpt

https://gradle-enterprise.elastic.co/scans/tests?search.timeZoneId=Europe/Warsaw&tests.container=org.elasticsearch.packaging.test.DockerTests&tests.test=test600Interrupt

the test fails due to infrastructure code reading from docker's stdout failing due to stdout being too long

<<Too large to read (113516 bytes), truncated>>" |  
-- | --
  | at __randomizedtesting.SeedInfo.seed([359E5782DDE88E6E:AF75E480BDF7F318]:0) |  
  | at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18) |  
  | at org.junit.Assert.assertThat(Assert.java:956) |  
  | at org.elasticsearch.packaging.test.DockerTests.test600Interrupt(DockerTests.java:1237)

the limit is 1001024charsize

we should iteratively check the stdout instead of copying the full output

[elasticsearchmachine]

Pinging @elastic/es-delivery (Team:Delivery)

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[mark-vieira]

we should iteratively check the stdout instead of copying the full output

We should but refactoring these assertions to work with streams is non-trivial and complicated further by the fact that we have multiple assertions.

A 100kB limit might be a bit conservative here. I know we don't want to store enormous strings in-memory, but we could 10x this limit w/o any appreciable effect on test execution memory usage. @rjernst thoughts?

[rjernst]

I'm fine to increase the limit, but I want to draw a line in the sand. 100kb was plenty before, what changed? These are very short lived instance, so they don't have a lot of log messages. At some point we have to fix this (it can be a new method and iteratively change to the iterative log assertions). For now though, let's increase to 1mB to unblock this.

[mark-vieira]

Most startup messages are simply INFO level stuff like loading modules, index templates, etc. These certainly grow over time as we add more modules to the distribution.

[williamrandolph]

I created an issue for changing how the tests work so that we can close this test-failure ticket. We talked about it in a team meeting today, and concluded that bumping the timeout counts as a temporary fix.

We can reopen this issue or create a new one if tests fail again in a similar way.