Testcontainers startup randomly fails on RHEL 8 with Podman

[gtroitsk]

Describe the bug

We have a RHEL8 machine, which uses podman as a container runtime. In our Jenkins pipeline almost every day we faced with Container startup error caused by Broken pipe exception.
I suppose some people can have the same problem. The issue for testcontainers already exists testcontainers/testcontainers-java#7539

Expected behavior

Containers are successfully started all the time

Actual behavior

02:27:25 01:27:25,374 INFO  [database] Initialize service (docker.io/library/postgres:15)
02:27:31 01:27:31,601 Creating container for image: docker.io/library/postgres:15
02:27:31 01:27:31,604 Could not start container: java.lang.RuntimeException: java.io.IOException: Broken pipe
02:27:31 	at com.github.dockerjava.zerodep.ApacheDockerHttpClientImpl.execute(ApacheDockerHttpClientImpl.java:195)
02:27:31 	at com.github.dockerjava.zerodep.ZerodepDockerHttpClient.execute(ZerodepDockerHttpClient.java:8)
02:27:31 	at org.testcontainers.dockerclient.HeadersAddingDockerHttpClient.execute(HeadersAddingDockerHttpClient.java:23)
02:27:31 	at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.execute(DefaultInvocationBuilder.java:228)
02:27:31 	at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.post(DefaultInvocationBuilder.java:124)
02:27:31 	at org.testcontainers.shaded.com.github.dockerjava.core.exec.CreateContainerCmdExec.execute(CreateContainerCmdExec.java:37)
02:27:31 	at org.testcontainers.shaded.com.github.dockerjava.core.exec.CreateContainerCmdExec.execute(CreateContainerCmdExec.java:13)
02:27:31 	at org.testcontainers.shaded.com.github.dockerjava.core.exec.AbstrSyncDockerCmdExec.exec(AbstrSyncDockerCmdExec.java:21)
02:27:31 	at org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd.exec(AbstrDockerCmd.java:33)
02:27:31 	at org.testcontainers.shaded.com.github.dockerjava.core.command.CreateContainerCmdImpl.exec(CreateContainerCmdImpl.java:608)
02:27:31 	at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:441)
02:27:31 	at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:357)
02:27:31 	at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
02:27:31 	at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:347)
02:27:31 	at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:333)
02:27:31 	at io.quarkus.test.services.containers.DockerContainerManagedResource.doStart(DockerContainerManagedResource.java:119)
02:27:31 	at io.quarkus.test.services.containers.DockerContainerManagedResource.start(DockerContainerManagedResource.java:69)
02:27:31 	at io.quarkus.test.bootstrap.BaseService.doStart(BaseService.java:284)
02:27:31 	at io.quarkus.test.bootstrap.BaseService.start(BaseService.java:198)
02:27:31 	at io.quarkus.test.bootstrap.QuarkusScenarioBootstrap.launchService(QuarkusScenarioBootstrap.java:170)
02:27:31 	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
02:27:31 	at io.quarkus.test.bootstrap.QuarkusScenarioBootstrap.beforeAll(QuarkusScenarioBootstrap.java:71)
02:27:31 	at io.quarkus.test.bootstrap.QuarkusScenarioBootstrap.beforeAll(QuarkusScenarioBootstrap.java:48)
02:27:31 	at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.lambda$invokeBeforeAllCallbacks$12(ClassBasedTestDescriptor.java:396)
02:27:31 	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
02:27:31 	at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.invokeBeforeAllCallbacks(ClassBasedTestDescriptor.java:396)
02:27:31 	at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.before(ClassBasedTestDescriptor.java:212)
02:27:31 	at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.before(ClassBasedTestDescriptor.java:85)
02:27:31 	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:148)
02:27:31 	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
02:27:31 	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
02:27:31 	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
02:27:31 	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
02:27:31 	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
02:27:31 	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
02:27:31 	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
02:27:31 	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)

How to Reproduce?

I can't reproduce failed tests locally. At first they are failing randomly, at second I think only if we run all Quarkus QE Test Suite tests.
Nevertheless, I provide here a reproducer for test that failing the most.

Reproducer:

1. git clone https://github.com/quarkus-qe/quarkus-test-suite
2. cd quarkus-test-suite
3. mvn clean verify -f sql-db/sql-app/ -Dit.test=XAPostgresIT

Output of uname -a or ver

4.18.0-513.11.1.el8_9.x86_64

Output of java -version

17.0.7, vendor: Red Hat, Inc.

Quarkus version or git rev

999-SNAPSHOT

Build tool (ie. output of mvnw --version or gradlew --version)

Apache Maven 3.8.6

Additional information

No response

[rsvoboda]

Hi @maxandersen @n1hility. We see Broken pipe when creating container with podman in our periodical podman jobs.

There is already testcontainers/testcontainers-java#7539 issue reported for that, but there is no progress yet on that issue.

Testcontainers are used a lot by Quarkus, thus this makes the user experience with podman worse for end users.

I asked my teammate to create also Quarkus issue to discuss if we can do anything about this on Quarkus side or if we have connections to promote that testcontainers/testcontainers-java#7539 issue.

[mocenas]

There is a similar issue on windows, where problem is basically the same, just the root exception is
Caused by: com.github.dockerjava.zerodep.shaded.org.apache.hc.core5.http.NoHttpResponseException: localhost:2375 failed to respond
This is already tracked in testcotainers in testcontainers/testcontainers-java#7310