Quarkus fails to clean test containers on Podman 4.5.1 #33985

fedinskiy · 2023-06-12T14:07:16Z

Describe the bug

I have an application, which uses Mariadb[1] database and depends on Quarkus to deploy the image in DevUI/tests. After I updated podman on my OS from 4.5.0 to 4.5.1, I noticed, that tests can not be run twice in a row. Investigation showed, that after the first run, the container for database stays running.
When Quarkus is not involved (Junit+testcontainers+same DB) everything works as expected.

[1] I was able to reproduce the bug with other DBs as well

Expected behavior

Any started containers should be stopped, same as for Docker and Podman 4.5.0

Actual behavior

Mariadb containers stays in running state

How to Reproduce?

Clone: git clone [email protected]:fedinskiy/reproducer.git -b podman-451
Run test: mvn clean verify -Dtest=QuarkusTest#testHelloEndpoint
Check containers:

$ podman ps
CONTAINER ID  IMAGE                           COMMAND     CREATED         STATUS         PORTS                    NAMES
da4063230ad2  docker.io/library/mariadb:10.6  mariadbd    17 seconds ago  Up 17 seconds  0.0.0.0:42573->3306/tcp  gracious_austin

Run test again: mvn clean verify -Dtest=QuarkusTest#testHelloEndpoint. Test hangs.

For comparison, non-Quarkus tests for the same DB work fine:

mvn clean verify -Dtest=JunitTest

$ podman ps
CONTAINER ID  IMAGE                                COMMAND     CREATED         STATUS         PORTS                    NAMES
5f9a12104c86  docker.io/testcontainers/ryuk:0.3.4  /app        11 seconds ago  Up 11 seconds  0.0.0.0:44759->8080/tcp  testcontainers-ryuk-3ab6a3d0-010c-43db-86ca-1b265394cc9f
# ryuk runs for ~20 more seconds, afterwards it stops

3. mvn clean verify -Dtest=JunitTest succeeds

Output of `uname -a` or `ver`

4.18.0-425.19.2.el8_7.x86_64

Output of `java -version`

17.0.5, vendor: GraalVM Community

GraalVM version (if different from Java)

No response

Quarkus version or git rev

3.0.2.Final

Build tool (ie. output of `mvnw --version` or `gradlew --version`)

Apache Maven 3.8.6 (84538c9988a25aec085021c365c560670ad80f63)

Additional information

$ podman version
Client:       Podman Engine
Version:      4.5.1
API Version:  4.5.1
Go Version:   go1.20.4
Built:        Fri May 26 19:58:48 2023
OS/Arch:      linux/amd64

See #33896 for similar problem

The text was updated successfully, but these errors were encountered:

gsmet · 2023-06-12T14:13:29Z

It does look like a Podman issue?

fedinskiy · 2023-06-12T14:17:12Z

@gsmet for comparison, running non-Qurkus tests for the same DB succeeds, so test-containers + podman work fine without Quarkus. But it is definitely a podman-related issue.

rsvoboda · 2023-06-26T09:39:59Z

@maxandersen pls check this, podman related.

When using just JUnit and TestContainers, no leftover container running.
But when using QuarkusTest, there is a leftover container

https://github.com/fedinskiy/reproducer/blob/podman-451/src/test/java/io/quarkus/qe/JunitTest.java
https://github.com/fedinskiy/reproducer/blob/podman-451/src/test/java/io/quarkus/qe/QuarkusTest.java

maxandersen · 2023-06-26T09:55:01Z

Using podman in both cases?

rsvoboda · 2023-06-26T09:55:39Z

yes

maxandersen · 2023-07-03T15:01:36Z

what do you have in ~/.testcontainers.properties

if you see testcontainers.reuse.enable=true please try testcontainers.reuse.enable=false and run again.

if still see issue or that property gets changed we need to figure out why that is happening.

fedinskiy · 2023-07-04T07:50:17Z

I tried switching testcontainers.reuse.enable to false

The tests run successfully several times in a row, but there is a warning[1], about an issue with ryuk image
I pull the required image manually: podman pull testcontainers/ryuk:0.3.4
I start the test, it succeeds, testcontainers.reuse.enable is still false, but ruyk container is not stopped afterwards.
If the test is started again, then testcontainers.reuse.enable is set to true right after the start
It looks like podman messes with ryuk container behavior. My settings[2] require privileges for ryuk container, maybe podman messes with it?

[1]

2023-07-04 09:37:16,495 ERROR [com.git.doc.api.asy.ResultCallbackTemplate] (docker-java-stream--626058940) Error during callback: java.lang.NullPointerException: Cannot invoke "String.matches(String)" because the return value of "com.github.dockerjava.api.model.PullResponseItem.getStatus()" is null
	at com.github.dockerjava.api.command.PullImageResultCallback.checkForDockerSwarmResponse(PullImageResultCallback.java:48)
	at com.github.dockerjava.api.command.PullImageResultCallback.onNext(PullImageResultCallback.java:35)
	at org.testcontainers.images.LoggedPullImageResultCallback.onNext(LoggedPullImageResultCallback.java:48)
	at org.testcontainers.images.TimeLimitedLoggedPullImageResultCallback.onNext(TimeLimitedLoggedPullImageResultCallback.java:73)
	at org.testcontainers.images.TimeLimitedLoggedPullImageResultCallback.onNext(TimeLimitedLoggedPullImageResultCallback.java:24)
	at org.testcontainers.shaded.com.github.dockerjava.core.exec.AbstrAsyncDockerCmdExec$1.onNext(AbstrAsyncDockerCmdExec.java:41)
	at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder$JsonSink.accept(DefaultInvocationBuilder.java:315)
	at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder$JsonSink.accept(DefaultInvocationBuilder.java:298)
	at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.lambda$executeAndStream$1(DefaultInvocationBuilder.java:275)
	at java.base/java.lang.Thread.run(Thread.java:833)

2023-07-04 09:37:16,502 ERROR [🐳 .io/.6]] (build-30) Could not start container: org.testcontainers.containers.ContainerLaunchException: Container startup failed
	at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:349)
	at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:322)
	at org.testcontainers.utility.RyukResourceReaper.maybeStart(RyukResourceReaper.java:78)
	at org.testcontainers.utility.RyukResourceReaper.register(RyukResourceReaper.java:68)
	at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:423)
	at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:344)
	at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
	at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:334)
	at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:322)
	at io.quarkus.devservices.mariadb.deployment.MariaDBDevServicesProcessor$1.startDatabase(MariaDBDevServicesProcessor.java:67)
	at io.quarkus.datasource.deployment.devservices.DevServicesDatasourceProcessor.startDevDb(DevServicesDatasourceProcessor.java:284)
	at io.quarkus.datasource.deployment.devservices.DevServicesDatasourceProcessor.launchDatabases(DevServicesDatasourceProcessor.java:136)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at io.quarkus.deployment.ExtensionLoader$3.execute(ExtensionLoader.java:909)
	at io.quarkus.builder.BuildContext.run(BuildContext.java:282)
	at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
	at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2513)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1538)
	at java.base/java.lang.Thread.run(Thread.java:833)
	at org.jboss.threads.JBossThread.run(JBossThread.java:501)
Caused by: org.testcontainers.containers.ContainerFetchException: Can't get Docker image: RemoteDockerImage(imageName=testcontainers/ryuk:0.3.4, imagePullPolicy=DefaultPullPolicy(), imageNameSubstitutor=org.testcontainers.utility.ImageNameSubstitutor$LogWrappedImageNameSubstitutor@48018513)
	at org.testcontainers.containers.GenericContainer.getDockerImageName(GenericContainer.java:1371)
	at org.testcontainers.containers.GenericContainer.logger(GenericContainer.java:651)
	at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:331)
	... 22 more
Caused by: java.lang.NullPointerException: Cannot invoke "String.matches(String)" because the return value of "com.github.dockerjava.api.model.PullResponseItem.getStatus()" is null
	at com.github.dockerjava.api.command.PullImageResultCallback.checkForDockerSwarmResponse(PullImageResultCallback.java:48)
	at com.github.dockerjava.api.command.PullImageResultCallback.onNext(PullImageResultCallback.java:35)
	at org.testcontainers.images.LoggedPullImageResultCallback.onNext(LoggedPullImageResultCallback.java:48)
	at org.testcontainers.images.TimeLimitedLoggedPullImageResultCallback.onNext(TimeLimitedLoggedPullImageResultCallback.java:73)
	at org.testcontainers.images.TimeLimitedLoggedPullImageResultCallback.onNext(TimeLimitedLoggedPullImageResultCallback.java:24)
	at org.testcontainers.shaded.com.github.dockerjava.core.exec.AbstrAsyncDockerCmdExec$1.onNext(AbstrAsyncDockerCmdExec.java:41)
	at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder$JsonSink.accept(DefaultInvocationBuilder.java:315)
	at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder$JsonSink.accept(DefaultInvocationBuilder.java:298)
	at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.lambda$executeAndStream$1(DefaultInvocationBuilder.java:275)
	at java.base/java.lang.Thread.run(Thread.java:833)

[2]

ryuk.container.privileged=true
docker.client.strategy=org.testcontainers.dockerclient.UnixSocketClientProviderStrategy
testcontainers.reuse.enable=false

fedinskiy · 2023-07-17T14:38:11Z

@n1hility could you take a look?

rsvoboda · 2023-08-28T08:22:33Z

@maxandersen @n1hility could you take a look?

fedinskiy · 2023-08-28T10:09:16Z

It seems, that Podman 4.6.1 is not affected by this

UPD: on Podman 4.6.1 it doesn't block tests anymore, but containers are still not being stopped. This may cause temporary data to be persisted between runs

n1hility · 2023-08-29T02:56:10Z

sure will take a look

gsmet · 2024-10-02T13:29:15Z

From what I can see, the issue is solved in recent Podman as they reenabled the test there: quarkus-qe/quarkus-test-suite#1380 .

fedinskiy added the kind/bug Something isn't working label Jun 12, 2023

quarkus-bot bot added the triage/needs-triage label Jun 12, 2023

fedinskiy mentioned this issue Jun 12, 2023

Make podman first class citizens for quarkus usecases #31253

Open

6 tasks

geoand added env/podman and removed triage/needs-triage labels Jun 13, 2023

jsmrcka mentioned this issue Aug 29, 2023

Enable tests which were disabled on outdated Podman 4.5.1 quarkus-qe/quarkus-test-suite#1380

Merged

9 tasks

andrewazores mentioned this issue Dec 21, 2023

[Task] Test runs leave dangling database containers cryostatio/cryostat#221

Closed

gsmet closed this as not planned Won't fix, can't repro, duplicate, stale Oct 2, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Quarkus fails to clean test containers on Podman 4.5.1 #33985

Quarkus fails to clean test containers on Podman 4.5.1 #33985

fedinskiy commented Jun 12, 2023 •

edited

Loading

gsmet commented Jun 12, 2023

fedinskiy commented Jun 12, 2023

rsvoboda commented Jun 26, 2023

maxandersen commented Jun 26, 2023

rsvoboda commented Jun 26, 2023

maxandersen commented Jul 3, 2023

fedinskiy commented Jul 4, 2023 •

edited

Loading

fedinskiy commented Jul 17, 2023

rsvoboda commented Aug 28, 2023

fedinskiy commented Aug 28, 2023 •

edited

Loading

n1hility commented Aug 29, 2023

gsmet commented Oct 2, 2024

Quarkus fails to clean test containers on Podman 4.5.1 #33985

Quarkus fails to clean test containers on Podman 4.5.1 #33985

Comments

fedinskiy commented Jun 12, 2023 • edited Loading

Describe the bug

Expected behavior

Actual behavior

How to Reproduce?

Output of uname -a or ver

Output of java -version

GraalVM version (if different from Java)

Quarkus version or git rev

Build tool (ie. output of mvnw --version or gradlew --version)

Additional information

gsmet commented Jun 12, 2023

fedinskiy commented Jun 12, 2023

rsvoboda commented Jun 26, 2023

maxandersen commented Jun 26, 2023

rsvoboda commented Jun 26, 2023

maxandersen commented Jul 3, 2023

fedinskiy commented Jul 4, 2023 • edited Loading

fedinskiy commented Jul 17, 2023

rsvoboda commented Aug 28, 2023

fedinskiy commented Aug 28, 2023 • edited Loading

n1hility commented Aug 29, 2023

gsmet commented Oct 2, 2024

fedinskiy commented Jun 12, 2023 •

edited

Loading

Output of `uname -a` or `ver`

Output of `java -version`

Build tool (ie. output of `mvnw --version` or `gradlew --version`)

fedinskiy commented Jul 4, 2023 •

edited

Loading

fedinskiy commented Aug 28, 2023 •

edited

Loading