unable to run dev container for serverless workflow

[raffaelespazzoli]

Describe the bug

running the hello world example with podman for serverless workflow (https://kiegroup.github.io/kogito-docs/serverlessworkflow/main/getting-started/create-your-first-workflow-service.html), I get the following error:

2022-08-22 20:37:51,314 INFO [org.tes.uti.RegistryAuthLocator] (build-37) Failure when attempting to lookup auth config. Please ignore if you don't have images in an authenticated registry. Details: (dockerImageName: quay.io/kiegroup/kogito-data-index-eph2022-08-22 20:37:51,363 ERROR [🐳 .io/.25]] (build-37) Could not start container: com.github.dockerjava.api.exception.InternalServerErrorException: Status 500: {"cause":"network already exists","message":"container b71de387226214473bfb91c10c099b81e821ef4fcf6d083853d69a77fda317dc is already connected to network "podman": network already exists","response":500}
notice that devcontainers generally work on my laptop, so this issue is specific to this devcontainer.

Expected behavior

dev container runs normally

Actual behavior

the error showed above

How to Reproduce?

follow the steps here: https://kiegroup.github.io/kogito-docs/serverlessworkflow/main/getting-started/create-your-first-workflow-service.html with fedora and podman.

Output of uname -a or ver

Linux rspazzol 5.18.16-200.fc36.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Aug 3 15:44:49 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Output of java -version

openjdk version "17.0.4" 2022-07-19 OpenJDK Runtime Environment (Red_Hat-17.0.4.0.8-1.fc36) (build 17.0.4+8) OpenJDK 64-Bit Server VM (Red_Hat-17.0.4.0.8-1.fc36) (build 17.0.4+8, mixed mode, sharing)

GraalVM version (if different from Java)

No response

Quarkus version or git rev

2.11.2 final

Build tool (ie. output of mvnw --version or gradlew --version)

Apache Maven 3.8.4 (9b656c72d54e5bacbed989b64718c159fe39b537) Maven home: /home/rspazzol/.m2/wrapper/dists/apache-maven-3.8.4-bin/52ccbt68d252mdldqsfsn03jlf/apache-maven-3.8.4 Java version: 11.0.16, vendor: Red Hat, Inc., runtime: /usr/lib/jvm/java-11-openjdk-11.0.16.0.8-1.fc36.x86_64 Default locale: en_US, platform encoding: UTF-8 OS name: "linux", version: "5.18.16-200.fc36.x86_64", arch: "amd64", family: "unix"

Additional information

No response

[dmarrazzo]

I was able to dig a bit more in this issue.

Apparently, the issue is caused by a request sent by testcontainer to add a network to a container which already has it.

Here how to replicate the problem from command line:

# podman run -d --net podman1 nginx
# podman network connect podman1 wizardly_goldberg
Error: container 548c6f182228f45c93a1f7d9b20775993d67cdf1e08a53fd54004570d0dc675c is already connected to network "podman1": network already exists

testcontainers relies on HTTP APIs (Libpod Restful) and fails because it gets back a 500 HTTP response.

I opened an issue against podman, but it's likely that they are not going to change this behavior because it's pretty correct.
(Personally, I believe that a better reply would be HTTP 409, but it would not change the testcontainers result).

See containers/podman#15499

Said that, I believe that kogito team should review the image to check if it's possible to remove the constraint at the core.

In second place, it would be nice if quarkus can tolerate this situation and go over it.

Finally, I'd like to recommend quarkus and testcontainers team to consider podman as a first citizen option.

[quarkus-bot]

/cc @geoand, @stuartwdouglas

[n1hility]

This looks to be caused by testcontainers expose host port feature, which launches an sshd container and sets up a remote port forward to do so. It then reads the network off of the sshd service and after all other container creations updates the container network to match, even if the container is already associated with the same network.

The Kogito Data Index services uses expose host to expose the web port back.

Testcontainers could get the network back and not reassociate but IMO containers/podman#15499 should behave differently.

Leaving this issue here and self assigning for tracking

[geoand]

Thanks for the explanation @n1hility

[ricardozanini]

Thanks, @n1hility. It seems like you guys fixed it already here: containers/podman#15516

[n1hility]

Your welcome, this will be part of the podman 4.2.1 release, which will be shipping soon. Closing since there is completed.

[ricardozanini]

@radtriste cc