testcontainers integration: is already connected to CNI network "podman": network already exists - docker behavior divergence

[dmarrazzo]

BUG REPORT

Description

This issue is cause by an apparent divergence between the docker behavior and podman.
Test containers is a project which automate the container creation to simplify the developer experience (it automatically starts the container to be used for testing purposing during the development lifecycle).

Specifically, quarkus project relies on testcontainers for its dev mode.
In general it works pretty well and how to tune up the environment is described here:
https://quarkus.io/blog/quarkus-devservices-testcontainers-podman/

This issue was noticed dealing with a specific project:

https://github.com/kiegroup/kogito-examples/tree/stable/serverless-workflow-examples/serverless-workflow-greeting-quarkus

Analyzing the logs, I was able to reproduce the problem using the curl command against the Libpod restfull APIs.

Steps to reproduce the issue:

1. create the container

curl -d '{"Hostname":null,"Domainname":null,"User":null,"AttachStdin":null,"AttachStdout":null,"AttachStderr":null,"PortSpecs":null,"Tty":null,"OpenStdin":null,"StdinOnce":null,"Env":[],"Cmd":[],"Healthcheck":null,"ArgsEscaped":null,"Entrypoint":null,"Image":"quay.io/kiegroup/kogito-data-index-ephemeral:1.25","Volumes":{},"WorkingDir":null,"MacAddress":null,"OnBuild":null,"NetworkDisabled":null,"ExposedPorts":{"8080/tcp":{}},"StopSignal":null,"StopTimeout":null,"HostConfig":{"Binds":[],"BlkioWeight":null,"BlkioWeightDevice":null,"BlkioDeviceReadBps":null,"BlkioDeviceWriteBps":null,"BlkioDeviceReadIOps":null,"BlkioDeviceWriteIOps":null,"MemorySwappiness":null,"NanoCpus":null,"CapAdd":null,"CapDrop":null,"ContainerIDFile":null,"CpuPeriod":null,"CpuRealtimePeriod":null,"CpuRealtimeRuntime":null,"CpuShares":null,"CpuQuota":null,"CpusetCpus":null,"CpusetMems":null,"Devices":null,"DeviceCgroupRules":null,"DeviceRequests":null,"DiskQuota":null,"Dns":null,"DnsOptions":null,"DnsSearch":null,"ExtraHosts":["host.testcontainers.internal:10.88.0.2"],"GroupAdd":null,"IpcMode":null,"Cgroup":null,"Links":[],"LogConfig":null,"LxcConf":null,"Memory":null,"MemorySwap":null,"MemoryReservation":null,"KernelMemory":null,"NetworkMode":null,"OomKillDisable":null,"Init":null,"AutoRemove":null,"OomScoreAdj":null,"PortBindings":{"8080/tcp":[{"HostIp":"","HostPort":"8180"}]},"Privileged":true,"PublishAllPorts":null,"ReadonlyRootfs":null,"RestartPolicy":null,"Ulimits":null,"CpuCount":null,"CpuPercent":null,"IOMaximumIOps":null,"IOMaximumBandwidth":null,"VolumesFrom":[],"Mounts":null,"PidMode":null,"Isolation":null,"SecurityOpt":null,"StorageOpt":null,"CgroupParent":null,"VolumeDriver":null,"ShmSize":null,"PidsLimit":null,"Runtime":null,"Tmpfs":null,"UTSMode":null,"UsernsMode":null,"Sysctls":null,"ConsoleSize":null,"CgroupnsMode":null},"Labels":{"kogito-dev-service-data-index":"kogito-data-index","org.testcontainers":"true","org.testcontainers.sessionId":"bb6927a0-74a2-4d39-818c-742157a8c60b"},"Shell":null,"NetworkingConfig":null}' -H "Content-Type: application/json" -X POST --unix-socket /run/user/(id -u)/podman/podman.sock http://localhost/v1.32/containers/create
{"Id":"1d703199a6820a96b3fee3baae20e7773fc2d75a07e1fb0cccdb16e79f3e5c0c","Warnings":[]}

2. connect the container to the network, using the container id returned from the previous command.

curl -d '{"containerConfig":null,"Container":"1d703199a6820a96b3fee3baae20e7773fc2d75a07e1fb0cccdb16e79f3e5c0c","EndpointConfig":null}' -H "Content-Type: application/json" -X POST --unix-socket /run/user/(id -u)/podman/podman.sock http://localhost/v1.32/networks/podman/connect
{"cause":"network already exists","message":"container 1d703199a6820a96b3fee3baae20e7773fc2d75a07e1fb0cccdb16e79f3e5c0c is already connected to network \"podman\": network already exists","response":500}

Results:
The error (500) prevents testcontainers and quarkus from completing the normal initialization.

Expected Result:
This problem does not happen using docker, my educated guess is that docker networking is different or it's more tolerant.

Output of podman version:

Tested with v3 and v4:

Client:       Podman Engine
Version:      4.2.0
API Version:  4.2.0
Go Version:   go1.18.4
Built:        Thu Aug 11 16:42:17 2022
OS/Arch:      linux/amd64

Output of podman info:

host:
  arch: amd64
  buildahVersion: 1.27.0
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.0-2.fc36.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.0, commit: '
  cpuUtilization:
    idlePercent: 91.37
    systemPercent: 1.89
    userPercent: 6.74
  cpus: 8
  distribution:
    distribution: fedora
    variant: workstation
    version: "36"
  eventLogger: journald
  hostname: mas480
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.18.19-200.fc36.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 450269184
  memTotal: 25070055424
  networkBackend: cni
  ociRuntime:
    name: crun
    package: crun-1.5-1.fc36.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.5
      commit: 54ebb8ca8bf7e6ddae2eb919f5b82d1d96863dea
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-0.2.beta.0.fc36.x86_64
    version: |-
      slirp4netns version 1.2.0-beta.0
      commit: 477db14a24ff1a3de3a705e51ca2c4c1fe3dda64
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 7896821760
  swapTotal: 8589930496
  uptime: 5h 58m 36.00s (Approximately 0.21 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/donato/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/donato/.local/share/containers/storage
  graphRootAllocated: 998902857728
  graphRootUsed: 467663851520
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 2
  runRoot: /run/user/1000/containers
  volumePath: /home/donato/.local/share/containers/storage/volumes
version:
  APIVersion: 4.2.0
  Built: 1660228937
  BuiltTime: Thu Aug 11 16:42:17 2022
  GitCommit: ""
  GoVersion: go1.18.4
  Os: linux
  OsArch: linux/amd64
  Version: 4.2.0

Package info (e.g. output of rpm -q podman or apt list podman):

podman-4.2.0-2.fc36.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

No

Additional environment details (AWS, VirtualBox, physical, etc.):
Physical Fedora 36

Optional - How to reproduce with quarkus / maven

1. clone the repository: https://github.com/kiegroup/kogito-examples/

2. switch on the tag

   git checkout 1.25.0.Final
   

3. change directory to serverless-workflow-examples/serverless-workflow-greeting-quarkus

4. launch quarkus in dev mode:

   mvn clean quarkus:dev -DskipTests
   

[mheon]

Podman is the name of the default network, so it could be that Docker reports 200 on a API network connect call when the network is already created, and Podman treats it as an error.

[mheon]

Sorry, when the network is already connected, not created

[dmarrazzo]

I tested the same http commands against docker and it reply with HTTP 200 code.

Actually, I believe that the proper result should be HTTP 409.

I don't know if this different behavior can be considered a bug: anyway 500 seems not correct.

it depends if you prefer to pursue the same user experience.

[n1hility]

IMO 200 is the correct state here, the request is effectively a harmless no-op, not an invalid transition. We should change this.

[kubealex]

I had the chance to debug the issue with @dmarrazzo yesterday and I ended up with two ideas:

1. Defining a new Error, like ErrNetworkConnected (the 'network already exists' message doesn't make sense there)
   Use the error here where the check that gives the error and propagates upper in the call chain.

2. handling the error here if the error is a ErrNetworkConnected just return a 200

In this way, the behaviour would be identical to the docker one, error on the CLI, 200 on the API.

[jgraettinger]

Howdy!

I ran into this while attempting to use the supabase CLI tool with podman instead of docker.

The supabase CLI spins up a bunch of containers, and at several points attempts to create a docker network if it doesn't already exist, as determined by docker library error helpers which are explicitly looking for HTTP 409 status, and don't properly interpret podman's 500.

I'm unsure if this belongs as a separate issue, but it's an example where the HTTP 500 rather than 409 behavior diverges from the Docker REST API.

[mheon]

Yes, separate issue would be best

[anchet19]

Hey, @jgraettinger I've hit the same problem, were you able to resolve your issue? If you created a new issue report could you link to it?

[ozozozd]

Here for the same issue re: podman + supabase