Running two containers on a podman container results on one of the containers failing to start

[flixman]

Issue Description

I am running a gitea runner workflow, and all the actions are run on a podman container. In one of the actions I create container images and push them to a registry using podman, and then I use podman-compose to start some of the containers to execute tests. Should I run only one container (redis or openldap), all works. Should I, instead, run both containers at the same time (redis and openldap), then they will both get stuck at "starting". I have tested other container combinations (postgresql and ldap, etc.) and the outcome is the same.

Trying to simplify the situation, instead of podman-compose-in-podman I am running two containers as podman-in-podman, by running

podman run --name redis_server1 -p 1234:6379 docker.io/redis:7.0-alpine3.19 &
podman run --name redis_server2 -p 1235:6379 docker.io/redis:7.0-alpine3.19 &

inside the container. I am getting an error from one of the containers stating

| 1:M 19 Oct 2024 07:59:21.872 # Warning: Could not create server TCP listening socket *:6379: bind: Address in use
| 1:M 19 Oct 2024 07:59:21.872 # Failed listening on port 6379 (TCP), aborting.

The container running all these commands is started with the options --device /dev/fuse --device /dev/net/tun --security-opt=unmask=/proc/sys and a custom network that looks like:

$ podman network inspect GITEA-ACTIONS-TASK-pull-request_WORKFLOW-Build-Image_JOB-build-build-network
[
     {
          "name": "GITEA-ACTIONS-TASK-pull-request_WORKFLOW-Build-Image_JOB-build-build-network",
          "id": "433a2ac8e4a84a09cfd00fecbd1372965129f37e7c35046eefa03f993eab4fd1",
          "driver": "bridge",
          "network_interface": "podman2",
          "created": "2024-10-19T10:08:11.243180077+02:00",
          "subnets": [
               {
                    "subnet": "10.89.1.0/24",
                    "gateway": "10.89.1.1"
               }
          ],
          "ipv6_enabled": false,
          "internal": false,
          "dns_enabled": true,
          "options": {
               "isolate": "true"
          },
          "ipam_options": {
               "driver": "host-local"
          },
          "containers": {
               "14289e89e18868fe7e76ee90aceb984fbbac330b0522630b6f8f2f3679abf864": {
                    "name": "GITEA-ACTIONS-TASK-pull-request_WORKFLOW-Build-Image_JOB-build",
                    "interfaces": {
                         "eth0": {
                              "subnets": [
                                   {
                                        "ipnet": "10.89.1.2/24",
                                        "gateway": "10.89.1.1"
                                   }
                              ],
                              "mac_address": "d2:8f:c7:c2:15:67"
                         }
                    }
               }
          }
     }
]

This might well be a configuration issue, but I have been looking around for days and I do not manage to see what I am doing wrong, if anything.

Steps to reproduce the issue

Run the two podman commands explained in the description.

Describe the results you received

one container starts, one container fails to start

Describe the results you expected

both containers start

podman info output

inside the container, I get

[podman@14289e89e188 myproject]$ podman info
host:
  arch: amd64
  buildahVersion: 1.37.2
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.10-1.fc40.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: '
  cpuUtilization:
    idlePercent: 96.82
    systemPercent: 1.15
    userPercent: 2.04
  cpus: 16
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: container
    version: "40"
  eventLogger: file
  freeLocks: 2044
  hostname: 14289e89e188
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 1
      size: 999
    - container_id: 1000
      host_id: 1001
      size: 64535
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 1
      size: 999
    - container_id: 1000
      host_id: 1001
      size: 64535
  kernel: 6.11.3-arch1-1
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 1491095552
  memTotal: 15909416960
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.12.1-1.fc40.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.12.1
    package: netavark-1.12.2-1.fc40.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.12.2
  ociRuntime:
    name: crun
    package: crun-1.15-1.fc40.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.15
      commit: e6eacaf4034e84185fd8780ac9262bbf57082278
      rundir: /tmp/storage-run-1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240821.g1d6142f-1.fc40.x86_64
    version: |
      pasta 0^20240821.g1d6142f-1.fc40.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /tmp/storage-run-1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 0
  swapTotal: 0
  uptime: 136h 33m 50.00s (Approximately 5.67 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  docker.io:
    Blocked: false
    Insecure: false
    Location: docker.io
    MirrorByDigestOnly: false
    Mirrors:
    - Insecure: false
      Location: registry.kngnt.org
      PullFromMirror: ""
    Prefix: docker.io
    PullFromMirror: ""
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
store:
  configFile: /home/podman/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 1
    stopped: 1
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/podman/.local/share/containers/storage
  graphRootAllocated: 500856545280
  graphRootUsed: 259797925888
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 20
  runRoot: /tmp/storage-run-1000/containers
  transientStore: false
  volumePath: /home/podman/.local/share/containers/storage/volumes
version:
  APIVersion: 5.2.2
  Built: 1724198400
  BuiltTime: Wed Aug 21 00:00:00 2024
  GitCommit: ""
  GoVersion: go1.22.6
  Os: linux
  OsArch: linux/amd64
  Version: 5.2.2

in the machine running the wrapping container, I get

[myuser@mymachine myproject]$ podman info
host:
  arch: amd64
  buildahVersion: 1.37.4
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-1:2.1.12-1
    path: /usr/bin/conmon
    version: 'conmon version 2.1.12, commit: e8896631295ccb0bfdda4284f1751be19b483264'
  cpuUtilization:
    idlePercent: 96.82
    systemPercent: 1.15
    userPercent: 2.04
  cpus: 16
  databaseBackend: sqlite
  distribution:
    distribution: arch
    version: unknown
  eventLogger: journald
  freeLocks: 1983
  hostname: mymachine
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.11.3-arch1-1
  linkmode: dynamic
  logDriver: journald
  memFree: 1459404800
  memTotal: 15909416960
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.12.2-1
      path: /usr/lib/podman/aardvark-dns
      version: aardvark-dns 1.12.2
    package: netavark-1.12.2-1
    path: /usr/lib/podman/netavark
    version: netavark 1.12.2
  ociRuntime:
    name: crun
    package: crun-1.17-1
    path: /usr/bin/crun
    version: |-
      crun version 1.17
      commit: 000fa0d4eeed8938301f3bcf8206405315bc1017
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-2024_09_06.6b38f07-1
    version: |
      pasta 2024_09_06.6b38f07
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 0
  swapTotal: 0
  uptime: 136h 35m 55.00s (Approximately 5.67 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  docker.io:
    Blocked: false
    Insecure: false
    Location: docker.io
    MirrorByDigestOnly: false
    Mirrors:
    - Insecure: false
      Location: my.registry.org
      PullFromMirror: ""
    Prefix: docker.io
    PullFromMirror: ""
  search:
  - docker.io
store:
  configFile: /home/myuser/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/myuser/.local/share/containers/storage
  graphRootAllocated: 500856545280
  graphRootUsed: 259797966848
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 123
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/myuser/.local/share/containers/storage/volumes
version:
  APIVersion: 5.2.4
  Built: 1728466240
  BuiltTime: Wed Oct  9 11:30:40 2024
  GitCommit: 76d0859d9f6e5d7c84392da9dcd2fae3447f789d
  GoVersion: go1.23.2
  Os: linux
  OsArch: linux/amd64
  Version: 5.2.4

Podman in a container

Yes

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[Luap99]

Assuming you are using quay.io/podman/stable? In that case we are using --network=host by default
https://github.com/containers/image_build/blob/f27333ae1ff3efbc9e5a507bb44c90e3bacc937e/podman/containers.conf#L2

You would have to add --network bridge, --network pasta or --network slirp4netns depending on what you want to get a private namespace so that the ports do not conflict

[flixman]

Yes, I am using quay.io/podman/stable. After adding --network pasta to the set of options the act runner uses for the podman container, the results do not change. Also the podman network inspect GITEA-ACTIONS-TASK-pull-request_WORKFLOW-Build-Image_JOB-build-build-network returns the same output than previously:

[
     {
          "name": "GITEA-ACTIONS-TASK-pull-request_WORKFLOW-Build-Image_JOB-build-build-network",
          "id": "9f7e9c168c8722c1b8e969ea622f96920cf89c264a9b72548cc896e8c23c3cab",
          "driver": "bridge",
          "network_interface": "podman2",
          "created": "2024-10-26T08:55:01.297693602+02:00",
          "subnets": [
               {
                    "subnet": "10.89.1.0/24",
                    "gateway": "10.89.1.1"
               }
          ],
          "ipv6_enabled": false,
          "internal": false,
          "dns_enabled": true,
          "options": {
               "isolate": "true"
          },
          "ipam_options": {
               "driver": "host-local"
          },
          "containers": {
               "ad630635e74ec1690fab3a27043ef9daa711bf1871cc7039c8192d4dccf3164a": {
                    "name": "GITEA-ACTIONS-TASK-pull-request_WORKFLOW-Build-Image_JOB-build",
                    "interfaces": {
                         "eth0": {
                              "subnets": [
                                   {
                                        "ipnet": "10.89.1.2/24",
                                        "gateway": "10.89.1.1"
                                   }
                              ],
                              "mac_address": "12:bf:6d:4e:55:d7"
                         }
                    }
               }
          }
     }
]

What else could be off?

[flixman]

@Luap99 This is the command line I am using to have act_runner run the workflow:
gitea_runner/act_runner exec -i myregistry.org/gitea/runners/podman:extended --event pull_request --container-daemon-socket unix:///run/user/1000/podman/podman.sock --container-opts --device /dev/fuse --device /dev/net/tun --security-opt=unmask=/proc/sys -v buildah-cache:/var/tmp/buildah-cache-1000 --network pasta

This is the dockerfile I am using to create podman:extended

FROM quay.io/podman/stable

RUN dnf install -y git nodejs python3-pip openssl && \
    pip install podman-compose && \
    dnf clean all && \
    rm -rf /var/cache/yum

USER podman
WORKDIR /home/podman

Finally, this is the action I am running:

---
name: Build_Image
on:
    pull_request:
        types: [opened, edited, synchronize, reopened]
        branches:
            - main

jobs:
    build:
        runs-on: myregistry.org/gitea/runners/podman:extended
        steps:
            - name: Test podman
              run: |
                  podman run --name redis_server1 -p 1234:6379 docker.io/redis:7.0-alpine3.19 &
                  podman run --name redis_server2 -p 1235:6379 docker.io/redis:7.0-alpine3.19 &
                  echo "Waiting..."
                  sleep 600

Is there anything else I can provide to help in the debugging? I am going through podman documentation, but I cannot spot what am I doing wrong.

[Luap99]

you have to set --network on the inner containers not the outer one. quay.io/podman/stable contains a special containers.conf file that uses different defaults such as host networking as linked above.

[flixman]

oh! my apologies! I had not understood (obviously) what you meant with your previous comment. Thank you very much! Edit: On top of that, it makes sense :-/