podman service: inconsistent exit code when called from docker

[oleastre]

Issue Description

I use some shared scripts between docker and podman environments. To allow some compatibility, I run podman as a service and configure docker to use the podman socket.
I noticed some strange behaviour in one script where we check for the exit code of a run command.
In some cases, I correctly get the command exit code, in other cases, I simply get 0.

Some informations:

• Podman version: 4.5.0
• fedora package: podman-4.5.0-1.fc37.x86_64
• docker package: moby-engine-20.10.23-1.fc37.x86_64

Steps to reproduce the issue

Steps to reproduce the issue

1. start podman as a service systemctl start --user podman.socket
2. set environment variable: export DOCKER_HOST="unix://$(podman info -f "{{.Host.RemoteSocket.Path}}")"
3. Run some containers that exit in error and compare exit code of the command:

$ podman run busybox /bin/sh -c 'exit 3' ; echo $?
3
$ podman run --rm busybox /bin/sh -c 'exit 3' ; echo $?
3
$ docker run busybox /bin/sh -c 'exit 3' ; echo $?
3
$ docker run --rm busybox /bin/sh -c 'exit 3' ; echo $?
0

Describe the results you received

I would expect all commands to return 3, but when ran through docker with the --rm flag I get 0

Describe the results you expected

All variants to return the same exit code.

podman info output

host:
  arch: amd64
  buildahVersion: 1.30.0
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.7-2.fc37.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: '
  cpuUtilization:
    idlePercent: 94.88
    systemPercent: 1.58
    userPercent: 3.54
  cpus: 16
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: workstation
    version: "37"
  eventLogger: journald
  hostname: patator
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.3.4-101.fc37.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 2931601408
  memTotal: 37629321216
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.8.5-1.fc37.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.5
      commit: b6f80f766c9a89eb7b1440c0a70ab287434b17ed
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-8.fc37.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 8588619776
  swapTotal: 8589930496
  uptime: 23h 3m 49.00s (Approximately 0.96 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/olivier/.config/containers/storage.conf
  containerStore:
    number: 5
    paused: 0
    running: 0
    stopped: 5
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/olivier/.local/share/containers/storage
  graphRootAllocated: 510389125120
  graphRootUsed: 341180133376
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 124
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/olivier/.local/share/containers/storage/volumes
version:
  APIVersion: 4.5.0
  Built: 1681486976
  BuiltTime: Fri Apr 14 17:42:56 2023
  GitCommit: ""
  GoVersion: go1.19.7
  Os: linux
  OsArch: linux/amd64
  Version: 4.5.0

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

No

Additional environment details

Additional environment details

Additional information

No response

[vrothberg]

Thanks for reaching out! To me it looks like a Docker bug. run --rm busybox /bin/sh -c 'exit 3' ; echo $? should exit 3 not 0 as Docker does.

[oleastre]

If I run the same command against a docker engine/socket, it works as expected.
So, the problem is when the socket is provided by the podman service.
That's why I suspect it to be a bug in podman service, not in docker.
Is there anything I could try to find on which side of the socket the problem is ?

[Luap99]

I am 99% sure that this was report before but I cannot find that issue right now (if it even exists).

[Luap99]

#16056 but it was closed without proper resolution so I am good with keeping this one open here.

[vrothberg]

Thanks for clarifying, @oleastre. I agree, it looks an issue on the podman side.

[rhatdan]

I would figure this is a race between the container being removed and the exit code being checked? I still see this as a docker daemon bug, since the client is exiting with the wrong exit code. Unless docker intends to ignore any containers exit code, if the container is a --rm?

[vrothberg]

Let's dig a bit deeper. The docker client works fine with the docker daemon but it consistently returns 0 running against podman.

[vrothberg]

With --rm, Docker sends wait?condition=removed` which Podman doesn't process correctly. I'll wrap up a fix.

[vrothberg]

Opened #18912