`podman stop $cid1 $cid2` successfully stops both but sometimes errors that $cid2 does not exist #7384

asottile · 2020-08-19T22:48:48Z

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

This one is pretty hard to track down and I haven't isolated a highly-reproducible scenario however I am able to observe this about ~10% of the time with my given workload.

The output below shows the failure scenario and the output of podman container inspect $cid for both of the containers just before running podman stop $cid1 $cid2

It appears that sometimes when stopping both containers podman successfully stops both but then loses track of the second container causing an error.

I can also reproduce this when running podman stop $cid1 and podman stop $cid2 separately

Steps to reproduce the issue:

I'm still working on more reproducible results but I'm currently running something similar to this in a loop and get an error every ~10 or so runs. I will comment / edit if I find a more reproducible run and without private components.

cid1=$(podman run --rm --detach -v "$workdir:/work:rw" -v "$input:/input:ro" -v "$pc:/pc:rw" $img sleep infiinty)
cid2=$(podman run --net=none --rm --detach -v "$workdir:/work:rw" -v "$input:/input:ro" -v "$pc:/pc:rw" $img sleep infiinty)
# several podman exec -i "$cid1"... and one  podman exec "$cid2" ...
podman status "$cid1"
podman status "$cid2"
podman stop "$cid1" "$cid2"

Describe the results you received:

`podman inspect $cid1` (right before running stop)


[
  {
    "Id": "295e59ca74f4486cf27dece22be6d7b16ef0d9e54e5053418fb7ebf910b20ac8",
    "Created": "2020-08-19T14:54:21.826048257-07:00",
    "Path": "dumb-init",
    "Args": [
      "--",
      "sleep",
      "infinity"
    ],
    "State": {
      "OciVersion": "1.0.2-dev",
      "Status": "running",
      "Running": true,
      "Paused": false,
      "Restarting": false,
      "OOMKilled": false,
      "Dead": false,
      "Pid": 4613,
      "ConmonPid": 4592,
      "ExitCode": 0,
      "Error": "",
      "StartedAt": "2020-08-19T14:54:26.251034733-07:00",
      "FinishedAt": "0001-01-01T00:00:00Z",
      "Healthcheck": {
        "Status": "",
        "FailingStreak": 0,
        "Log": null
      }
    },
    "Image": "42f29002c8b9ee4b1f89044cd301b14aebb41cd84954ceb9f1416e71fcf9286a",
    "ImageName": "localhost/pre-commit:1",
    "Rootfs": "",
    "Pod": "",
    "ResolvConfPath": "/run/user/1000/containers/vfs-containers/295e59ca74f4486cf27dece22be6d7b16ef0d9e54e5053418fb7ebf910b20ac8/userdata/resolv.conf",
    "HostnamePath": "/run/user/1000/containers/vfs-containers/295e59ca74f4486cf27dece22be6d7b16ef0d9e54e5053418fb7ebf910b20ac8/userdata/hostname",
    "HostsPath": "/run/user/1000/containers/vfs-containers/295e59ca74f4486cf27dece22be6d7b16ef0d9e54e5053418fb7ebf910b20ac8/userdata/hosts",
    "StaticDir": "/home/asottile/.local/share/containers/storage/vfs-containers/295e59ca74f4486cf27dece22be6d7b16ef0d9e54e5053418fb7ebf910b20ac8/userdata",
    "OCIConfigPath": "/home/asottile/.local/share/containers/storage/vfs-containers/295e59ca74f4486cf27dece22be6d7b16ef0d9e54e5053418fb7ebf910b20ac8/userdata/config.json",
    "OCIRuntime": "runc",
    "LogPath": "/home/asottile/.local/share/containers/storage/vfs-containers/295e59ca74f4486cf27dece22be6d7b16ef0d9e54e5053418fb7ebf910b20ac8/userdata/ctr.log",
    "LogTag": "",
    "ConmonPidFile": "/run/user/1000/containers/vfs-containers/295e59ca74f4486cf27dece22be6d7b16ef0d9e54e5053418fb7ebf910b20ac8/userdata/conmon.pid",
    "Name": "relaxed_franklin",
    "RestartCount": 0,
    "Driver": "vfs",
    "MountLabel": "",
    "ProcessLabel": "",
    "AppArmorProfile": "",
    "EffectiveCaps": [
      "CAP_AUDIT_WRITE",
      "CAP_CHOWN",
      "CAP_DAC_OVERRIDE",
      "CAP_FOWNER",
      "CAP_FSETID",
      "CAP_KILL",
      "CAP_MKNOD",
      "CAP_NET_BIND_SERVICE",
      "CAP_NET_RAW",
      "CAP_SETFCAP",
      "CAP_SETGID",
      "CAP_SETPCAP",
      "CAP_SETUID",
      "CAP_SYS_CHROOT"
    ],
    "BoundingCaps": [
      "CAP_AUDIT_WRITE",
      "CAP_CHOWN",
      "CAP_DAC_OVERRIDE",
      "CAP_FOWNER",
      "CAP_FSETID",
      "CAP_KILL",
      "CAP_MKNOD",
      "CAP_NET_BIND_SERVICE",
      "CAP_NET_RAW",
      "CAP_SETFCAP",
      "CAP_SETGID",
      "CAP_SETPCAP",
      "CAP_SETUID",
      "CAP_SYS_CHROOT"
    ],
    "ExecIDs": [],
    "GraphDriver": {
      "Name": "vfs",
      "Data": null
    },
    "Mounts": [
      {
        "Type": "bind",
        "Name": "",
        "Source": "/tmp/tmpybpszytg/work1",
        "Destination": "/work",
        "Driver": "",
        "Mode": "",
        "Options": [
          "rbind"
        ],
        "RW": true,
        "Propagation": "rprivate"
      },
      {
        "Type": "bind",
        "Name": "",
        "Source": "/tmp/tmpybpszytg/pc1",
        "Destination": "/pc",
        "Driver": "",
        "Mode": "",
        "Options": [
          "rbind"
        ],
        "RW": true,
        "Propagation": "rprivate"
      },
      {
        "Type": "bind",
        "Name": "",
        "Source": "/tmp/tmpybpszytg/input1",
        "Destination": "/input",
        "Driver": "",
        "Mode": "",
        "Options": [
          "rbind"
        ],
        "RW": false,
        "Propagation": "rprivate"
      }
    ],
    "Dependencies": [],
    "NetworkSettings": {
      "EndpointID": "",
      "Gateway": "",
      "IPAddress": "",
      "IPPrefixLen": 0,
      "IPv6Gateway": "",
      "GlobalIPv6Address": "",
      "GlobalIPv6PrefixLen": 0,
      "MacAddress": "",
      "Bridge": "",
      "SandboxID": "",
      "HairpinMode": false,
      "LinkLocalIPv6Address": "",
      "LinkLocalIPv6PrefixLen": 0,
      "Ports": {},
      "SandboxKey": "/run/user/1000/netns/cni-b0de4a50-b6b7-a6fa-46ca-5bce30be9196"
    },
    "ExitCommand": [
      "/usr/bin/podman",
      "--root",
      "/home/asottile/.local/share/containers/storage",
      "--runroot",
      "/run/user/1000/containers",
      "--log-level",
      "error",
      "--cgroup-manager",
      "cgroupfs",
      "--tmpdir",
      "/run/user/1000/libpod/tmp",
      "--runtime",
      "runc",
      "--storage-driver",
      "vfs",
      "--events-backend",
      "file",
      "container",
      "cleanup",
      "--rm",
      "295e59ca74f4486cf27dece22be6d7b16ef0d9e54e5053418fb7ebf910b20ac8"
    ],
    "Namespace": "",
    "IsInfra": false,
    "Config": {
      "Hostname": "295e59ca74f4",
      "Domainname": "",
      "User": "",
      "AttachStdin": false,
      "AttachStdout": false,
      "AttachStderr": false,
      "Tty": false,
      "OpenStdin": false,
      "StdinOnce": false,
      "Env": [
        "PATH=/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
        "TERM=xterm",
        "container=podman",
        "PRE_COMMIT_HOME=/pc",
        "HOSTNAME=295e59ca74f4",
        "HOME=/root"
      ],
      "Cmd": [
        "sleep",
        "infinity"
      ],
      "Image": "localhost/pre-commit:1",
      "Volumes": null,
      "WorkingDir": "/",
      "Entrypoint": "dumb-init --",
      "OnBuild": null,
      "Labels": null,
      "Annotations": {
        "io.container.manager": "libpod",
        "io.kubernetes.cri-o.Created": "2020-08-19T14:54:21.826048257-07:00",
        "io.kubernetes.cri-o.TTY": "false",
        "io.podman.annotations.autoremove": "TRUE",
        "io.podman.annotations.init": "FALSE",
        "io.podman.annotations.privileged": "FALSE",
        "io.podman.annotations.publish-all": "FALSE",
        "org.opencontainers.image.stopSignal": "15"
      },
      "StopSignal": 15,
      "CreateCommand": [
        "podman",
        "run",
        "--detach",
        "--rm",
        "--volume",
        "/tmp/tmpybpszytg/input1:/input:ro",
        "--volume",
        "/tmp/tmpybpszytg/work1:/work:rw",
        "--volume",
        "/tmp/tmpybpszytg/pc1:/pc:rw",
        "pre-commit:1",
        "sleep",
        "infinity"
      ]
    },
    "HostConfig": {
      "Binds": [
        "/tmp/tmpybpszytg/work1:/work:rw,rprivate,rbind",
        "/tmp/tmpybpszytg/pc1:/pc:rw,rprivate,rbind",
        "/tmp/tmpybpszytg/input1:/input:ro,rprivate,rbind"
      ],
      "CgroupMode": "host",
      "ContainerIDFile": "",
      "LogConfig": {
        "Type": "k8s-file",
        "Config": null
      },
      "NetworkMode": "slirp4netns",
      "PortBindings": {},
      "RestartPolicy": {
        "Name": "",
        "MaximumRetryCount": 0
      },
      "AutoRemove": true,
      "VolumeDriver": "",
      "VolumesFrom": null,
      "CapAdd": [],
      "CapDrop": [],
      "Dns": [],
      "DnsOptions": [],
      "DnsSearch": [],
      "ExtraHosts": [],
      "GroupAdd": [],
      "IpcMode": "private",
      "Cgroup": "",
      "Cgroups": "default",
      "Links": null,
      "OomScoreAdj": 0,
      "PidMode": "private",
      "Privileged": false,
      "PublishAllPorts": false,
      "ReadonlyRootfs": false,
      "SecurityOpt": [],
      "Tmpfs": {},
      "UTSMode": "private",
      "UsernsMode": "",
      "ShmSize": 65536000,
      "Runtime": "oci",
      "ConsoleSize": [
        0,
        0
      ],
      "Isolation": "",
      "CpuShares": 0,
      "Memory": 0,
      "NanoCpus": 0,
      "CgroupParent": "",
      "BlkioWeight": 0,
      "BlkioWeightDevice": null,
      "BlkioDeviceReadBps": null,
      "BlkioDeviceWriteBps": null,
      "BlkioDeviceReadIOps": null,
      "BlkioDeviceWriteIOps": null,
      "CpuPeriod": 0,
      "CpuQuota": 0,
      "CpuRealtimePeriod": 0,
      "CpuRealtimeRuntime": 0,
      "CpusetCpus": "",
      "CpusetMems": "",
      "Devices": [],
      "DiskQuota": 0,
      "KernelMemory": 0,
      "MemoryReservation": 0,
      "MemorySwap": 0,
      "MemorySwappiness": 0,
      "OomKillDisable": false,
      "PidsLimit": 0,
      "Ulimits": [],
      "CpuCount": 0,
      "CpuPercent": 0,
      "IOMaximumIOps": 0,
      "IOMaximumBandwidth": 0
    }
  }
]

`podman inspect $cid2` (right before running stop)


[
  {
    "Id": "c8cd61b084c71ecb4cce00faf211482d0dc9b6024bae4950153b7cfb0c1803a0",
    "Created": "2020-08-19T14:54:21.851442236-07:00",
    "Path": "dumb-init",
    "Args": [
      "--",
      "sleep",
      "infinity"
    ],
    "State": {
      "OciVersion": "1.0.2-dev",
      "Status": "running",
      "Running": true,
      "Paused": false,
      "Restarting": false,
      "OOMKilled": false,
      "Dead": false,
      "Pid": 4615,
      "ConmonPid": 4602,
      "ExitCode": 0,
      "Error": "",
      "StartedAt": "2020-08-19T14:54:26.256411375-07:00",
      "FinishedAt": "0001-01-01T00:00:00Z",
      "Healthcheck": {
        "Status": "",
        "FailingStreak": 0,
        "Log": null
      }
    },
    "Image": "42f29002c8b9ee4b1f89044cd301b14aebb41cd84954ceb9f1416e71fcf9286a",
    "ImageName": "localhost/pre-commit:1",
    "Rootfs": "",
    "Pod": "",
    "ResolvConfPath": "",
    "HostnamePath": "/run/user/1000/containers/vfs-containers/c8cd61b084c71ecb4cce00faf211482d0dc9b6024bae4950153b7cfb0c1803a0/userdata/hostname",
    "HostsPath": "",
    "StaticDir": "/home/asottile/.local/share/containers/storage/vfs-containers/c8cd61b084c71ecb4cce00faf211482d0dc9b6024bae4950153b7cfb0c1803a0/userdata",
    "OCIConfigPath": "/home/asottile/.local/share/containers/storage/vfs-containers/c8cd61b084c71ecb4cce00faf211482d0dc9b6024bae4950153b7cfb0c1803a0/userdata/config.json",
    "OCIRuntime": "runc",
    "LogPath": "/home/asottile/.local/share/containers/storage/vfs-containers/c8cd61b084c71ecb4cce00faf211482d0dc9b6024bae4950153b7cfb0c1803a0/userdata/ctr.log",
    "LogTag": "",
    "ConmonPidFile": "/run/user/1000/containers/vfs-containers/c8cd61b084c71ecb4cce00faf211482d0dc9b6024bae4950153b7cfb0c1803a0/userdata/conmon.pid",
    "Name": "objective_hugle",
    "RestartCount": 0,
    "Driver": "vfs",
    "MountLabel": "",
    "ProcessLabel": "",
    "AppArmorProfile": "",
    "EffectiveCaps": [
      "CAP_AUDIT_WRITE",
      "CAP_CHOWN",
      "CAP_DAC_OVERRIDE",
      "CAP_FOWNER",
      "CAP_FSETID",
      "CAP_KILL",
      "CAP_MKNOD",
      "CAP_NET_BIND_SERVICE",
      "CAP_NET_RAW",
      "CAP_SETFCAP",
      "CAP_SETGID",
      "CAP_SETPCAP",
      "CAP_SETUID",
      "CAP_SYS_CHROOT"
    ],
    "BoundingCaps": [
      "CAP_AUDIT_WRITE",
      "CAP_CHOWN",
      "CAP_DAC_OVERRIDE",
      "CAP_FOWNER",
      "CAP_FSETID",
      "CAP_KILL",
      "CAP_MKNOD",
      "CAP_NET_BIND_SERVICE",
      "CAP_NET_RAW",
      "CAP_SETFCAP",
      "CAP_SETGID",
      "CAP_SETPCAP",
      "CAP_SETUID",
      "CAP_SYS_CHROOT"
    ],
    "ExecIDs": [],
    "GraphDriver": {
      "Name": "vfs",
      "Data": null
    },
    "Mounts": [
      {
        "Type": "bind",
        "Name": "",
        "Source": "/tmp/tmpybpszytg/input2",
        "Destination": "/input",
        "Driver": "",
        "Mode": "",
        "Options": [
          "rbind"
        ],
        "RW": false,
        "Propagation": "rprivate"
      },
      {
        "Type": "bind",
        "Name": "",
        "Source": "/tmp/tmpybpszytg/work2",
        "Destination": "/work",
        "Driver": "",
        "Mode": "",
        "Options": [
          "rbind"
        ],
        "RW": true,
        "Propagation": "rprivate"
      },
      {
        "Type": "bind",
        "Name": "",
        "Source": "/tmp/tmpybpszytg/pc2",
        "Destination": "/pc",
        "Driver": "",
        "Mode": "",
        "Options": [
          "rbind"
        ],
        "RW": true,
        "Propagation": "rprivate"
      }
    ],
    "Dependencies": [],
    "NetworkSettings": {
      "EndpointID": "",
      "Gateway": "",
      "IPAddress": "",
      "IPPrefixLen": 0,
      "IPv6Gateway": "",
      "GlobalIPv6Address": "",
      "GlobalIPv6PrefixLen": 0,
      "MacAddress": "",
      "Bridge": "",
      "SandboxID": "",
      "HairpinMode": false,
      "LinkLocalIPv6Address": "",
      "LinkLocalIPv6PrefixLen": 0,
      "Ports": {},
      "SandboxKey": ""
    },
    "ExitCommand": [
      "/usr/bin/podman",
      "--root",
      "/home/asottile/.local/share/containers/storage",
      "--runroot",
      "/run/user/1000/containers",
      "--log-level",
      "error",
      "--cgroup-manager",
      "cgroupfs",
      "--tmpdir",
      "/run/user/1000/libpod/tmp",
      "--runtime",
      "runc",
      "--storage-driver",
      "vfs",
      "--events-backend",
      "file",
      "container",
      "cleanup",
      "--rm",
      "c8cd61b084c71ecb4cce00faf211482d0dc9b6024bae4950153b7cfb0c1803a0"
    ],
    "Namespace": "",
    "IsInfra": false,
    "Config": {
      "Hostname": "c8cd61b084c7",
      "Domainname": "",
      "User": "",
      "AttachStdin": false,
      "AttachStdout": false,
      "AttachStderr": false,
      "Tty": false,
      "OpenStdin": false,
      "StdinOnce": false,
      "Env": [
        "PATH=/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
        "TERM=xterm",
        "container=podman",
        "PRE_COMMIT_HOME=/pc",
        "HOSTNAME=c8cd61b084c7",
        "HOME=/root"
      ],
      "Cmd": [
        "sleep",
        "infinity"
      ],
      "Image": "localhost/pre-commit:1",
      "Volumes": null,
      "WorkingDir": "/",
      "Entrypoint": "dumb-init --",
      "OnBuild": null,
      "Labels": null,
      "Annotations": {
        "io.container.manager": "libpod",
        "io.kubernetes.cri-o.Created": "2020-08-19T14:54:21.851442236-07:00",
        "io.kubernetes.cri-o.TTY": "false",
        "io.podman.annotations.autoremove": "TRUE",
        "io.podman.annotations.init": "FALSE",
        "io.podman.annotations.privileged": "FALSE",
        "io.podman.annotations.publish-all": "FALSE",
        "org.opencontainers.image.stopSignal": "15"
      },
      "StopSignal": 15,
      "CreateCommand": [
        "podman",
        "run",
        "--detach",
        "--rm",
        "--net=none",
        "--volume",
        "/tmp/tmpybpszytg/input2:/input:ro",
        "--volume",
        "/tmp/tmpybpszytg/work2:/work:rw",
        "--volume",
        "/tmp/tmpybpszytg/pc2:/pc:rw",
        "pre-commit:1",
        "sleep",
        "infinity"
      ]
    },
    "HostConfig": {
      "Binds": [
        "/tmp/tmpybpszytg/input2:/input:ro,rprivate,rbind",
        "/tmp/tmpybpszytg/work2:/work:rw,rprivate,rbind",
        "/tmp/tmpybpszytg/pc2:/pc:rw,rprivate,rbind"
      ],
      "CgroupMode": "host",
      "ContainerIDFile": "",
      "LogConfig": {
        "Type": "k8s-file",
        "Config": null
      },
      "NetworkMode": "none",
      "PortBindings": {},
      "RestartPolicy": {
        "Name": "",
        "MaximumRetryCount": 0
      },
      "AutoRemove": true,
      "VolumeDriver": "",
      "VolumesFrom": null,
      "CapAdd": [],
      "CapDrop": [],
      "Dns": [],
      "DnsOptions": [],
      "DnsSearch": [],
      "ExtraHosts": [],
      "GroupAdd": [],
      "IpcMode": "private",
      "Cgroup": "",
      "Cgroups": "default",
      "Links": null,
      "OomScoreAdj": 0,
      "PidMode": "private",
      "Privileged": false,
      "PublishAllPorts": false,
      "ReadonlyRootfs": false,
      "SecurityOpt": [],
      "Tmpfs": {},
      "UTSMode": "private",
      "UsernsMode": "",
      "ShmSize": 65536000,
      "Runtime": "oci",
      "ConsoleSize": [
        0,
        0
      ],
      "Isolation": "",
      "CpuShares": 0,
      "Memory": 0,
      "NanoCpus": 0,
      "CgroupParent": "",
      "BlkioWeight": 0,
      "BlkioWeightDevice": null,
      "BlkioDeviceReadBps": null,
      "BlkioDeviceWriteBps": null,
      "BlkioDeviceReadIOps": null,
      "BlkioDeviceWriteIOps": null,
      "CpuPeriod": 0,
      "CpuQuota": 0,
      "CpuRealtimePeriod": 0,
      "CpuRealtimeRuntime": 0,
      "CpusetCpus": "",
      "CpusetMems": "",
      "Devices": [],
      "DiskQuota": 0,
      "KernelMemory": 0,
      "MemoryReservation": 0,
      "MemorySwap": 0,
      "MemorySwappiness": 0,
      "OomKillDisable": false,
      "PidsLimit": 0,
      "Ulimits": [],
      "CpuCount": 0,
      "CpuPercent": 0,
      "IOMaximumIOps": 0,
      "IOMaximumBandwidth": 0
    }
  }
]

output from `podman stop $cid1 $cid2` (exit status 125)

for example 'podman', 'stop', '295e59ca74f4486cf27dece22be6d7b16ef0d9e54e5053418fb7ebf910b20ac8', 'c8cd61b084c71ecb4cce00faf211482d0dc9b6024bae4950153b7cfb0c1803a0'

note that I've only seen the second cid go "missing"

Error: container c8cd61b084c71ecb4cce00faf211482d0dc9b6024bae4950153b7cfb0c1803a0 does not exist in database: no such container

Describe the results you expected:

I expect there to not be an error

Additional information you deem important (e.g. issue happens only occasionally):

I've only been able to reproduce this about ~10% of the time

Output of podman version:

$ podman version
Version:      2.0.4
API Version:  1
Go Version:   go1.14.2
Built:        Wed Dec 31 16:00:00 1969
OS/Arch:      linux/amd64

Output of podman info --debug:

$ podman info --debug
host:
  arch: amd64
  buildahVersion: 1.15.0
  cgroupVersion: v1
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.20, commit: '
  cpus: 2
  distribution:
    distribution: ubuntu
    version: "20.04"
  eventLogger: file
  hostname: asottile-VirtualBox
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.4.0-42-generic
  linkmode: dynamic
  memFree: 261697536
  memTotal: 4127395840
  ociRuntime:
    name: runc
    package: 'containerd.io: /usr/bin/runc'
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc10
      commit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
      spec: 1.0.1-dev
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.1.4
      commit: unknown
      libslirp: 4.3.1-git
      SLIRP_CONFIG_VERSION_MAX: 3
  swapFree: 1964122112
  swapTotal: 1964396544
  uptime: 12m 40.38s
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /home/asottile/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 0
    stopped: 2
  graphDriverName: vfs
  graphOptions: {}
  graphRoot: /home/asottile/.local/share/containers/storage
  graphStatus: {}
  imageStore:
    number: 8
  runRoot: /run/user/1000/containers
  volumePath: /home/asottile/.local/share/containers/storage/volumes
version:
  APIVersion: 1
  Built: 0
  BuiltTime: Wed Dec 31 16:00:00 1969
  GitCommit: ""
  GoVersion: go1.14.2
  OsArch: linux/amd64
  Version: 2.0.4

Package info (e.g. output of rpm -q podman or apt list podman):

$ apt list podman
Listing... Done
podman/unknown,now 2.0.4~1 amd64 [installed]
podman/unknown 2.0.4~1 arm64
podman/unknown 2.0.4~1 armhf
podman/unknown 2.0.4~1 s390x

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

This is in virtualbox Version 6.0.20 r137117 (Qt5.6.2)

The text was updated successfully, but these errors were encountered:

rhatdan · 2020-08-20T12:44:08Z

Is the container actually still running? Or was it stopped and the error was bogus?

rhatdan · 2020-08-20T12:46:20Z

If the container crashed for some reason, I would expect this behaviour. You could attempt this without the --rm and then maybe we could examine the state of the container, along with the exit code.

mheon · 2020-08-20T13:30:07Z

There is a race that can cause this, between gathering the containers we need to work on, and working on the individual containers - if the container stops/is removed in the gap between those, we get an error. I think we can safely ignore those errors in some cases (e.g. --all specified) but not others (we want to let the user know when a container they specified at the command line stopped existing)

rhatdan · 2020-08-20T13:44:14Z

After we do the podman stop, are we checking to see if the container was stopped? And if the rm happened, then the container no longer exists, so it was stopped. We probably should handle this case and not report an error.

asottile · 2020-08-20T16:13:48Z

Is the container actually still running? Or was it stopped and the error was bogus?

yes, from the podman container inspect ... output you can see it is still running just before running stop. after stop it is stopped so I believe the error is bogus

If the container crashed for some reason, I would expect this behaviour. You could attempt this without the --rm and then maybe we could examine the state of the container, along with the exit code.

yeah good idea, though I'd be hard pressed to see sleep infinity crash :) -- let me try and reproduce it without --rm

asottile · 2020-08-20T16:55:05Z

I've run the same scenario without --rm 150+ times without failure (whereas before I'd get about ~15 failures in that amount of time) so now I'm suspecting it's something related to --rm 🤔

vrothberg · 2020-09-08T09:53:13Z

Thanks for the report, @asottile! It really smells like an internal race caused by --rm.

vrothberg · 2020-09-08T10:03:11Z

Running the above reproducer, I've also seen ERRO[0010] Storage for container d9e60a9b3ff3bee2e32c38c42e5ebb3ec1ff18dc6cd0aadfb6b8c2494cc1ee4d has been removed

Do not perform a container clean up for containers configured for auto-removal (e.g., via `podman run --rm`). There is a small race window with the other process performing the removal where a clean up during podman-stop may fail since the container has already been removed and cleaned up. As the removing process will clean up the container, we don't have to do it during podman-stop. Fixes: containers#7384 Signed-off-by: Valentin Rothberg <[email protected]>

asottile · 2020-09-08T16:19:37Z

awesome, thanks for the fix on this -- I know how tricky finding and fixing these races can be!

openshift-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Aug 19, 2020

vrothberg self-assigned this Sep 8, 2020

vrothberg mentioned this issue Sep 8, 2020

podman stop: do not cleanup for auto-removal #7557

Merged

openshift-merge-robot closed this as completed in #7557 Sep 8, 2020

Luap99 mentioned this issue Sep 1, 2021

podman stop returns before removing a container started with --rm #11384

Closed

github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 22, 2023

github-actions bot locked as resolved and limited conversation to collaborators Sep 22, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`podman stop $cid1 $cid2` successfully stops both but sometimes errors that $cid2 does not exist #7384

`podman stop $cid1 $cid2` successfully stops both but sometimes errors that $cid2 does not exist #7384

asottile commented Aug 19, 2020

rhatdan commented Aug 20, 2020

rhatdan commented Aug 20, 2020

mheon commented Aug 20, 2020

rhatdan commented Aug 20, 2020

asottile commented Aug 20, 2020

asottile commented Aug 20, 2020

vrothberg commented Sep 8, 2020

vrothberg commented Sep 8, 2020

asottile commented Sep 8, 2020

podman stop $cid1 $cid2 successfully stops both but sometimes errors that $cid2 does not exist #7384

podman stop $cid1 $cid2 successfully stops both but sometimes errors that $cid2 does not exist #7384

Comments

asottile commented Aug 19, 2020

podman inspect $cid1 (right before running stop)

podman inspect $cid2 (right before running stop)

output from podman stop $cid1 $cid2 (exit status 125)

rhatdan commented Aug 20, 2020

rhatdan commented Aug 20, 2020

mheon commented Aug 20, 2020

rhatdan commented Aug 20, 2020

asottile commented Aug 20, 2020

asottile commented Aug 20, 2020

vrothberg commented Sep 8, 2020

vrothberg commented Sep 8, 2020

asottile commented Sep 8, 2020

`podman stop $cid1 $cid2` successfully stops both but sometimes errors that $cid2 does not exist #7384

`podman stop $cid1 $cid2` successfully stops both but sometimes errors that $cid2 does not exist #7384

`podman inspect $cid1` (right before running stop)

`podman inspect $cid2` (right before running stop)

output from `podman stop $cid1 $cid2` (exit status 125)