[APIv2] killing a container hangs after "Switch podman stop/kill/wait handlers to use abi"

[riyad]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Trying to find regressions by exercising the APIv2 trough docker-py's test suite (see #5386) I came across a regressions in the RestartContainerTest::test_restart_with_low_timeout test.

Killing a container hangs with no result returned and the CPU spinning like crazy.

It specifically broke with commit 81a3ba3 .

Steps to reproduce the issue:

1. curl -sS --unix-socket /var/run/user/1000/podman/podman.sock -XPOST 'http://localhost/v1.40/containers/create' -H 'Content-Type: application/json' -d '{"Tty": false, "OpenStdin": false, "StdinOnce": false, "AttachStdin": false, "AttachStdout": true, "AttachStderr": true, "Cmd": ["sleep", "9999"], "Image": "alpine", "NetworkDisabled": false}'

2. curl -sS --unix-socket /var/run/user/1000/podman/podman.sock -XPOST 'http://localhost/v1.40/containers/398e20bb6c01742c1aebe09cd196efbfa6d82873561968fbf96dd4ca0b16198f/start'

3. curl -sS --unix-socket /var/run/user/1000/podman/podman.sock -XPOST 'http://localhost/v1.40/containers/398e20bb6c01742c1aebe09cd196efbfa6d82873561968fbf96dd4ca0b16198f/kill'

Describe the results you received:

The kill call hangs with no response while the CPU starts spinning like crazy.

Describe the results you expected:

Kill the container and return result immediately.

Additional information you deem important (e.g. issue happens only occasionally):

Running podman ps -a in another terminal while the kill call spins CPU shows that the container has stopped:

$ podman ps -a
CONTAINER ID  IMAGE                            COMMAND     CREATED        STATUS                      PORTS   NAMES
398e20bb6c01  docker.io/library/alpine:latest  sleep 9999  3 minutes ago  Exited (137) 2 minutes ago          modest_brown

Output of podman version:

Version:      3.0.0-dev
API Version:  3.0.0
Go Version:   go1.15.6
Git Commit:   81a3ba36ae8d22b767e620c0b4c804764340de52
Built:        Tue Feb  2 21:21:44 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.19.3
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.24, commit: '
  cpus: 4
  distribution:
    distribution: ubuntu
    version: "20.10"
  eventLogger: journald
  hostname: acnologia
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.8.0-41-generic
  linkmode: dynamic
  memFree: 1009451008
  memTotal: 16540594176
  ociRuntime:
    name: runc
    package: 'cri-o-runc: /usr/lib/cri-o-runc/sbin/runc'
    path: /usr/lib/cri-o-runc/sbin/runc
    version: 'runc version spec: 1.0.2-dev'
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_AUDIT_WRITE,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_MKNOD,CAP_NET_BIND_SERVICE,CAP_NET_RAW,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    selinuxEnabled: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.1.8
      commit: unknown
      libslirp: 4.3.1-git
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.4.3
  swapFree: 2136207360
  swapTotal: 2147479552
  uptime: 24h 16m 14.48s (Approximately 1.00 days)
registries:
  search:
  - docker.io
store:
  configFile: /home/riyad/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: 'fuse-overlayfs: /usr/bin/fuse-overlayfs'
      Version: |-
        fusermount3 version: 3.9.3
        fuse-overlayfs: version 1.4
        FUSE library version 3.9.3
        using FUSE kernel interface version 7.31
    overlay.mountopt: nodev
  graphRoot: /home/riyad/podman/storage
  graphStatus:
    Backing Filesystem: zfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 23
  runRoot: /run/user/1000/containers
  volumePath: /home/riyad/podman/storage/volumes
version:
  APIVersion: 3.0.0
  Built: 1612297304
  BuiltTime: Tue Feb  2 21:21:44 2021
  GitCommit: 81a3ba36ae8d22b767e620c0b4c804764340de52
  GoVersion: go1.15.6
  OsArch: linux/amd64
  Version: 3.0.0-dev

[baude]

@rhatdan that is your recent commit ... mind peeking at this in the chance that the code is fresh in your mind?