Podman 3.0 API behavior change in containers/<id>/wait breaks nomad-driver-podman

[brent-holden]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

There was a change in REST API behavior from v2.2 to v3.0 using /containers/<id>/wait?condition=running

Previously, in v2.1/2, you could make a post to this API and a running container would return immediately. In v3.0 that behavior has changed to wait indefinitely, which breaks the podman Nomad driver as the driver times out waiting for a successful response to the container entering a running state. This was observed on a CentOS Stream host using the latest updates to podman.

Steps to reproduce the issue:

Using a redis container as an example:

1. podman run -d --rm redis

2. curl --unix-socket /run/podman/podman.sock -v 'http://d/v1.0.0/libpod/containers/json' | jq

3. curl --unix-socket /run/podman/podman.sock -v -X POST 'http://d/v1.0.0/libpod/containers/<id>/wait?condition=running'

Describe the results you received:

From the first API request, I received the Id, which I was then able to replace into the second POST command. Once that has been done, the API request hangs indefinitely. Id, in this case, has been replaced by <id>:

curl --unix-socket /run/podman/podman.sock -v -X POST 'http://d/v1.0.0/libpod/containers/<id>/wait?condition=running'
*   Trying /run/podman/podman.sock...
* Connected to d (/run/podman/podman.sock) port 80 (#0)
> POST /v1.0.0/libpod/containers/<id>/wait?condition=running HTTP/1.1
> Host: d
> User-Agent: curl/7.61.1
> Accept: */*
>

Describe the results you expected:

curl --unix-socket /run/podman/podman.sock -v -X POST 'http://d/v1.0.0/libpod/containers/<id>/wait?condition=running'
*   Trying /run/podman/podman.sock...
* Connected to d (/run/podman/podman.sock) port 80 (#0)
> POST /v1.0.0/libpod/containers/<id>/wait?condition=running HTTP/1.1
> Host: d
> User-Agent: curl/7.61.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Api-Version: 1.40
< Content-Type: text/plain; charset=us-ascii
< Libpod-Api-Version: 2.0.0
< Server: Libpod/2.0.0 (linux)
< Date: Tue, 16 Feb 2021 20:43:19 GMT
< Content-Length: 3
<
-1
* Connection #0 to host d left intact

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

podman version
Version:      3.0.0
API Version:  3.0.0
Go Version:   go1.14.12
Built:        Sun Feb 14 12:48:35 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.19.2
  cgroupManager: systemd
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.26-2.el8.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.26, commit: 8833a1127dabe0d76c18cc8f894ac48b62dfbda9'
  cpus: 40
  distribution:
    distribution: '"centos"'
    version: "8"
  eventLogger: journald
  hostname: daedalus.hostname
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 4.18.0-277.el8.x86_64
  linkmode: dynamic
  memFree: 62314332160
  memTotal: 67460542464
  ociRuntime:
    name: crun
    package: crun-0.16-1.el8.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.16
      commit: eb0145e5ad4d8207e84a327248af76663d4e50dd
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    selinuxEnabled: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 4294963200
  swapTotal: 4294963200
  uptime: 2h 25m 59.49s (Approximately 0.08 days)
registries:
  search:
  - docker.io
  - registry.fedoraproject.org
  - registry.centos.org
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageStore:
    number: 11
  runRoot: /var/run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 3.0.0
  Built: 1613324915
  BuiltTime: Sun Feb 14 12:48:35 2021
  GitCommit: ""
  GoVersion: go1.14.12
  OsArch: linux/amd64
  Version: 3.0.0

Package info (e.g. output of rpm -q podman or apt list podman):

podman-3.0.0-2.el8.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

Bare-metal host using CentOS Stream release 8

[rhatdan]

Are you talking to the libpod interface or the compatible. I believe only the compatibilty one waits.

[brent-holden]

Looks like this is a dup of #9351