Podman pod logic misbehaves when Kata is the runtime

[zer0def]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

When trying to run pods with Podman using Kata runtime, the user either gets separated containers or just the pod infra container running. This is a copy of the issue posted at kata-containers/runtime#2147.

Steps to reproduce the issue:

1. With Kata Containers installed, the following lines are present in /etc/containers/libpod.conf:

runtime = "kata"
runtime_supports_json = ["runc", "kata"]
kata = ["/usr/bin/kata-runtime"]

2.1. Run: podman pod create --infra=false -n asdf && for i in $(seq 2); do podman create --pod asdf alpine /bin/sh -c 'sleep 3600'; done && podman pod start asdf

2.2. Run: podman pod create -n asdf && podman pod start asdf && for i in $(seq 2); do podman run --pod asdf alpine /bin/sh -c 'sleep 3600'; done

Describe the results you received:

In the case of 2.1, I get two distinct instances of QEMU VMs as runtimes for containers.
In the case of 2.2. I get a running pod infra container and trying to run containers inside errors with Error: Failed to add qdisc for network index 4 : file exists: OCI runtime error. The same situation occurs when trying to run multi-container pods from podman play kube.

Describe the results you expected:

I expected to have a single instance of a Kata runtime (a QEMU VM, essentially) that constitutes a pod, while adding containers to said pod only creates separate shims that effectively are containers within the pod's runtime, as they ordinarily would with something like runc. This behavior works properly with, for example, containerd.

Output of podman version:

Version:            1.6.1
RemoteAPI Version:  1
Go Version:         go1.13.1
OS/Arch:            linux/amd64

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.13.1
  podman version: 1.6.1
host:
  BuildahVersion: 1.11.2
  CgroupVersion: v1
  Conmon:
    package: Unknown
    path: /usr/bin/conmon
    version: 'conmon version 2.0.1, commit: 4dc8bcfec41e10ca760c8e2089474c2843dfd066'
  Distribution:
    distribution: arch
    version: unknown
  MemFree: 3558879232
  MemTotal: 16736239616
  OCIRuntime:
    package: Unknown
    path: /usr/bin/kata-runtime
    version: |-
      kata-runtime  : 1.9.0
         commit   : 6950b6ec8bda3c680ae3f0a4cc5333f45b968e43
         OCI specs: 1.0.1-dev
  SwapFree: 33341927424
  SwapTotal: 33472475136
  arch: amd64
  cpus: 4
  eventlogger: journald
  hostname: hellothere
  kernel: 5.3.7.b-1-hardened
  os: linux
  rootless: false
  uptime: 54h 35m 32.23s (Approximately 2.25 days)
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 1
  GraphDriverName: overlay
  GraphOptions:
    overlay.mountopt: nodev
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  ImageStore:
    number: 5
  RunRoot: /var/run/containers/storage
  VolumePath: /var/lib/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

Name            : podman
Version         : 1.6.1-1
Description     : Tool and library for running OCI-based containers in pods
Architecture    : x86_64
URL             : https://github.com/containers/libpod
Licenses        : Apache
Groups          : None
Provides        : None
Depends On      : cni-plugins  conmon  device-mapper  iptables  libseccomp  ostree  runc  skopeo  btrfs-progs  slirp4netns  libsystemd
Optional Deps   : podman-docker: for Docker-compatible CLI
Required By     : None
Optional For    : None
Conflicts With  : None
Replaces        : None
Installed Size  : 102.76 MiB
Packager        : Bartłomiej Piotrowski <[email protected]>
Build Date      : Fri 04 Oct 2019 02:34:03 PM CEST
Install Date    : Sun 13 Oct 2019 09:49:24 PM CEST
Install Reason  : Explicitly installed
Install Script  : No
Validated By    : Signature

[zer0def]

After poking around in #podman@freenode, there was a number of suggestions:

• @mheon proposed to create a pod without an infra container and use pod create --annotation to attach pods to an appropriate sandbox; thing about this is that there's no first container (and therefore no sandbox) to attach to, but it's a trivial workaround to create one and attach the rest, like in the following example, which solves 2.1:

#!/bin/sh -ex

# create our pod
POD_ID="$(podman pod create --infra=false)"

# podman containers are by default created with an annotation of `io.kubernetes.cri-o.ContainerType=sandbox`, which is sufficient for our purposes
# equivalent to `io.kubernetes.cri.container-type=sandbox`
# so let's mimick an infra container to create our sandbox
SANDBOX_ID="$(podman run -d --pod ${POD_ID} k8s.gcr.io/pause:3.1 /pause)"

# create the first container to exec into later (equivalent to `io.kubernetes.cri.container-type=container` and `io.kubernetes.cri.sandbox-id=${SANDBOX_ID}`)
podman run -d --pod ${POD_ID} --annotation io.kubernetes.cri-o.SandboxID=${SANDBOX_ID} --annotation io.kubernetes.cri-o.ContainerType=container alpine:edge /bin/sh -c 'while true; do sleep 3600; done'

# and another to play around with now
podman run -ti --pod ${POD_ID} --annotation io.kubernetes.cri-o.SandboxID=${SANDBOX_ID} --annotation io.kubernetes.cri-o.ContainerType=container alpine:edge /bin/sh

• pehunt in the same channel proposed to build your own infra image with appropriate labels/annotations that would signal that it's a sandbox (infra containers in pods started by podman have no annotations), but, to my current knowledge, that would require knowing the infra container's sha256 ID in advance, so 2.2 is still busted.

Destroying the pod before destroying attached containers will result in the latter not getting properly cleaned up, so be sure to clean those up before removing the pod.

[mheon]

Should not be difficult to get those annotations added automatically now that we know what to do

…

On Mon, Oct 28, 2019, 17:44 zer0def ***@***.***> wrote: After poking around in ***@***.***, there was a number of suggestions: - @mheon <https://github.com/mheon> proposed to create a pod without an infra container and use pod create --annotation to attach pods to an appropriate sandbox; thing about this is that there's no first container (and therefore no sandbox) to attach to, but it's a trivial workaround to create one and attach the rest, like in the following example, which solves 2.1: #!/bin/sh -ex # create our pod POD_ID="$(podman pod create --infra=false)" # podman containers are by default created with an annotation of `io.kubernetes.cri-o.ContainerType=sandbox`, which is sufficient for our purposes# equivalent to `io.kubernetes.cri.container-type=sandbox`# so let's mimick an infra container to create our sandbox SANDBOX_ID="$(podman run -d --pod ${POD_ID} k8s.gcr.io/pause:3.1 /pause)" # create the first container to exec into later (equivalent to `io.kubernetes.cri.container-type=container`) podman run -d --pod ${POD_ID} --annotation io.kubernetes.cri-o.SandboxID=${SANDBOX_ID} --annotation io.kubernetes.cri-o.ContainerType=container alpine:edge /bin/sh -c 'while true; do sleep 3600; done' # and another to play around with now podman run -ti --pod ${POD_ID} --annotation io.kubernetes.cri-o.SandboxID=${SANDBOX_ID} --annotation io.kubernetes.cri-o.ContainerType=container alpine:edge /bin/sh - pehunt in the same channel proposed to build your own infra image with appropriate labels/annotations that would signal that it's a sandbox (infra containers in pods started by podman have no annotations), but, to my current knowledge, that would require knowing the infra container's sha256 ID in advance, so 2.2 is still busted. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4353>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB3AOCDMMGTBPMCBJESLYPTQQ5MLPANCNFSM4JF4RAFA> .

[rhatdan]

Peter what do you think about adding these annotations automatically?

[haircommander]

sure, I should be able to get to it today or tomorrow

[haircommander]

I've opened a PR to add the annotations automatically, so that part should be solved. The label problem will be solved by an evolution of #4265 (where we can set security and namespace information on pod creation)

[zer0def]

I think that since #4368 got merged into master and covers two main ways of launching pods, this issue can be marked as resolved.

[haircommander]

Thanks for the guidance @zer0def !