podman play kube fails to launch initContainers without a PATH in the environment

[dcermak]

Issue Description

It appears that podman play kube is not able to handle a initContainers in a deployment where the image has no PATH defined in Config.Env.

E.g. the following deployment fails to launch:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: rmt-front
spec:
  template:
    spec:
      containers:
      - name: "rmt-front"
        image: "registry.suse.com/suse/rmt-nginx:1.19"
        imagePullPolicy: Always
      initContainers:
      - name: "rmt-front-init"
        image: "registry.suse.com/bci/bci-micro:15.3"
        imagePullPolicy: Always
        command: ['sh', '-c', 'echo foo']

You get the following error:

 ❯ podman play kube /tmp/podman-play/rmt-app.yml
Trying to pull registry.suse.com/bci/bci-micro:15.3...
Getting image source signatures
Copying blob 534639c69224 skipped: already exists  
Copying config 0ddf45aab0 done  
Writing manifest to image destination
Storing signatures
Trying to pull registry.suse.com/suse/rmt-nginx:1.19...
Getting image source signatures
Copying blob 7448fc1a6468 skipped: already exists  
Copying blob a8e7a8d5a3ff skipped: already exists  
Copying config 2e5e2fc16e done  
Writing manifest to image destination
Storing signatures
Error: encountered while bringing up pod rmt-front-pod: crun: executable file `sh` not found in $PATH: No such file or directory: OCI runtime attempted to invoke a command that was not found

However, if you swap out "registry.suse.com/bci/bci-micro:15.3" for e.g. "docker.io/busybox", then everything works.

The difference is that registry.suse.com/bci/bci-micro:15.3 has not been built via docker, but with kiwi and has no PATH entry in Config.Env:

❯ podman inspect -f {{.Config.Env}} registry.suse.com/bci/bci-micro:15.3
[]

❯ podman inspect -f {{.Config.Env}} docker.io/busybox
[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin]

Also, the missing PATH entry is not the actual problem, because if I try to emulate what podman play kube does internally with initContainers (see

podman/pkg/specgen/generate/kube/kube.go

Line 337 in 19152fa

if !opts.IsInfra && len(opts.Container.Command) != 0 {

), i.e. set CMD to nil and ENTRYPOINT to CMD as follows, everything still works:

❯ podman run --rm --entrypoint '["sh", "-c", "echo hello"]' registry.suse.com/bci/bci-base:15.3
hello

Steps to reproduce the issue

Steps to reproduce the issue

1. create the above yaml snippet
2. run podman play kube /path/to/yaml

Describe the results you received

Error: encountered while bringing up pod rmt-front-pod: crun: executable file sh not found in $PATH: No such file or directory: OCI runtime attempted to invoke a command that was not found

Describe the results you expected

The first initContainers should have been created.

podman info output

I think this is irrelevant, but for completeness:
host:
  arch: amd64
  buildahVersion: 1.30.0
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.7-2.fc38.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: '
  cpuUtilization:
    idlePercent: 87.91
    systemPercent: 1.98
    userPercent: 10.11
  cpus: 16
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    version: "38"
  eventLogger: journald
  hostname: Skirnir
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 10000
      size: 65536
    - container_id: 65537
      host_id: 524288
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 10000
      size: 65536
    - container_id: 65537
      host_id: 524288
      size: 65536
  kernel: 6.2.12-300.fc38.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 3897225216
  memTotal: 33354240000
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.8.3-2.fc38.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.3
      commit: 59f2beb7efb0d35611d5818fd0311883676f6f7e
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-12.fc38.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 8579444736
  swapTotal: 8589930496
  uptime: 20h 16m 52.00s (Approximately 0.83 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
  - registry.suse.com
store:
  configFile: /home/dan/.config/containers/storage.conf
  containerStore:
    number: 3
    paused: 0
    running: 1
    stopped: 2
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/dan/.local/share/containers/storage
  graphRootAllocated: 510389125120
  graphRootUsed: 180980338688
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 257
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/dan/.local/share/containers/storage/volumes
version:
  APIVersion: 4.5.0
  Built: 1681486942
  BuiltTime: Fri Apr 14 17:42:22 2023
  GitCommit: ""
  GoVersion: go1.20.2
  Os: linux
  OsArch: linux/amd64
  Version: 4.5.0

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

I have tried this with 4.5.0 and with the current HEAD (19152fa)

Additional information

No response

[dcermak]

Ok, so the issue is probably, that generate.CompleteSpec is not called for initContainers.