play kube with symlinked working directory in container

[0pq76r]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description
podman play kube cannot create a container in the pod if the container's working directory is a symlink.

Steps to reproduce the issue:

sudo podman play kube - << EOF 
# Save the output of this file and use kubectl create -f to import
# it into Kubernetes.
#
# Created with podman-4.0.1
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: "2022-02-25T08:06:50Z"
  labels:
    app: amq
  name: amq
spec:
  containers:
  - image: docker.io/rmohr/activemq:5.15.6
    name: tenderhypatia
    resources: {}
    securityContext:
      capabilities:
        drop:
        - CAP_MKNOD
        - CAP_NET_RAW
        - CAP_AUDIT_WRITE
    tty: true
  restartPolicy: Never
status: {}
EOF

Describe the results you received:

error starting container 79b8fb8111c547e7cefdb3d3aa9b804d59dea04cb17d5371a9893a9a7b59138f: workdir "/opt/activemq" does not exist on container 79b8fb8111c547e7cefdb3d3aa9b804d59dea04cb17d5371a9893a9a7b59138f]
Pod:
157a4041f7cf2cf37dab7710e7bf1bb4a13ab29952c51e3ec57afb890ebaebcc
Container:
79b8fb8111c547e7cefdb3d3aa9b804d59dea04cb17d5371a9893a9a7b59138f

error starting container 79b8fb8111c547e7cefdb3d3aa9b804d59dea04cb17d5371a9893a9a7b59138f: workdir "/opt/activemq" does not exist on container 79b8fb8111c547e7cefdb3d3aa9b804d59dea04cb17d5371a9893a9a7b59138f

Error: failed to start 1 containers

Describe the results you expected:

The container should be started as if running:

sudo podman run -dt --pod amq docker.io/rmohr/activemq:5.15.6

Additional information you deem important (e.g. issue happens only occasionally):

WorkingDir points to a symlink. Replacing it with its target workingDir: /opt/apache-activemq-5.15.6 allows to start the pod as expected.

sudo podman play kube - << EOF
# Save the output of this file and use kubectl create -f to import
# it into Kubernetes.
#
# Created with podman-4.0.1
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: "2022-02-25T08:06:50Z"
  labels:
    app: amq
  name: amq
spec:
  containers:
  - image: docker.io/rmohr/activemq:5.15.6
    name: tenderhypatia
    resources: {}
    securityContext:
      capabilities:
        drop:
        - CAP_MKNOD
        - CAP_NET_RAW
        - CAP_AUDIT_WRITE
    tty: true
    workingDir: /opt/apache-activemq-5.15.6
  restartPolicy: Never
status: {}
EOF

Output of podman version:

sudo podman version
Client:       Podman Engine
Version:      4.0.1
API Version:  4.0.1
Go Version:   go1.17.7

Built:      Thu Jan  1 01:00:00 1970
OS/Arch:    linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.24.1
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: 'conmon: /usr/bin/conmon'
    path: /usr/bin/conmon
    version: 'conmon version 2.0.25, commit: unknown'
  cpus: 8
  distribution:
    distribution: debian
    version: unknown
  eventLogger: journald
  hostname: debian
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.16.0-2-amd64
  linkmode: dynamic
  logDriver: journald
  memFree: 6762123264
  memTotal: 16446877696
  networkBackend: cni
  ociRuntime:
    name: runc
    package: 'runc: /usr/bin/runc'
    path: /usr/bin/runc
    version: |-
      runc version 1.1.0+ds1
      commit: 1.1.0+ds1-1
      spec: 1.0.2-dev
      go: go1.17.6
      libseccomp: 2.5.3
  os: linux
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: true
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.0.1
      commit: 6a7b16babc95b6a3056b33fb45b74a6f62262dd4
      libslirp: 4.6.1
  swapFree: 3221221376
  swapTotal: 3221221376
  uptime: 1h 44m 45.63s (Approximately 0.04 days)
plugins:
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  registry.avintis:5000:
    Blocked: false
    Insecure: true
    Location: registry.avintis:5000
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: registry.avintis:5000
  search:
  - docker.io
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 9
    paused: 0
    running: 3
    stopped: 6
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 18
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.0.1
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.17.7
  OsArch: linux/amd64
  Version: 4.0.1

Package info (e.g. output of rpm -q podman or apt list podman):

apt list podman
Listing... Done
podman/experimental,now 4.0.1+ds1-1 amd64 [installed]
podman/unknown 100:3.0.1-2 armhf
podman/unstable,unstable 3.4.4+ds1-1 i386

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes

[mheon]

@umohnani8 @flouthoc Mind taking a look?

[flouthoc]

@0pq76r Thanks for creating the issue

This is more of a podman issue rather than direct podman play kube, podman seems to ignore if workdir dir is symlink and config does not allow podman to create a new workdir mostly happens with run --workdir /symlink/path and since podman play kube curates similar config in indirect manner it faces the same issue.

Above PR should close this issue, I have also added a podman play kube test to verify for kube as well but its a podman bug general bug.