Play Kube --log-opt causing panic

[judokan9]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Podman throwing an exception when run with --log-opt tag="{{.ImageName}}"

Steps to reproduce the issue:

1. Install Debian 11 (Raspberry Pi OS Lite 64Bit 28.01.2022), update the system with testing repos and add experimental repos

2. Install Podman with sudo apt -t experimental install podman

3. Run podman play kube gitea-pod.yml --configmap=gitea-app-configmap.yml,gitea-db-configmap.yml --log-driver=journald --log-opt tag="{{.ImageName}}"

Describe the results you received:

panic: assignment to entry in nil map

goroutine 1 [running]:
github.com/containers/podman/pkg/specgen/generate/kube.ToSpecGen({0x1af9fc8, 0x4000a13ce0}, 0x400101e0c8)
	github.com/containers/podman/pkg/specgen/generate/kube/kube.go:167 +0x2100
github.com/containers/podman/pkg/domain/infra/abi.(*ContainerEngine).playKubePod(0x4000010580, {0x1af9fc8, 0x4000a13ce0}, {0x40007dda28, 0x5}, 0x4000dbf650, {{0x0, 0x0}, 0x0, {0x0, ...}, ...}, ...)
	github.com/containers/podman/pkg/domain/infra/abi/play.go:424 +0x183c
github.com/containers/podman/pkg/domain/infra/abi.(*ContainerEngine).PlayKube(0x4000010580, {0x1af9fc8, 0x4000a13ce0}, {0x7feac23760, 0xd}, {{0x0, 0x0}, 0x0, {0x0, 0x0}, ...})
	github.com/containers/podman/pkg/domain/infra/abi/play.go:82 +0x3f8
github.com/containers/podman/cmd/podman/play.playkube({0x7feac23760, 0xd})
	github.com/containers/podman/cmd/podman/play/kube.go:221 +0x9c
github.com/containers/podman/cmd/podman/play.kube(0x263d660, {0x40004bcaf0, 0x1, 0x5})
	github.com/containers/podman/cmd/podman/play/kube.go:179 +0x344
github.com/spf13/cobra.(*Command).execute(0x263d660, {0x400013c030, 0x5, 0x5})
	github.com/spf13/cobra/command.go:856 +0x678
github.com/spf13/cobra.(*Command).ExecuteC(0x2646260)
	github.com/spf13/cobra/command.go:974 +0x424
github.com/spf13/cobra.(*Command).Execute(...)
	github.com/spf13/cobra/command.go:902
github.com/spf13/cobra.(*Command).ExecuteContext(...)
	github.com/spf13/cobra/command.go:895
main.Execute()
	github.com/containers/podman/cmd/podman/root.go:100 +0x110
main.main()
	github.com/containers/podman/cmd/podman/main.go:39 +0xc4

Describe the results you expected:
An Running Pod with tags in journald logs

Additional information you deem important (e.g. issue happens only occasionally):
Other options like --log-opt max-size=10mb,path=/var/log/mycontainer.json do not raise an exception but also don't seem to work, because nothing is written to the file even when the Container puts out logs (file permissions are looking good).

I used the following YAML to in the Command:

(The Config maps only contain secrets and env vars)

gitea-pod.yml

---
apiVersion: v1
kind: Pod
metadata:
  labels:
    app: gitea
  name: gitea
spec:
  containers:
  - name: gitea-app
    image: gitea/gitea:1.16.1-rootless
    envFrom:
    - configMapRef:
        name: gitea-app-configmap
        optional: false
    ports:
    - containerPort: 3000
      hostPort: 3000
      protocol: TCP
    - containerPort: 2222
      hostPort: 2222
      protocol: TCP
    volumeMounts:
    - mountPath: /var/lib/gitea
      name: gitea-app-data
    - mountPath: /etc/gitea
      name: gitea-app-config
    - mountPath: /etc/timezone
      name: gitea-app-timezone
    - mountPath: /etc/localtime
      name: gitea-app-localtime

  - name: gitea-db
    image: postgres:14.2-alpine
    envFrom:
    - configMapRef:
        name: gitea-db-configmap
        optional: false
    volumeMounts:
    - mountPath: /var/lib/postgresql/data
      name: gitea-db-data

  - name: gitea-cache
    image: redis:6.2.6-alpine

  volumes:
  - hostPath:
      path: app
      type: DirectoryOrCreate
    name: gitea-app-data
  - hostPath:
      path: config
      type: DirectoryOrCreate
    name: gitea-app-config
  - hostPath:
      path: /etc/timezone
      readOnly: true
      type: File
    name: gitea-app-timezone
  - hostPath:
      path: /etc/localtime
      readOnly: true
      type: File
    name: gitea-app-localtime
  - hostPath:
      path: db
      type: DirectoryOrCreate
    name: gitea-db-data
...

$ lscpu 
Architecture:                    aarch64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
CPU(s):                          4
On-line CPU(s) list:             0-3
Thread(s) per core:              1
Core(s) per socket:              4
Socket(s):                       1
Vendor ID:                       ARM
Model:                           3
Model name:                      Cortex-A72
Stepping:                        r0p3
CPU max MHz:                     1800.0000
CPU min MHz:                     600.0000
BogoMIPS:                        108.00
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Spec store bypass: Vulnerable
Vulnerability Spectre v1:        Mitigation; __user pointer sanitization
Vulnerability Spectre v2:        Vulnerable
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fp asimd evtstrm crc32 cpuid

Output of podman version:

$ podman version
Client:       Podman Engine
Version:      4.0.1
API Version:  4.0.1
Go Version:   go1.17.7

Built:      Thu Jan  1 01:00:00 1970
OS/Arch:    linux/arm64

Output of podman info --debug:

$ podman info --debug
host:
  arch: arm64
  buildahVersion: 1.24.1
  cgroupControllers:
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: 'conmon: /usr/bin/conmon'
    path: /usr/bin/conmon
    version: 'conmon version 2.0.25, commit: unknown'
  cpus: 4
  distribution:
    distribution: debian
    version: unknown
  eventLogger: journald
  hostname: raspberrypi
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.10.92-v8+
  linkmode: dynamic
  logDriver: journald
  memFree: 4413526016
  memTotal: 8244846592
  networkBackend: cni
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version 0.17
      commit: 0e9229ae34caaebcb86f1fde18de3acaf18c6d9a
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.0.1
      commit: 6a7b16babc95b6a3056b33fb45b74a6f62262dd4
      libslirp: 4.6.1
  swapFree: 104853504
  swapTotal: 104853504
  uptime: 30m 20.08s
plugins:
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}
store:
  configFile: /home/pi/.config/containers/storage.conf
  containerStore:
    number: 14
    paused: 0
    running: 9
    stopped: 5
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: 'fuse-overlayfs: /usr/bin/fuse-overlayfs'
      Version: |-
        fusermount3 version: 3.10.5
        fuse-overlayfs: version 1.7.1
        FUSE library version 3.10.5
        using FUSE kernel interface version 7.31
  graphRoot: /home/pi/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 13
  runRoot: /run/user/1000/containers
  volumePath: /home/pi/.local/share/containers/storage/volumes
version:
  APIVersion: 4.0.1
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.17.7
  OsArch: linux/arm64
  Version: 4.0.1

Package info (e.g. output of rpm -q podman or apt list podman):

apt list podman
Listing... Done
podman/experimental,now 4.0.1+ds1-2 arm64 [installed]
podman/testing 3.4.4+ds1-1 armhf

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

[mheon]

@umohnani8 PTAL

[vrothberg]

I can tackle this one (looking for some small issues until lunch break)

[vrothberg]

Unassigning, this is not a quick-fix. Fixing the nil-deref is easy but it seems these options don't do what they are supposed to, so containers won't start anymore.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@cdoern PTAL

[cdoern]

I think I can figure out why the options are misfiring in such a drastic way. I will work on this tmrw

[cdoern]

@vrothberg I have put some debugs in here and it seems like adding a make(... for the options map fixes the issue. Conmon gets the logging tag as seen in:

podman/libpod/oci_conmon_linux.go

Lines 1406 to 1407 in c3d871a

	if logTag != "" {
	args = append(args, "--log-tag", logTag)

I am trying to see how this is a larger issue beyond the nil pointer deref but can't seem to find a larger issue. Thoughts?

[vrothberg]

@cdoern, looking at my earlier comment, containers didn't start anymore when I attempted to fix it. If that is not the case in your fix, that is good news :)