kube play: Cannot set shmsize when running in the host IPC Namespace

[karta0807913]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Get the error message Error: invalid config provided: cannot set shmsize when running in the {host } IPC Namespace when using the root full kube play command.

Steps to reproduce the issue:

You can use this command to reproduce the error.

podman run --rm -i --privileged quay.io/podman/stable:v4.3 podman kube play - < ./busybox.yaml

busybox.yaml

apiVersion: v1
kind: Pod
metadata:
  name: busybox1
  labels:
    name: busybox
spec:
  containers:
  - image: busybox:1.28
    command:
      - sleep
      - "3600"
    name: busybox

Describe the results you received:

Resolved "busybox" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull docker.io/library/busybox:1.28...
Getting image source signatures
Copying blob sha256:07a152489297fc2bca20be96fab3527ceac5668328a30fd543a160cd689ee548
Copying config sha256:8c811b4aec35f259572d0f79207bc0678df4c736eeec50bc9fec37ed936a472a
Writing manifest to image destination
Storing signatures
Error: invalid config provided: cannot set shmsize when running in the {host } IPC Namespace

Describe the results you expected:

Container runs successfully.

Output of podman version:

Client:       Podman Engine
Version:      4.3.1
API Version:  4.3.1
Go Version:   go1.19.2
Built:        Fri Nov 11 15:01:27 2022
OS/Arch:      linux/amd64

Output of podman info:

host:
  arch: amd64
  buildahVersion: 1.28.0
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.1.5-1.fc37.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.5, commit: '
  cpuUtilization:
    idlePercent: 94.31
    systemPercent: 1.54
    userPercent: 4.15
  cpus: 8
  distribution:
    distribution: fedora
    variant: container
    version: "37"
  eventLogger: file
  hostname: 8be4787bd6d4
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.15.0-53-generic
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 3810107392
  memTotal: 16131186688
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.7-1.fc37.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.7
      commit: 40d996ea8a827981895ce22886a9bac367f87264
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-8.fc37.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 2147479552
  swapTotal: 2147479552
  uptime: 18h 52m 19.00s (Approximately 0.75 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.imagestore: /var/lib/shared
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.9-6.fc37.x86_64
      Version: |-
        fusermount3 version: 3.10.5
        fuse-overlayfs: version 1.9
        FUSE library version 3.10.5
        using FUSE kernel interface version 7.31
    overlay.mountopt: nodev,fsync=0
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 502392610816
  graphRootUsed: 244982870016
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.3.1
  Built: 1668178887
  BuiltTime: Fri Nov 11 15:01:27 2022
  GitCommit: ""
  GoVersion: go1.19.2
  Os: linux
  OsArch: linux/amd64
  Version: 4.3.1

Package info (e.g. output of rpm -q podman or apt list podman or brew info podman):

podman-4.3.1-1.fc37.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

I checked the code and find an override logic in Inherit function.

podman/pkg/specgen/generate/container_create.go

Lines 614 to 617 in a891199

	// this causes errors when shmSize is the default value, it will still get passed down unless we manually override.
	if s.IpcNS.NSMode == specgen.Host && (compatibleOptions.ShmSize != nil && compatibleOptions.IsDefaultShmSize()) {
	s.ShmSize = nil
	}

I think this override logic never be executed.
It's because the variable s was unmarshaled from compatibleOptions, and libpod.InfraInherit doesn't have the ipcns field. And that means s.IpcNS.NSMode always is empty.

Therefore, I think this is the correct version; change the s.IpcNS.NSMode to the inheritSpec.IpcNS.NSMode

// this causes errors when shmSize is the default value, it will still get passed down unless we manually override.
if inheritSpec.IpcNS.NSMode == specgen.Host && (compatibleOptions.ShmSize != nil && compatibleOptions.IsDefaultShmSize()) {
    s.ShmSize = nil
}

[karta0807913]

After I made my patch, I get another error.
I am not sure my patch correct or not :(

Error: pod 404a401e6d02cb326115906a24e5124fb674cd2299637b0d543d2e25d9fc3400 cgroup is not set: internal libpod error

[rhatdan]

Can you open a PR for the first fix?

@giuseppe thoughts on the second issue?

You might have to move to cgroupv2.

[karta0807913]

hello @rhatdan
I create a PR here #16668

Also, I think the second error is not caused by cgroupv2.
That's because the command podman run --rm -i --privileged quay.io/podman/stable:v4.3 podman run --rm -i --pod new:pod-name ubuntu:20.04 works fine on my machine.

Moreover, here is the default configure file in the podman image; you can see that cgroup is disabled in the root mode.
/etc/containers/containers.conf

[containers]
netns="host"
userns="host"
ipcns="host"
utsns="host"
cgroupns="host"
cgroups="disabled"
log_driver = "k8s-file"
[engine]
cgroup_manager = "cgroupfs"
events_logger="file"
runtime="crun"

That means the error won't be raised (L365).

podman/libpod/runtime_ctr.go

Lines 350 to 376 in c49de22

	if !ctr.config.NoCgroups {
	ctr.config.CgroupManager = r.config.Engine.CgroupManager
	switch r.config.Engine.CgroupManager {
	case config.CgroupfsCgroupsManager:
	if ctr.config.CgroupParent == "" {
	if pod != nil && pod.config.UsePodCgroup && !ctr.IsInfra() {
	podCgroup, err := pod.CgroupPath()
	if err != nil {
	return nil, fmt.Errorf("retrieving pod %s cgroup: %w", pod.ID(), err)
	}
	expectPodCgroup, err := ctr.expectPodCgroup()
	if err != nil {
	return nil, err
	}
	if expectPodCgroup && podCgroup == "" {
	return nil, fmt.Errorf("pod %s cgroup is not set: %w", pod.ID(), define.ErrInternal)
	}
	canUseCgroup := !rootless.IsRootless() || isRootlessCgroupSet(podCgroup)
	if canUseCgroup {
	ctr.config.CgroupParent = podCgroup
	}
	} else if !rootless.IsRootless() {
	ctr.config.CgroupParent = CgroupfsDefaultCgroupParent
	}
	} else if strings.HasSuffix(path.Base(ctr.config.CgroupParent), ".slice") {
	return nil, fmt.Errorf("systemd slice received as cgroup parent when using cgroupfs: %w", define.ErrInvalidArg)
	}

(if cgroup="disabled", s.NoCgroups will be true)

podman/libpod/options.go

Lines 1142 to 1160 in c49de22

	func WithCgroupsMode(mode string) CtrCreateOption {
	return func(ctr *Container) error {
	if ctr.valid {
	return define.ErrCtrFinalized
	}
	switch mode {
	case "disabled":
	ctr.config.NoCgroups = true
	ctr.config.CgroupsMode = mode
	case "enabled", "no-conmon", cgroupSplit:
	ctr.config.CgroupsMode = mode
	default:
	return fmt.Errorf("invalid cgroup mode %q: %w", mode, define.ErrInvalidArg)
	}
	return nil
	}
	}

However, s.CgroupsMode is empty in podman kuebe play command.
So, I check the code again and find the function FillOutSpecGen which is called by podman run:

podman/pkg/specgenutil/specgen.go

Lines 529 to 537 in c49de22

	if len(s.CgroupParent) == 0 || len(c.CgroupParent) != 0 {
	s.CgroupParent = c.CgroupParent
	}
	if len(s.CgroupsMode) == 0 {
	s.CgroupsMode = c.CgroupsMode
	}
	if s.CgroupsMode == "" {
	s.CgroupsMode = rtc.Cgroups()
	}

It fills up the s.CgroupsMode and returns specgen.SpecGenerator and entities.ContainerCreateOptions, but the ToSpecGen function doesn't set this field.

At the end, I did a very roughly patch to fix my problem.
karta0807913@4527b5e

I hope I described before can help you fix this problem easier :D.

[giuseppe]

At the end, I did a very roughly patch to fix my problem.
karta0807913@4527b5e

the patch LGTM. Could you add it to your open PR?

[karta0807913]

Sure, let me add a unit test too

[karta0807913]

Merged