podman generate kube adds mutually exclusive keys CapAdd and privileged

[pciavald]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Copied content from #9077 which was describing 2 issues, feel free to edit. Refer to issue title to know which is which.

When generating the kubernetes manifest for an X11 app that works, the resulting manifest cannot be played for 2 reasons:

• podman generate adds both CapAdd and privileged keys to the manifest resulting in

Error: invalid config provided: CapAdd and privileged are mutually exclusive options

• podman play is unable to open X display on the resulting file while the initial pod was able to.

[1:1:0124/140257.102197:ERROR:browser_main_loop.cc(1473)] Unable to open X display.

Steps to reproduce the issue:

1. create the pod and container, chromium appears, generate the manifest and remove the pod

$ podman pod create --name x11
$ podman run -d \
  --pod x11 \
  --privileged \
  --net host \
  -v /tmp/.X11-unix:/tmp/.X11-unix \
  -e DISPLAY=unix$DISPLAY \
  -v /dev/shm:/dev/shm \
  --name chromium \
  docker.io/jess/chromium
$ podman generate kube x11 >> x11.yml
$ podman pod rm -f x11

2. play the pod: fails because of mutually exclusive options

$ podman play kube ./x11.yml
Error: invalid config provided: CapAdd and privileged are mutually exclusive options
$ podman pod rm -f x11

3. comment out capabilities: or privileged: true and run again

$ podman play kube ./x11.yml
$ podman logs x11-chromium
[1:1:0124/142639.642503:ERROR:browser_main_loop.cc(1473)] Unable to open X display.

Describe the results you received:

• the played pod does not manage to open display
• the generated manifest is invalid

Describe the results you expected:

• the played pod should be able to open display like the original pod
• the generated manifest should be valid

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

$ podman version
Version:      2.2.1
API Version:  2.1.0
Go Version:   go1.15.5
Built:        Tue Dec  8 15:37:50 2020
OS/Arch:      linux/amd64

Output of podman info --debug:

$ podman info --debug
host:
  arch: amd64
  buildahVersion: 1.18.0
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.21-3.fc33.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.21, commit: 0f53fb68333bdead5fe4dc5175703e22cf9882ab'
  cpus: 16
  distribution:
    distribution: fedora
    version: "33"
  eventLogger: journald
  hostname: macpro-fedora
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.10.8-200.fc33.x86_64
  linkmode: dynamic
  memFree: 3506880512
  memTotal: 33651884032
  ociRuntime:
    name: crun
    package: crun-0.16-3.fc33.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.16
      commit: eb0145e5ad4d8207e84a327248af76663d4e50dd
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.8-1.fc33.x86_64
    version: |-
      slirp4netns version 1.1.8
      commit: d361001f495417b880f20329121e3aa431a8f90f
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.0
  swapFree: 3468001280
  swapTotal: 4294963200
  uptime: 95h 52m 39.28s (Approximately 3.96 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /home/peac/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 1
    stopped: 1
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.3.0-1.fc33.x86_64
      Version: |-
        fusermount3 version: 3.9.3
        fuse-overlayfs: version 1.3
        FUSE library version 3.9.3
        using FUSE kernel interface version 7.31
  graphRoot: /home/peac/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 27
  runRoot: /run/user/1000/containers
  volumePath: /home/peac/.local/share/containers/storage/volumes
version:
  APIVersion: 2.1.0
  Built: 1607438270
  BuiltTime: Tue Dec  8 15:37:50 2020
  GitCommit: ""
  GoVersion: go1.15.5
  OsArch: linux/amd64
  Version: 2.2.1

Package info (e.g. output of rpm -q podman or apt list podman):

podman-2.2.1-1.fc33.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

[pciavald]

Looks like it should be pretty easy to fix, Basically don't add cap-add if --privileged is set.

[rhatdan]

Interested in opening a PR?

[pciavald]

I will in the next 10 days if that's alright.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@pciavald Are you still interested in working on this or was this already done?

[rhatdan]

Ok this looks like it is fixed, but while testing I found another issue.

[pciavald]

@rhatdan i can still fix this if needed, sorry i got caught with a never-ending deadline.

[rhatdan]

I believe it is fixed?

[jangel97]

Hello there,

I believe this has not been fixed yet.

podman run -d --name mycontainer  --privileged=True  --volume /sys/fs/cgroup:/sys/fs/cgroup:ro  --tmpfs=/run --tmpfs=/tmp  --cap-add=sys_rawio  --hostname=mycontainer  registry.access.redhat.com/ubi8/ubi-init /usr/sbin/init

Returns:

Error: invalid config provided: CapAdd and privileged are mutually exclusive options

My environment:

root@centos8 ]# podman --version
podman version 3.0.2-dev

[root@centos8]# rpm -q podman
podman-3.0.1-7.module_el8.4.0+830+8027e1c4.x86_64

[root@centos8]# cat /etc/redhat-release 
CentOS Linux release 8.4.2105

[root@centos8]# uname -a
Linux centos8.lab 4.18.0-305.3.1.el8.x86_64 #1 SMP Tue Jun 1 16:14:33 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Podman package has been installed from AppStream CentOS 8 repository.

[mheon]

This is deliberate - --privileged implies all capabilities are added, so the flags are mutually exclusive. Is there a reason why you want to use the two flags together?

[petersenna]

@mheon The reason I have for wanting to specify both flags is to mimic Docker behavior. For the transition period we are going to have docker and podman running, and would be great to not require to make changes to client that currently consumes docker containers.

[ubergeek77]

@mheon

This is deliberate - --privileged implies all capabilities are added, so the flags are mutually exclusive. Is there a reason why you want to use the two flags together?

(Apologies if this is off topic, this is the only discussion I could find of this issue, and I'm not sure if it's a bug)

In rootless mode, I need --privileged to allow the container to use namespaces (for nsjail), but I need --cap-add SYS_ADMIN to allow the container to mount images via FUSE (fuse2fs to mount an ext4 image).

I don't know why, but having just --privileged does not allow a rootless container to mount FUSE filesystems, unless it's running as virtual-root (-u 0). In that case, --privileged is sufficient. Without -u 0, non-UID-0 guest users can only mount FUSE filesystems if --cap-add SYS_ADMIN is specified. But then I lose the ability to use nsjail.

Is this a bug? I know rootless mode has some quirks and limitations, but if --cap-add SYS_ADMIN is really satisfied by --privileged, with the former granting far less permissions and still working fine as a non-root user, I don't understand why the UID of the guest user would make a difference with --privileged.

If you're wondering about my use case - I'm compiling Android in an Ubuntu container, and I would like to use rootless mode if possible. The Android build system complains if it can't use nsjail to isolate the build processes, and another part of my build requires me to mount an ext4 image to copy some files.

(Podman 4.0.1)

[mheon]

Please open a bug report about that, because that sounds like a bug. --privileged should add all capabilities.

[ubergeek77]

Understood. On the off chance a future reader comes across this issue with the same problem, here is the new issue:

#13449