Unable to access /dev/tty0 in a rootless container

[GrabbenD]

Issue Description

I'm attempting to run Sway in rootless container but I'm unable to access /dev/tty0 thus I can't create a graphical session.

Possibly related: #15878

Here's what I've tried:

• Passing --device /dev/tty0 without success: ls: cannot access '/dev/tty0': No such file or directory
• Adding this as a volume instead -v=/dev/tty0:/dev/tty0:rslave results in [seatd/seat.c:60] Could not open tty0 to update VT: Permission denied
• Persisting user's permissions with --group-add=keep-groups + crun doesn't make a difference

Steps to reproduce the issue

Using OpenSuSE Tumbleweed (SystemD) as Host, I've added my main user to the correct groups in order to use --group-add=keep-groups (run.oci.keep_original_groups=1):

# HOST
$ sudo usermod -G video,render,audio,tty -a $USER
$ exec su -l $USER # Update groups without creating a subshell
$ groups
me tty wheel video render audio

For demonstration purposes I've created a rootless container which uses the root user inside of it. The same result can be observed if you use a regular user account inside the container and also if you don't specify --group-add=keep-groups:

# HOST
$ sudo zypper in -y crun podman
$ podman run \
  --replace --name=desktop-glx \
  --privileged --user=root:root --userns=keep-id --group-add=keep-groups --cap-add=ALL \
  --env-host --ipc=host --pid=host --ulimit=host --network=host --tz=local \
  --mount=type=devpts,destination=/dev/pts \
  --device /dev/dri \
  --device /dev/tty0 \
  -v=/dev/tty0:/dev/tty0:rslave \
  -v=$HOME:$HOME:rslave \
  -v=/run/dbus:/run/dbus:rslave \
  -v=/run/user:/run/user:rslave \
  -v=/run/systemd:/run/systemd:rslave \
  -v=/run/udev:/run/udev:rslave \
  -it \
  registry.opensuse.org/opensuse/tumbleweed:latest

# CONTAINER
$ zypper refresh && zypper dup -y
$ zypper in -y -t pattern sway
$ zypper in -y seatd
$ seatd -u $USER & sway -d
00:05:28.816 [common/terminal.c:149] Could not open target tty: Permission denied
00:05:28.816 [seatd/seat.c:60] Could not open tty0 to update VT: Permission denied
00:05:28.816 [common/terminal.c:149] Could not open target tty: Permission denied
00:05:28.816 [seatd/seat.c:71] Could not open terminal for VT 0: Permission denied
00:05:28.816 [seatd/seat.c:449] Could not open VT for client
00:05:28.816 [common/terminal.c:149] Could not open target tty: Permission denied
00:05:28.816 [seatd/seat.c:85] Could not open terminal to clean up VT 0: Permission denied
$ ls -ld /dev/tty0
crw--w---- 1 65534 65534 4, 0 Jun 10 09:05 /dev/tty0

I've verified that --group-add=keep-groups works correctly with crun:

# HOST
$ whoami
me
$ mkdir ~/test; chmod 0770 ~/test; sudo chown root:video ~/test
$ ls -ld ~/test
drwxrwx--- 1 root video 0 Jun 10 11:40 /home/me/test

$ ls -ld /dev/tty0
crw--w---- 1 root tty 4, 0 Jun 10 16:01 /dev/tty0

$ podman start desktop-glx
$ podman exec -it desktop-glx /bin/bash

# CONTAINER
$ touch /home/me/test/file # WORKS
$ ls -ld /home/me/test
drwxrwx--- 1 65534 65534 22 Jun 10 11:45 /home/me/test

Describe the results you received

Various scenarios for /dev/tty0

• Doesn't exist when passed as a device
• Exists when mounted as a volume but it's inaccessible due to permission issues which can't be worked around with --group-add=keep-groups

Describe the results you expected

Ability to access /dev/tty0

podman info output

host:
  arch: amd64
  buildahVersion: 1.30.0
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.7-1.3.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: unknown'
  cpuUtilization:
    idlePercent: 99.92
    systemPercent: 0.03
    userPercent: 0.05
  cpus: 32
  databaseBackend: boltdb
  distribution:
    distribution: '"opensuse-tumbleweed"'
    version: "20230605"
  eventLogger: journald
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.3.4-1-default
  linkmode: dynamic
  logDriver: journald
  memFree: 63150395392
  memTotal: 67310841856
  networkBackend: cni
  ociRuntime:
    name: crun
    package: crun-1.8.3-1.1.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.3
      commit: 59f2beb7efb0d35611d5818fd0311883676f6f7e
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-1.2.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: unknown
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 5
      libseccomp: 2.5.4
  swapFree: 0
  swapTotal: 0
  uptime: 4h 48m 48.00s (Approximately 0.17 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.opensuse.org
  - registry.suse.com
  - docker.io
store:
  configFile: /home/me/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/me/.local/share/containers/storage
  graphRootAllocated: 1999860994048
  graphRootUsed: 4332466176
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 2
  runRoot: /tmp/containers-user-1000/containers
  transientStore: false
  volumePath: /home/me/.local/share/containers/storage/volumes
version:
  APIVersion: 4.5.1
  Built: 1685318400
  BuiltTime: Mon May 29 02:00:00 2023
  GitCommit: ""
  GoVersion: go1.20.4
  Os: linux
  OsArch: linux/amd64
  Version: 4.5.1

### Podman in a container

No

### Privileged Or Rootless

Rootless

### Upstream Latest Release

Yes

### Additional environment details

Tested with
- Clean installation of `OpenSuSE Tumbleweed` and `OpenSuSE MicroOS` on bare metal, both uses SystemD.
- No SELinux
- No Apparmor

### Additional information

_No response_

[GrabbenD]

I've also tried removing --privileged and then adding --systemd=false to no avail.
Feels like I'm missing something?

---

For reference:

• Sway works in Docker with almost identical commands from above (after removing incompatible parameters).
• Same goes for rootful Podman containers, I was able to launch Sway just fine:

  sudo podman run \
    --name=desktop-glx \
    --user=root:root --cap-add=ALL --security-opt label=disable \
    --ipc=host --pid=host --network=host \
    --device /dev/dri \
    --mount type=bind,src=/dev/tty0,target=/dev/tty0 \
    -v=/home/me:/home/me:rslave \
    -v=/run/dbus:/run/dbus:rslave \
    -v=/run/user:/run/user:rslave \
    -v=/run/systemd:/run/systemd:rslave \
    -v=/run/udev:/run/udev:rslave \
    -it \
    registry.opensuse.org/opensuse/tumbleweed:latest

[Luap99]

If this is a regression please specify the last working version (and possibly even git bisect if you are able to do that).

[rhatdan]

Does sway work with rootful Podman?

@giuseppe PTAL

[giuseppe]

the same commands you've provided work well for me on Fedora 38.

Could it be something else blocking the access? AppArmor?

[GrabbenD]

Thanks for the assistance guys

If this is a regression please specify the last working version (and possibly even git bisect if you are able to do that).

@Luap99 I'm not sure if this ever worked in a rootless container. I noted it as a regression since Docker can access /dev/tty* with --privileged while Podman can't if you run Podman from a regular user account.

#15878

Does sway work with rootful Podman?

@rhatdan Yes, if I run Podman with sudo it works as expected, I don't even need --privileged

the same commands you've provided work well for me on Fedora 38.

@giuseppe Are you actually able to start Sway?

I can give Fedora a shot, did you use Fedora Server or Fedora Workstation?

Could it be something else blocking the access? AppArmor?

With OpenSuSE MicroOS and OpenSuSE Tumbleweed I've explicitly disabled SELinux and Apparmor during the installation:

# SELinux
$ sudo sestatus
bash: sestatus: command not found

$ sudo getenforce
bash: getenforce: command not found

# Apparmor
$ sudo less /sys/kernel/security/apparmor/profiles
/sys/kernel/security/apparmor/profiles: No such file or directory

$ sudo apparmor_status
apparmor module is loaded.
apparmor filesystem is not mounted.

$ sudo systemctl status apparmor
○ apparmor.service - Load AppArmor profiles
     Loaded: loaded (/usr/lib/systemd/system/apparmor.service; enabled; preset: enabled)
     Active: inactive (dead)
  Condition: start condition failed at Mon 2023-06-12 12:02:22 CEST; 8min ago

[Luap99]

If this is a regression please specify the last working version (and possibly even git bisect if you are able to do that).

@Luap99 I'm not sure if this ever worked in a rootless container. I noted it as a regression since Docker can access /dev/tty* with --privileged while Podman can't if you run Podman from a regular user account.

Then it is not a regression for us, a regression means it worked in a previous version and then stopped working. Also docker runs as root by default unless you explicitly setup rootless docker.

[giuseppe]

the same commands you've provided work well for me on Fedora 38.

@giuseppe Are you actually able to start Sway?

yes, it starts. I am using Fedora Workstation 38

[GrabbenD]

yes, it starts. I am using Fedora Workstation 38

Great find @giuseppe!
I gave it a shot on bare metal:

Fedora Workstation 38

This uses GNOME 44 out of the box.
I was able to run Sway in a rootless Podman but only from the actual desktop environment/Gnome.
Seatd doesn't even have to be installed in the container. I believe this runs Sway in a nested window under Gnome's session manager in that case?

However, if I instead use tty2 I get:

[wlr] [backend/wayland/backend.c:555] Could not connect to remote display: Connection refused
[sway/server.c:73] Unable to create backend

For clarity:

$ ls -ld /dev/tty0
crw--w----. 1 root tty     4,  0 Jun 12 14:55 /dev/tty0

Fedora Server 38

This gives the same error as OpenSuSE Tumbleweed and OpenSuSE MicroOS: Could not open tty0 to update VT: Permission denied

$ ls -ld /dev/tty0
crw--w----. 1 root tty 4, 0 Jun 12 13:51 /dev/tty0

---

In this case, how can a rootless container be given permission to handle seats/tty0?

[giuseppe]

could be wayland to block the access?

[GrabbenD]

could be wayland to block the access?

Hmm @giuseppe, I can't find any indication why Wayland would be the cause. Outside of Podman Sway works fine on NixOS (which is a distro that runs installed packages in user space).

However, in Podman there's a check for tty0 which prohibits rootless containers from taking control if I understand the code correctly? #15878

podman/pkg/util/utils_linux.go

Lines 114 to 124 in 8d0fcd4

	/* The following devices should not be mounted in rootless containers:
	*
	* /dev/ptmx: The host-provided /dev/ptmx should not be shared to
	* the rootless containers for security reasons, and
	* the container runtime will create it for us
	* anyway (ln -s /dev/pts/ptmx /dev/ptmx);
	* /dev/tty and
	* /dev/tty[0-9]+: Prevent the container from taking over the host's
	* virtual consoles, even when not in systemd mode
	* for backwards compatibility.
	*/

[GrabbenD]

Just to clarify, Sway in Podman with:

• rootfull-privileged & rootfull-unprivileged works.
• rootless-privileged & rootless-unprivileged doesn't work.

[giuseppe]

the device should still be present in the container if you explicitly bind mount it through -v

[GrabbenD]

the device should still be present in the container if you explicitly bind mount it through -v

That makes sense @giuseppe but it's not accessible with either 🙁

• -v=/dev/tty0:/dev/tty0
• -v=/dev/tty0:/dev/tty0:rslave
• --mount type=bind,src=/dev/tty0,target=/dev/tty0

Permission denied is a message from the system:
https://github.com/kennylevinsen/seatd/blob/master/seatd/seat.c#L58-L66

[giuseppe]

I've converted the issue to a discussion since it doesn't seem to be a Podman issue but rather a configuration problem

[yellowhat]

Hi,
did someone managed to run sway in rootless mode?

It works with:

sudo podman run \
    --name htpc \
    --rm \
    --interactive \
    --tty \
    --privileged \
    --cap-add ALL \
    --device /dev/dri \
    --security-opt label=disable \
    --volume /dev/tty0:/dev/tty0 \
    --volume /run/dbus:/run/dbus:rslave \
    --volume /run/user:/run/user:rslave \
    --volume /run/systemd:/run/systemd:rslave \
    --volume /run/udev:/run/udev:rslave \
    --env XDG_RUNTIME_DIR=/run/user/1000 \
    registry.fedoraproject.org/fedora

dnf install -y seatd sway
seatd -u root &
sway

But a more fine-grained mode would be appreciated.

Thanks

[GrabbenD]

Here's my current approach:

I'm personally running a minimal host which has the latest kernel, drivers, firmware and microcode.
Each container is fully contained meaning it has its own DBUS communication, GPU drivers, Pipewire sound and Wayland Compositor/xorg-server:

Updates

• Avoid rslave, especially with tty devices as this might occasionally cause a hard lockup in host

build_args=(
  # Reuse host cache
  --volume /var/cache/pacman/pkg:/var/cache/pacman/pkg
  # Config files
  --volume /home:/home
)

run_args=(
  --name='game'

  # Support /sbin/init
  --systemd=always
  --volume /sys/fs/cgroup:/sys/fs/cgroup:ro

  # Detect file changes in Containerfile
  --rm='true'
  --replace='true'

  # Bridge network
  --network='host'

  # GPU access
  --privileged='true'
  --ipc='host'
  --device /dev/dri:/dev/dri:rwm
  --device /dev/fb0:/dev/fb0:rwm
  --volume /dev/shm:/dev/shm

  # Seats
  --volume /dev/tty0:/dev/tty0
  --volume /dev/tty1:/dev/tty1

  # SELinux
  --security-opt seccomp=unconfined
  --security-opt unmask=/sys/dev

  # Peripherals
  --volume /dev/input:/dev/input
  --volume /run/udev:/run/udev
  --volume /dev/snd:/dev/snd
)

# CTRL+P+Q to quit
clear && \
  sudo podman build \
    "${build_args[@]}" \
    -t local/game:latest \
    -f Containerfile.game \
    "$@" \
    ./ && \
  sudo podman run \
    "${run_args[@]}" \
    "${build_args[@]}" \
    -d \
    local/game:latest && \
  systemctl stop \
    [email protected] && \
  sudo podman exec \
    -it game \
    /bin/login

I'm working on making this into a Github repository, it's still a work in progress though!
Please let me know if you guys have any tips/recommendations on how to improve this :)

[yellowhat]

Please post here the link to your future repository.

Thanks for sharing this information

[yellowhat]

Could you share the "just working" configuration you were using in rootless mode?

[GrabbenD]

Thanks for sharing this information

No worries!

By the way, be careful when using --cap-add ALL @yellowhat . I noticed that (regular) user accounts inside the container can perform rootful operations when this argument is used when building the image with Containerfile

[GrabbenD]

Could you share the "just working" configuration you were using in rootless mode?

I never managed to access tty0 in a embedded rootless container, not even --cap-add SYS_TTY_CONFIG was enough. However, I think it might be doable if you:

1. Install seatd/elogind outside of the container.
2. Bind the seat socket to the container.
3. Add the user account (which runs podman) to the relevant groups that can access the seat socket.
4. Install crun and add --group-add=keep-groups (there's also --userns=keep-id but I don't think it's needed with this approach) argument(s) to podman run.
5. Make sure to create a new session (meaning logout) after adding the relevant groups to the user as otherwise you might run into permission issues (or use su - $USER)

I think it's doable since it's possible to run a WM in nested mode with rootless podman under Gnome with Fedora Workstation #18870 (comment) and the parameters that I used are here #18870 (comment)

However, considering performance is worse in rootless containers and the fact that I want to run as little as possible on my host machine - I don't have much interest to play around with rootless containers for now 🙂

My recommendation would be to make the host immutable (e.g. with OSTree or Elemental Toolkit) and keep persistent data in seperate partitions which you access through volumes via rootful containers