Toolbox can't start if user has any group-only permissions for device nodes in folders

[mtalexan]

Describe the bug
If a user is a member of any group that grants access to device nodes in folders that have 750 permissions, all toolboxes can no longer be entered.

For example when VirtualBox is installed, the /dev/vboxusb/* device nodes are created with 750 permissions on both the folder and the files, and they're owned by the vboxusers group (this behavior is consistent with standards for device node permissions, and is like the /dev/ttyUSB* nodes on Fedora systems). Toolbox maps the entire /dev into the container, which uses the user permissions from outside the container to map all visible files into the container. The crun mapping of sub-folders uses the permissions inside the container namespaces to create a matching folder however. This causes an error because the folder has permissions granted only via an unmapped group.

TL;DR
Blanket mapping of /dev into the container without also mapping all the user's groups breaks in very common conditions.

Steps how to reproduce the behaviour

1. Create device node in a folder with 750 permissions, owned by a group the user is part of but not by the user directly.
   1. sudo mkdir -Z -m 750 /dev/testFolder
   2. sudo mkdnod -Z -m 750 /dev/testFolder/testNode b 7 30 (loop device 30)
   3. sudo chown -R root:dialout /dev/testFolder (dialout used for convenience rather than
      creating a new group)
   4. groups, and confirm user is part of dialout group
   5. ls /dev/testFolder, confirm the user can see the files using the dialout group permissions
2. toolbox create
3. toolbox enter

Expected behaviour
Toolbox is entered successfully.

Actual behaviour
Error, unable to enter toolbox.

Error: unable to start container XXXX: crun: mkdir `/dev/testFolder`: Permission denied: OCI permission denied

Screenshots

N/A

Output of toolbox --version (v0.0.90+)

toolbox version 0.0.99.4

Toolbox package info (rpm -q toolbox)

toolbox version 0.0.99.4

Output of podman version

Client:       Podman Engine
Version:      4.6.0
API Version:  4.6.0
Go Version:   go1.20.6
Built:        Fri Jul 21 08:23:26 2023
OS/Arch:      linux/amd64

Podman package info (rpm -q podman)

podman-4.6.0-1.fc38.x86_64

Info about your OS
Fedora Kinoite 38

Additional context

The issue becomes immediately very obvious if you install VirtualBox. This creates /dev/vboxusb/* USB proxy device nodes, where the /dev/vboxusb and all contents are owned by the vboxusers group. If the user isn't part of the vboxusers group, toolbox works because the folder doesn't appear to be present under the user permissions. But if the user is a member of the group (as can be expected), the folder can be listed from outside the container, but the permissions for interacting with it inside the container namespace aren't available to crun.

---

The only viable solutions are:

1. Stop mounting /dev as a whole into the container and instead iterate over all device nodes found recursively from outside the container, filtered down to the ones owned directly by the user or with 755 permissions.
2. Use --group-add keep-groups and have toolbox init-container inside the container create the necessary matching groups/gids that are added to the user in the container.
3. Allow a an option to toolbox create, --keep-group={group_name}, that can list multiple groups that should be mapped from the host user into the container. Allow a default list to be specified in the toolbox.conf.

The second option is obviously much better than 1 since it solves a number of other complaints (e.g. /dev/ttyUSB* access), but has the negative effect that the list of group names to gids from the user outside the container must be replicated inside the container, and traceability of what those were must be kept inside the container so if a user is removed from a group they will also be removed from the group inside the container.

Option 3 is probably the best though, since it allows the current existing behavior as-is, but also allows users to specifically pass thru certain group permissions. It would suffer from the same complexities as option 2 in terms of implementation though, and would add the need to parse extra command-line options and config file fields.

[mtalexan]

This is a more generic description of the root problem underlying the very generically named #1297.

If the third solution were implemented, it would solve the following:

• toolbox enter failed with "Error: failed to start container fedora-toolbox-37" #1297
• Cannot change group membership of toolbox user #632
• Cannot use the host's Docker socket #430
• Let toolbox access dialout group, to be able to use usb devices at /dev/tty* #432
• Allow to customize the groups the user will be part of when creating a toolbox #785

[debarshiray]

Thank you for your investigation into this, @mtalexan !

#430 has some notes from the past.

[debarshiray]

One quick observation. Our minimum Podman requirement is 1.6.4, which is too old for --group-add keep-groups. Instead, we can use --annotation run.oci.keep_original_groups=1.

[AdamWill]

@debarshiray as using VirtualBox is a pretty common thing to do, is there any chance we could prioritize this a bit?

[debarshiray]

@debarshiray as using VirtualBox is a pretty common thing to do, is there any chance we could prioritize this a bit?

The status quo is that there doesn't seem to be a good way to do this by default for all Toolbx containers.

Using podman create --annotation run.oci.keep_original_groups=1 or podman create --group-add keep-groups retains the groups from the host, but overrides the user's membership in the wheel group inside the container. That means that you can access the Docker socket, but you can't use sudo(8) anymore.

We can dodge the sudo(8) issue by not basing it on group membership in the container, but we might not be able to do that for other things that really need that in the container.

The reason I want to solve this in the default case is that the configuration of OCI containers is immutable. If we provided an option, the user would have to create a new container with the option to make it work.

So, if we can't fix it by default, then I think we can add a command to easily fork the contents of an existing container and change the configuration. However, that can still leave the users with a container that's broken in other ways.

[nixuser58]

If I am not mistaken we're looking at this problem from the perspective of needing to provide access to these special device nodes from inside the container, but is that really necessary?

In the VirtualBox use case it is most definitely not necessary, and I would suggest in most other cases it's not.

Maybe the solution is for the container to ignore nodes it doesn't have access to rather than barf on a fatal error. That is, just remove them from the list. I think this would solve most real world use cases while not being "academically" wrong in any way.

I do understand there could be unusual cases where you may need to access problematic device nodes, but wouldn't it be possible to add yourself to the group once inside the container?

The problem here is with problematic device nodes you cannot start the container so you cannot get inside it to change anything.

[mtalexan]

If I am not mistaken we're looking at this problem from the perspective of needing to provide access to these special device nodes from inside the container, but is that really necessary?

In the VirtualBox use case it is most definitely not necessary, and I would suggest in most other cases it's not.

Maybe the solution is for the container to ignore nodes it doesn't have access to rather than barf on a fatal error. That is, just remove them from the list. I think this would solve most real world use cases while not being "academically" wrong in any way.

I do understand there could be unusual cases where you may need to access problematic device nodes, but wouldn't it be possible to add yourself to the group once inside the container?

The problem here is with problematic device nodes you cannot start the container so you cannot get inside it to change anything.

While the problem is most easily encountered and demonstrated with the VirtualBox device nodes, it's not unique to it. It's any folder under /dev that has access granted only thru group membership.

[nixuser58]

I totally understand that. My point is when users have access so such nodes in /dev through group membership, it is because of some special use case (VirtualBox being simply one of them) but in those cases I would argue that it is VERY unlikely that you would need access to those nodes from inside a container. I acknowledged in my comment above that it was possible you would, but only in rare cases.

To me ignoring the nodes is a better compromise than failing to start a container.

[FlorianLudwig]

@nixuser58 - there are real world use cases for having access to device nodes. For example:

On fedora there is a group called dialout which makes tty/serial devices accessible to the user. Having containerized development tools or running on something like silverblue it becomes essential to have access to devices with only group-access.

[dylangerdaly]

Hitting this, no idea how this hasn't been fixed yet.

[SohamG]

Just hit this, wanted to mention that I don't hit this when testing with distrobox. I'm new to the dev/user containers landscape so can't analyze further. Lemme know if I can help with logs

[Knogle]

Still an issue.

[poiNt3D]

This is laughable.
They say you can't break an immutable system.
But some things are broken out of the box.

[benjamimgois]

I also just hit this problem. I need to access my network switch with a serial-usb converter and just can't at the moment. Tried adding my user to the dialout group but it also doesn't work.

[mtalexan]

I also just hit this problem. I need to access my network switch with a serial-usb converter and just can't at the moment. Tried adding my user to the dialout group but it also doesn't work.

If your toolbox starts at all, it's not the same problem. The problem here is that some things like VirtualBox create a folder under /dev and put all their device nodes in it. To get access to the folder contents you need to be a member of the dedicated group. Toolbox attempts to mount all of /dev into the container, but doesn't preserve group permissions when doing so. This causes the mounting of /dev to always fail, and the toolbox start to error out.
The only work around is to not use Toolbox (maybe look at Distrobox instead that doesn't have this problem).

If you have individual device nodes that are missing permissions, the Toolbox should still start, you just won't be able to use the device in the toolbox. That's a separate issue.

[Rivers47]

As a workaround (I can't believe it took me 2 minutes to think of this) for the virtual box case, I created a new user without the problematic group

[mtalexan]

I think this might be solved now, and it was actually a podman issue.
There have been a number of bugs and discussions within podman over the last year about permissions issues related to group-membership granting access so that might not be surprising.

Using toolbox version 0.0.99.3, with podman 4.9.3, I'm no longer experiencing this issue even though I have the /dev/vboxusb/ folder with device nodes in it that I only have access to via the vboxusers group.

So TL;DR, upgrade your podman version (seems to be the solution).

[Yiannis128]

I think this might be solved now, and it was actually a podman issue. There have been a number of bugs and discussions within podman over the last year about permissions issues related to group-membership granting access so that might not be surprising.

Using toolbox version 0.0.99.3, with podman 4.9.3, I'm no longer experiencing this issue even though I have the /dev/vboxusb/ folder with device nodes in it that I only have access to via the vboxusers group.

So TL;DR, upgrade your podman version (seems to be the solution).

Still not working:

❯ toolbox --version
toolbox version 0.1.1
❯ podman -v
podman version 5.3.1

This needs to get fixed as VirtualBox is a very common software... It's been 1.5 years already.

[nixuser58]

Confirming the above, please note this is NOT fixed in Fedora 41 running:

toolbox-0.1.1-1.fc41.x86_64
podman-5.3.1-1.fc41.x86_64

Removing myself from vboxusers allowed toolbox to work, then after re-adding myself (and rebooting to make sure, not just logging out and in again) toolbox would not work.