Dev Containers: improve Podman compatibility (low-hanging fruit)

[sicherha]

This is a proposed change to the Dev Containers extension in the wider context of #6759. It aims to improve the out-of-the-box experience with Podman without breaking Docker compatibility.

Proposed change

1. In the Dev Containers extension, set the environment variable PODMAN_USERNS=keep-id before launching the container executable (podman/docker).
2. In the corresponding documentation, remove this block as it is no longer needed:

"runArgs": [
  "--userns=keep-id"
],

Why?

This allows the owner of a Git repository to provide a workspace configuration that works out-of-the-box for users of both Podman and Docker.

Background

Podman is mostly drop-in-compatible with Docker, but there are a small number of differences stemming from the fact that Podman is a rootless container engine. Notably, the host-to-container user mapping works differently: in order to get the same behaviour as Docker, users have to pass the --userns=keep-id option to Podman as described in the Podman section of the documentation.

Unfortunately, Docker does not accept keep-id as a valid value of the --userns= option. So at the moment the owner of a Git repository has two options:
a) Provide a workspace configuration that works for Docker, and tell Podman users to manually adjust their user-wide configuration.
b) Provide a workspace configuration that works for Podman but breaks Docker.

This dilemma could be circumvented if the Dev Containers extension set the environment variable PODMAN_USERNS=keep-id:

• Docker simply ignores it.
• Podman would now do the correct thing - and if needed, users could still override its value via the --userns command-line parameter.

Podman's selection logic is documented in detail: default value < containers.conf < PODMAN_USERNS < --pod < --userns.

User story

As the owner of a Git repository,
I want to provide a .devcontainer and .vscode workspace configuration that works out-of-the-box for users of both Docker and Podman,
so that neither have to make any modifications to their local development environment.

[chrmarti]

Would disabling the UID update also work? (Try "updateRemoteUserUID": false in the devcontainer.json.) I was under the impression that this causes issues because it is not aware of the UID mapping from rootless mode.

[sicherha]

That's a good point. There is definitely chatter on the internet about "updateRemoteUserUID": false breaking in conjunction with --userns=keep-id. I'll try to reproduce this with a minimal devcontainer.json.

Obviously, this would complicate the matter a bit. I wonder if it would be feasible to conditionally set PODMAN_USERNS to either keep-id or host depending on the value of updateRemoteUserUID.

[chrmarti]

I was wondering if disabling the UID update would be an alternative to keep-id because the UID update isn't needed in the rootless setup, but causes the issue it was meant to solve: Making the UID in the container match the one outside.

[sicherha]

I have tested on Fedora 41 with Visual Studio Code 1.96.0, Podman 5.3.1 and the following minimal devcontainer.json:

{
  "name": "C++",
  "image": "mcr.microsoft.com/devcontainers/cpp:1-debian-12",
  "containerEnv": {
    "HOME": "/home/vscode"
  },
  "securityOpt": [
    "label=disable"
  ],
  "updateRemoteUserUID": false
}

I varied updateRemoteUserUID between false and true, and launched code from my terminal with an unmodified environment and with PODMAN_USERNS=keep-id. I rebuilt the devcontainer in every run.

The condition for success is the execution of the following shell command in the workspace directory inside the devcontainer:

touch foo && rm foo

On success this command prints nothing; on failure it produces an error message:

touch: cannot touch 'foo': Permission denied

Here are the results:

PODMAN_USERNS=	"updateRemoteUserUID":	Success
(unset)	false	❌
(unset)	true	❌
keep-id	false	✅
keep-id	true	✅

Conclusions:

• "updateRemoteUserUID": false does not have the desired effect.
• At least with this minimal example, I did not come across a conflict between PODMAN_USERNS=keep-id and "updateRemoteUserUID": false .

[sicherha]

In all four cases, the UID and GID are 1000:1000 inside the container.
On the host system, my UID and GID are 1000:1000 as well.

[chrmarti]

Sounds good. When the dev container user is root, would it make sense to omit the env variable? IIRC 0:0 in the container is mapped to the local user outside. So existing devcontainer.json used with user root would break with keep-id?

[sicherha]

Yes, setting the environment variable based on the remote user sounds like a good idea.

Unfortunately, the overall situation is somewhat tricky. I tested with a new devcontainer.json that deliberately uses a different base image (fedora:latest) which does not set a USER by default:

{
  "name": "C++",
  "image": "fedora:latest",
  "securityOpt": [
    "label=disable"
  ]
}

With this image, the situation looks as follows.

PODMAN_USERNS=	"remoteUser":	Result
(unset)	(unset)	✅
(unset)	"root"	✅
keep-id	(unset)	❌
keep-id	"root"	✅

This time, ❌ means that the container already fails during startup:

[1839 ms] 
[1839 ms] Start: Run in container:  (command -v getent >/dev/null 2>&1 && getent passwd '1000' || grep -E '^1000|^[^:]*:[^:]*:1000:' /etc/passwd || true)
[1841 ms] Start: Run in container: command -v git >/dev/null 2>&1 && ROOT_FOLDER="$(git -C '/workspaces/devcontainer' rev-parse --show-toplevel)" && test "$(stat -c %u "$ROOT_FOLDER")" != "$(id -u)" && echo -n "$ROOT_FOLDER"
[1842 ms] 
[1842 ms] 
[1842 ms] Exit code 1
[1842 ms] Start: Updating configuration state
[1845 ms] Start: Setup shutdown monitor
[1846 ms] Forking shutdown monitor: /home/christoph/.vscode/extensions/ms-vscode-remote.remote-containers-0.394.0/dist/shutdown/shutdownMonitorProcess /run/user/1000/vscode-remote-containers-03491454-063a-4d86-baa4-3b69ecdbfe9d.sock singleContainer Debug /home/christoph/.config/Code/logs/20241214T104443/window1/exthost/ms-vscode-remote.remote-containers 1734169491758
[1854 ms] Start: Run in container: test -d '/.vscode-server'
[1855 ms] 
[1855 ms] 
[1855 ms] Exit code 1
[1855 ms] Start: Run in container: test -d '/.vscode-remote'
[1856 ms] 
[1856 ms] 
[1856 ms] Exit code 1
[1856 ms] Start: Run in container: test ! -f '/.vscode-server/data/Machine/.writeMachineSettingsMarker' && set -o noclobber && mkdir -p '/.vscode-server/data/Machine' && { > '/.vscode-server/data/Machine/.writeMachineSettingsMarker' ; } 2> /dev/null
[1858 ms] 
[1858 ms] mkdir: cannot create directory '/.vscode-server': Permission denied
[1858 ms] Exit code 1
[1858 ms] Start: Run in container: cat /.vscode-server/data/Machine/settings.json
[1859 ms] 
[1859 ms] cat: /.vscode-server/data/Machine/settings.json: No such file or directory
[1859 ms] Exit code 1
[1859 ms] Start: Run in container: test -d '/.vscode-server/bin/138f619c86f1199955d53b4166bef66ef252935c'
[1859 ms] 
[1860 ms] 
[1860 ms] Exit code 1
[1860 ms] Start: Run in container: test -d '/vscode/vscode-server/bin/linux-x64/138f619c86f1199955d53b4166bef66ef252935c'
[1861 ms] 
[1861 ms] 
[1861 ms] Start: Run in container: mkdir -p '/.vscode-server/bin' && ln -snf '/vscode/vscode-server/bin/linux-x64/138f619c86f1199955d53b4166bef66ef252935c' '/.vscode-server/bin/138f619c86f1199955d53b4166bef66ef252935c'
[1862 ms] 
[1862 ms] mkdir: cannot create directory '/.vscode-server': Permission denied
[1863 ms] Exit code 1
[1865 ms] Command in container failed: mkdir -p '/.vscode-server/bin' && ln -snf '/vscode/vscode-server/bin/linux-x64/138f619c86f1199955d53b4166bef66ef252935c' '/.vscode-server/bin/138f619c86f1199955d53b4166bef66ef252935c'
[1865 ms] mkdir: cannot create directory '/.vscode-server': Permission denied
[1865 ms] Exit code 1

[sicherha]

When I repeat the experiment with the mcr.microsoft.com/devcontainers/cpp:1-debian-12 base image, then all combinations of PODMAN_USERNS= and "remoteUser": work fine.

This brings me into speculative territory, but apparently Podman's keep-id user-ID mapping works fine when the container has a specified USER but breaks when it doesn't. 🤔

[chrmarti]

getent passwd '1000' in the log above indicates that the UID was somehow 1000. 🤔 I guess it didn't find that user and wrongly continued without home folder (hence the attempt to work in /).

[sicherha]

I think this makes sense: the host user gets mapped into the container but the container file system lacks the corresponding home directory. Presumably, that's precisely why the documentation recommends explicitly setting the HOME environment variable.

I want to tread carefully here because I wish to avoid breaking existing configurations. But unless I'm still missing something, it looks like we can indeed set PODMAN_USERNS=keep-id unconditionally without stepping on anyone's toes.

• Docker users won't notice because Docker ignores this environment variable.
• Existing Podman configurations already explicitly set HOME and pass --userns=<something>; the latter is given higher precedence than PODMAN_USERNS.
• New Podman configurations may still have to set HOME but no longer need to pass --userns=keep-id.