Error when pulling

[gatoniel]

With my standard user everything works out fine. I created a testuser with UID=1004. When I try to use the PodmanSpawner with this user, I get this error:

Trying to pull docker.io/jupyterhub/singleuser...
Getting image source signatures
Copying blob
Writing manifest to image destination
Storing signatures
time="2020-03-29T13:59:11+02:00" level=error msg="Error while applying layer: ApplyLayer exit status 1 stdout: stderr: there might not be enough IDs available in the namespace (requested 0:42 for /etc/gshadow): lchown /etc/gshadow: invalid argument"
ApplyLayer exit status 1 stdout: stderr: there might not be enough IDs available in the namespace (requested 0:42 for /etc/gshadow): lchown /etc/gshadow: invalid argument
Error: unable to pull docker.io/jupyterhub/singleuser: unable to pull image: Error committing the finished image: error adding layer with blob "sha256:423ae2b273f4c17ceee9e8482fa8d071d90c7d052ae208e1fe4963fceb3d6954": ApplyLayer exit status 1 stdout: stderr: there might not be enough IDs available in the namespace (requested 0:42 for /etc/gshadow): lchown /etc/gshadow: invalid argument

From this comment in an issue (containers/podman#2542 (comment)) I get, that there might be some difference in how docker and podman run commands regarding the usage of user namespaces. Maybe we need to make also different images for podman?

This is the line for testuser in /etc/subuid:

testuser:362144:65536

Also there is no file under /run/user/1004/libpod/pause.pid

[gatoniel]

I solved this problem by reading this issue. The solution was to run podman system migrate for new users. The environment variable $XDG_RUNTIME_DIR seems to confuse the command, if set by a former user due to suing into the new user.

I will need to exec this command in the correct environment context prior to the podman run image call. But how? Actually this is not the job of the spawner, but more of the system or podman management... Maybe the useradd command could be hooked with system migrate?

EDIT: I created a new user and logged in to this user via ssh. Everything worked fine. So we need to imitate exactly the behavior sshd does when a new user logs in. We especially need to run pam_open_session, which is already implemented in the PAMAuthenticator. But what are the CAPABILITIES, a non-root process needs to do this. So there are two possibilities:

1. via JupyterHub Authenticator create the correct user context via pam_open_session
2. before each podman run we execute podman system migrate with the correct environment variables, viz. $XDG_RUNTIME_DIR not set...

[vwbusguy]

@gatoniel Thanks! Running podman system migrate as a regular user fixed it for me.

[gatoniel]

Should be fixed with commit efc7c28