podman on silverblue fails to create uid namespace #4047

npmccallum · 2019-09-16T21:16:25Z

/kind bug

Description

The current setup on Fedora Silverblue doesn't work for me.

Steps to reproduce the issue:

First, install Fedora 31 Silverblue. Then:

$ podman run --rm -it fedora:latest
Trying to pull docker.io/library/fedora:latest...
Getting image source signatures
Copying blob 5a915a173fbc done
Copying config e9ed59d2ba done
Writing manifest to image destination
Storing signatures
  Error processing tar file(exit status 1): there might not be enough IDs available in the namespace (requested 192:192 for /run/systemd/netif): lchown /run/systemd/netif: invalid argument
Trying to pull registry.fedoraproject.org/fedora:latest...
  Get https://registry.fedoraproject.org/v2/: local error: tls: unexpected message
Trying to pull quay.io/fedora:latest...
  error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2 Final//EN\">\n<title>404 Not Found</title>\n<h1>Not Found</h1>\n<p>The requested URL was not found on the server.  If you entered the URL manually please check your spelling and try again.</p>\n"
Trying to pull registry.access.redhat.com/fedora:latest...
  name unknown: Repo not found
Trying to pull registry.centos.org/fedora:latest...
  manifest unknown: manifest unknown
Error: unable to pull fedora:latest: 5 errors occurred:
	* Error committing the finished image: error adding layer with blob "sha256:5a915a173fbc36dc8e1410afdd9de2b08f71efb226f8eb1ebcdc00a1acbced62": Error processing tar file(exit status 1): there might not be enough IDs available in the namespace (requested 192:192 for /run/systemd/netif): lchown /run/systemd/netif: invalid argument
	* Error initializing source docker://registry.fedoraproject.org/fedora:latest: pinging docker registry returned: Get https://registry.fedoraproject.org/v2/: local error: tls: unexpected message
	* Error initializing source docker://quay.io/fedora:latest: Error reading manifest latest in quay.io/fedora: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2 Final//EN\">\n<title>404 Not Found</title>\n<h1>Not Found</h1>\n<p>The requested URL was not found on the server.  If you entered the URL manually please check your spelling and try again.</p>\n"
	* Error initializing source docker://registry.access.redhat.com/fedora:latest: Error reading manifest latest in registry.access.redhat.com/fedora: name unknown: Repo not found
	* Error initializing source docker://registry.centos.org/fedora:latest: Error reading manifest latest in registry.centos.org/fedora: manifest unknown: manifest unknown

Output of podman version:

$ podman version
Version:            1.5.1-dev
RemoteAPI Version:  1
Go Version:         go1.13rc1
OS/Arch:            linux/amd64

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.13rc1
  podman version: 1.5.1-dev
host:
  BuildahVersion: 1.10.1
  Conmon:
    package: podman-1.5.1-2.17.dev.gitce64c14.fc31.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.0, commit: 118fcdfca36d706f766bad2663b11bd2c41bf2e7'
  Distribution:
    distribution: fedora
    version: "31"
  MemFree: 10766471168
  MemTotal: 16370573312
  OCIRuntime:
    package: crun-0.8-1.fc31.x86_64
    path: /usr/bin/crun
    version: |-
      crun 0.8
      spec: 1.0.0
      +SYSTEMD +SELINUX +CAP +SECCOMP +EBPF +YAJL
  SwapFree: 8266969088
  SwapTotal: 8266969088
  arch: amd64
  cpus: 4
  eventlogger: journald
  hostname: localhost.localdomain
  kernel: 5.3.0-0.rc6.git0.1.fc31.x86_64
  os: linux
  rootless: true
  uptime: 43m 0.32s
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ConfigFile: /home/nmccallu/.config/containers/storage.conf
  ContainerStore:
    number: 0
  GraphDriverName: overlay
  GraphOptions:
  - overlay.mount_program=/usr/bin/fuse-overlayfs
  GraphRoot: /var/home/nmccallu/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 0
  RunRoot: /run/user/16827
  VolumePath: /var/home/nmccallu/.local/share/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

podman-1.5.1-2.17.dev.gitce64c14.fc31.x86_64

Additional environment details (AWS, VirtualBox, physical, etc.):

$ cat /etc/sub*id
nmccallu:10000:65536
nmccallu:10000:65536

$ podman unshare cat /proc/self/uid_map
         0      16827          1

$ getenforce 
Permissive

The text was updated successfully, but these errors were encountered:

mheon · 2019-09-16T21:17:49Z

@giuseppe PTAL

giuseppe · 2019-09-17T07:07:23Z

could you try?

podman system migrate && podman unshare cat /proc/self/uid_map

If you still see the same error, what is the output for getcap /usr/bin/newuidmap /usr/bin/newgidmap?

debarshiray · 2019-09-17T19:29:12Z

I see that you also got:

	* Error initializing source docker://registry.fedoraproject.org/fedora:latest: pinging docker registry returned: Get https://registry.fedoraproject.org/v2/: local error: tls: unexpected message

While it's not the main point of this issue, you can set the GODEBUG environment variable to tls13=0 to work around it until the Fedora registry is fixed to work with Go's TLS 1.3 implementation.

npmccallum · 2019-09-17T19:33:19Z

@giuseppe

$ podman system migrate && podman unshare cat /proc/self/uid_map
WARN[0000] using rootless single mapping into the namespace. This might break some images. Check /etc/subuid and /etc/subgid for adding subids 
         0      16827          1

$ podman run --rm -it fedora:latest
Trying to pull docker.io/library/fedora:latest...
Getting image source signatures
Copying blob 5a915a173fbc done
Copying config e9ed59d2ba done
Writing manifest to image destination
Storing signatures
  Error processing tar file(exit status 1): there might not be enough IDs available in the namespace (requested 192:192 for /run/systemd/netif): lchown /run/systemd/netif: invalid argument
Trying to pull registry.fedoraproject.org/fedora:latest...
  Get https://registry.fedoraproject.org/v2/: local error: tls: unexpected message
Trying to pull quay.io/fedora:latest...
  error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2 Final//EN\">\n<title>404 Not Found</title>\n<h1>Not Found</h1>\n<p>The requested URL was not found on the server.  If you entered the URL manually please check your spelling and try again.</p>\n"
Trying to pull registry.access.redhat.com/fedora:latest...
  name unknown: Repo not found
Trying to pull registry.centos.org/fedora:latest...
  manifest unknown: manifest unknown
Error: unable to pull fedora:latest: 5 errors occurred:
	* Error committing the finished image: error adding layer with blob "sha256:5a915a173fbc36dc8e1410afdd9de2b08f71efb226f8eb1ebcdc00a1acbced62": Error processing tar file(exit status 1): there might not be enough IDs available in the namespace (requested 192:192 for /run/systemd/netif): lchown /run/systemd/netif: invalid argument
	* Error initializing source docker://registry.fedoraproject.org/fedora:latest: pinging docker registry returned: Get https://registry.fedoraproject.org/v2/: local error: tls: unexpected message
	* Error initializing source docker://quay.io/fedora:latest: Error reading manifest latest in quay.io/fedora: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2 Final//EN\">\n<title>404 Not Found</title>\n<h1>Not Found</h1>\n<p>The requested URL was not found on the server.  If you entered the URL manually please check your spelling and try again.</p>\n"
	* Error initializing source docker://registry.access.redhat.com/fedora:latest: Error reading manifest latest in registry.access.redhat.com/fedora: name unknown: Repo not found
	* Error initializing source docker://registry.centos.org/fedora:latest: Error reading manifest latest in registry.centos.org/fedora: manifest unknown: manifest unknown

$ getcap /usr/bin/newuidmap /usr/bin/newgidmap
/usr/bin/newuidmap = cap_setuid+ep
/usr/bin/newgidmap = cap_setgid+ep

npmccallum · 2019-09-17T19:35:40Z

@debarshiray

$ GODEBUG=tls13=0 podman run --rm -it fedora:latest
Trying to pull docker.io/library/fedora:latest...
Getting image source signatures
Copying blob 5a915a173fbc done
Copying config e9ed59d2ba done
Writing manifest to image destination
Storing signatures
  Error processing tar file(exit status 1): there might not be enough IDs available in the namespace (requested 192:192 for /run/systemd/netif): lchown /run/systemd/netif: invalid argument
Trying to pull registry.fedoraproject.org/fedora:latest...
Getting image source signatures
Copying blob ed60cb1abc2e done
Copying config 02781e9f50 done
Writing manifest to image destination
Storing signatures
  Error processing tar file(exit status 1): there might not be enough IDs available in the namespace (requested 0:22 for /run/utmp): lchown /run/utmp: invalid argument
Trying to pull quay.io/fedora:latest...
  error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2 Final//EN\">\n<title>404 Not Found</title>\n<h1>Not Found</h1>\n<p>The requested URL was not found on the server.  If you entered the URL manually please check your spelling and try again.</p>\n"
Trying to pull registry.access.redhat.com/fedora:latest...
  name unknown: Repo not found
Trying to pull registry.centos.org/fedora:latest...
  manifest unknown: manifest unknown
Error: unable to pull fedora:latest: 5 errors occurred:
	* Error committing the finished image: error adding layer with blob "sha256:5a915a173fbc36dc8e1410afdd9de2b08f71efb226f8eb1ebcdc00a1acbced62": Error processing tar file(exit status 1): there might not be enough IDs available in the namespace (requested 192:192 for /run/systemd/netif): lchown /run/systemd/netif: invalid argument
	* Error committing the finished image: error adding layer with blob "sha256:ed60cb1abc2e112aa7a26e9f52cc0982551f7e959441522260e03f47437d42b9": Error processing tar file(exit status 1): there might not be enough IDs available in the namespace (requested 0:22 for /run/utmp): lchown /run/utmp: invalid argument
	* Error initializing source docker://quay.io/fedora:latest: Error reading manifest latest in quay.io/fedora: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2 Final//EN\">\n<title>404 Not Found</title>\n<h1>Not Found</h1>\n<p>The requested URL was not found on the server.  If you entered the URL manually please check your spelling and try again.</p>\n"
	* Error initializing source docker://registry.access.redhat.com/fedora:latest: Error reading manifest latest in registry.access.redhat.com/fedora: name unknown: Repo not found
	* Error initializing source docker://registry.centos.org/fedora:latest: Error reading manifest latest in registry.centos.org/fedora: manifest unknown: manifest unknown

npmccallum · 2019-09-18T15:40:59Z

The problem was that my user's UID was in the middle of the sub?id range. After fixing this, another problem was revealed. I have submitted it here: https://bugzilla.redhat.com/show_bug.cgi?id=1753328

debarshiray · 2019-09-18T16:37:42Z

@debarshiray

$ GODEBUG=tls13=0 podman run --rm -it fedora:latest
Trying to pull docker.io/library/fedora:latest...
Getting image source signatures
Copying blob 5a915a173fbc done
Copying config e9ed59d2ba done
Writing manifest to image destination
Storing signatures
  Error processing tar file(exit status 1): there might not be enough IDs available in the namespace (requested 192:192 for /run/systemd/netif): lchown /run/systemd/netif: invalid argument
Trying to pull registry.fedoraproject.org/fedora:latest...
Getting image source signatures
Copying blob ed60cb1abc2e done
Copying config 02781e9f50 done
Writing manifest to image destination
Storing signatures
  Error processing tar file(exit status 1): there might not be enough IDs available in the namespace (requested 0:22 for /run/utmp): lchown /run/utmp: invalid argument
Trying to pull quay.io/fedora:latest...
  error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2 Final//EN\">\n<title>404 Not Found</title>\n<h1>Not Found</h1>\n<p>The requested URL was not found on the server.  If you entered the URL manually please check your spelling and try again.</p>\n"
Trying to pull registry.access.redhat.com/fedora:latest...
  name unknown: Repo not found
Trying to pull registry.centos.org/fedora:latest...
  manifest unknown: manifest unknown
Error: unable to pull fedora:latest: 5 errors occurred:
	* Error committing the finished image: error adding layer with blob "sha256:5a915a173fbc36dc8e1410afdd9de2b08f71efb226f8eb1ebcdc00a1acbced62": Error processing tar file(exit status 1): there might not be enough IDs available in the namespace (requested 192:192 for /run/systemd/netif): lchown /run/systemd/netif: invalid argument
	* Error committing the finished image: error adding layer with blob "sha256:ed60cb1abc2e112aa7a26e9f52cc0982551f7e959441522260e03f47437d42b9": Error processing tar file(exit status 1): there might not be enough IDs available in the namespace (requested 0:22 for /run/utmp): lchown /run/utmp: invalid argument
	* Error initializing source docker://quay.io/fedora:latest: Error reading manifest latest in quay.io/fedora: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2 Final//EN\">\n<title>404 Not Found</title>\n<h1>Not Found</h1>\n<p>The requested URL was not found on the server.  If you entered the URL manually please check your spelling and try again.</p>\n"
	* Error initializing source docker://registry.access.redhat.com/fedora:latest: Error reading manifest latest in registry.access.redhat.com/fedora: name unknown: Repo not found
	* Error initializing source docker://registry.centos.org/fedora:latest: Error reading manifest latest in registry.centos.org/fedora: manifest unknown: manifest unknown

The error from registry.fedoraproject.org is gone now. :)

debarshiray · 2019-09-24T16:53:03Z

The problem was that my user's UID was in the middle of the sub?id range.

That sounds a bit like https://github.com/debarshiray/toolbox/issues/268

Do you remember what the exact problem was? eg., what was your UID and what the subuid range looked like? I am trying to figure out if we can have a better error message for cases like these.

npmccallum · 2020-02-12T22:20:20Z

@debarshiray I just noticed your comment here. The problem is that my UID came from sssd. So I didn't even think about what it was. A good check would be to see if the user's UID is in the middle of their subuid range and warn them. In general, I think a warning about how to set up a proper /etc/subuid could be useful.

This is partly the problem that the kernel doesn't provide any real allocation strategy for UID. We need essentially protected mode for UID/GID where login agents can reserve ranges for their use. Then, after login, user agents can request ranges. But someone has to own this allocation strategy.

As it stands right now, I believe there are vulnerabilities happening today where UID collisions across namespaces allow for escalation of privleges and accidental data disclosure.

openshift-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Sep 16, 2019

mheon added the rootless label Sep 16, 2019

npmccallum closed this as completed Sep 18, 2019

github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 23, 2023

github-actions bot locked as resolved and limited conversation to collaborators Sep 23, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

podman on silverblue fails to create uid namespace #4047

podman on silverblue fails to create uid namespace #4047

npmccallum commented Sep 16, 2019

mheon commented Sep 16, 2019

giuseppe commented Sep 17, 2019

debarshiray commented Sep 17, 2019

npmccallum commented Sep 17, 2019

npmccallum commented Sep 17, 2019

npmccallum commented Sep 18, 2019

debarshiray commented Sep 18, 2019

debarshiray commented Sep 24, 2019

npmccallum commented Feb 12, 2020

podman on silverblue fails to create uid namespace #4047

podman on silverblue fails to create uid namespace #4047

Comments

npmccallum commented Sep 16, 2019

mheon commented Sep 16, 2019

giuseppe commented Sep 17, 2019

debarshiray commented Sep 17, 2019

npmccallum commented Sep 17, 2019

npmccallum commented Sep 17, 2019

npmccallum commented Sep 18, 2019

debarshiray commented Sep 18, 2019

debarshiray commented Sep 24, 2019

npmccallum commented Feb 12, 2020