Podman via "podman machine": docker commands fail with ""potentially insufficient UIDs or GIDs available in user namespace"

[ahmetb]

/kind bug

Description

Fresh install of Podman CLI + VM on a macOS.
Some "docker" commands are working (aliased to "podman"), but "docker build" is failing with error:

Error: potentially insufficient UIDs or GIDs available in user namespace (requested 445605:89939 for /var/tmp/libpod_builder449833426/build/Dockerfile): Check /etc/subuid and /etc/subgid: lchown /var/tmp/libpod_builder449833426/build/Dockerfile: invalid argument

Steps to reproduce the issue:

1. brew install podman

2. podman machine init && podman machine start (note warning)

     INFO[0000] waiting for clients...
     INFO[0000] listening tcp://0.0.0.0:7777
     INFO[0000] new connection from  to /var/folders/9h/97gkgcl96vz3vvb9v52tw3gw00dl55/T/podman/qemu_podman-machine-default.sock
     Waiting for VM ...
     qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.80000001H:ECX.svm [bit 2]
   

3. podman machine list shows Running

4. alias docker=podman

Describe the results you received:

5. docker ps works, docker info shows Podman VM.

6. docker build in a basic sample image directory fails with error:

   Error: potentially insufficient UIDs or GIDs available in user namespace (requested 445605:89939 for /var/tmp/libpod_builder449833426/build/Dockerfile): Check /etc/subuid and /etc/subgid: lchown /var/tmp/libpod_builder449833426/build/Dockerfile: invalid argument

Describe the results you expected:

docker build should work?

Output of podman version:

Client:
Version:      3.3.1
API Version:  3.3.1
Go Version:   go1.17
Built:        Mon Aug 30 12:15:26 2021
OS/Arch:      darwin/amd64

Server:
Version:      3.3.0
API Version:  3.3.0
Go Version:   go1.16.6
Built:        Fri Aug 20 12:36:14 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.22.3
  cgroupControllers: []
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.29-2.fc34.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.29, commit: '
  cpus: 1
  distribution:
    distribution: fedora
    version: "34"
  eventLogger: journald
  hostname: localhost
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.13.12-200.fc34.x86_64
  linkmode: dynamic
  memFree: 1670017024
  memTotal: 2061852672
  ociRuntime:
    name: crun
    package: crun-0.21-1.fc34.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.21
      commit: c4c3cdf2ce408ed44a9e027c618473e6485c635b
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.9-1.fc34.x86_64
    version: |-
      slirp4netns version 1.1.8+dev
      commit: 6dc0186e020232ae1a6fcc1f7afbc3ea02fd3876
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.0
  swapFree: 0
  swapTotal: 0
  uptime: 7m 51.39s
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 0
  runRoot: /run/user/1000/containers
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 3.3.0
  Built: 1629488174
  BuiltTime: Fri Aug 20 19:36:14 2021
  GitCommit: ""
  GoVersion: go1.16.6
  OsArch: linux/amd64
  Version: 3.3.0

Package info (e.g. output of rpm -q podman or apt list podman):

$ brew info podman

podman: stable 3.3.1 (bottled), HEAD
Tool for managing OCI containers and pods
https://podman.io/
/usr/local/Cellar/podman/3.3.1 (170 files, 39MB) *
  Poured from bottle on 2021-09-01 at 13:02:11
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/podman.rb
License: Apache-2.0
==> Dependencies
Build: go ✘, go-md2man ✘
Required: qemu ✔
==> Options
--HEAD
	Install HEAD version
==> Caveats
zsh completions have been installed to:
  /usr/local/share/zsh/site-functions

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes.

Additional environment details (AWS, VirtualBox, physical, etc.): None. Running on QEMU on macOS via "podman machine".

[mheon]

This looks like a more general rootless issue. We allocate, by default, 65536 UIDs and GIDs for each rootless user; you're using an image that uses high UIDs and GIDs (445605:89939 are both higher than 65536), so it's not compatible with a default rootless configuration. You could SSH into the machine and manually edit /etc/subuid and /etc/subgid to increase the number of UIDs and GIDs available, or you can use the root connection that runs Podman in the VM as root.

@baude What's the recommended way to swap to the root connection in the VM?

[baude]

please provide your Dockerfile or Containerfile. I just tried a basic Containerfile I have and it worked perfectly

[ahmetb]

This is the image used https://github.com/GoogleContainerTools/skaffold/tree/main/examples/getting-started

In general I am not a very smart user who can adjust uid ranges whatnot, so it would be good if it just worked.

[mheon]

@baude It's specific to the image, he has high UIDs and GIDs that render rootless Podman unusable without adjustment. I thought we had a root connection also available by default in machine - setting that to the default should resolve things?

[hiyoko3m]

I have also encountered the same problem. In my fresh install of Podman (via brew install podman) on macOS 10.15.7 with Intel, even with this simple Dockerfile:

FROM docker.io/library/httpd:latest
RUN echo '<html><head></head><body>It works!</body></html>' >  /usr/local/apache2/htdocs/index.html

podman build . showed the same error:

$ podman build .
Error: potentially insufficient UIDs or GIDs available in user namespace (requested 891479843:1066215063
for /var/tmp/libpod_builder366076066/build/Dockerfile): Check /etc/subuid and /etc/subgid:
lchown /var/tmp/libpod_builder366076066/build/Dockerfile: invalid argument

I have also tried to adjust UIDs and GIDs by looking https://github.com/containers/podman/blob/main/docs/tutorials/rootless_tutorial.md#etcsubuid-and-etcsubgid-configuration :

$ podman machine ssh
Connecting to vm podman-machine-default. To close connection, use `~.` or `exit`
Warning: Permanently added '[localhost]:61148' (ECDSA) to the list of known hosts.
Fedora CoreOS 34.20210821.1.1
Tracker: https://github.com/coreos/fedora-coreos-tracker
Discuss: https://discussion.fedoraproject.org/c/server/coreos/

Last login: Fri Sep  3 03:28:21 2021 from 192.168.127.1
[core@localhost ~]$ sudo usermod --add-subuids 890000000-900000000 --add-subgids 1060000000-1070000000 core
[core@localhost ~]$ podman system migrate
[core@localhost ~]$ exit
logout
Connection to localhost closed.

$ podman info
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    - container_id: 65537
      host_id: 1060000000
      size: 10000001
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    - container_id: 65537
      host_id: 890000000
      size: 10000001
~~~~~~~~~~~~~~~~~~~~~~~~~~~

but still got:

$ podman build .
Error: potentially insufficient UIDs or GIDs available in user namespace (requested 891479843:1066215063
for /var/tmp/libpod_builder063750535/build/Dockerfile): Check /etc/subuid and /etc/subgid:
lchown /var/tmp/libpod_builder063750535/build/Dockerfile: invalid argument                 

[hiyoko3m]

Ah, sorry, I have thought the number of uid and gid correspond to host_id in podman info's output, but these actually correspond to container_id. So trying with this /etc/subuid and /etc/subgid (the same file)

core:100000:1500000000

now podman build . works. (But I wonder why such a big number is assigned.)

[mheon]

Aha - it’s just the Dockerfile. Is your UID on the host equal to what’s being shown in the error message?

…

On Thu, Sep 2, 2021 at 23:47 hiyoko3m ***@***.***> wrote: I have also encountered the same problem. In my fresh install of Podman (via brew install podman) on macOS 10.15.7 with Intel, even with this simple Dockerfile: FROM docker.io/library/httpd:latest RUN echo '<html><head></head><body>It works!</body></html>' > /usr/local/apache2/htdocs/index.html podman build . showed the same error: $ podman build . Error: potentially insufficient UIDs or GIDs available in user namespace (requested 891479843:1066215063 for /var/tmp/libpod_builder366076066/build/Dockerfile): Check /etc/subuid and /etc/subgid: lchown /var/tmp/libpod_builder366076066/build/Dockerfile: invalid argument I have also tried to adjust UIDs and GIDs by looking https://github.com/containers/podman/blob/main/docs/tutorials/rootless_tutorial.md#etcsubuid-and-etcsubgid-configuration : $ podman machine ssh Connecting to vm podman-machine-default. To close connection, use `~.` or `exit` Warning: Permanently added '[localhost]:61148' (ECDSA) to the list of known hosts. Fedora CoreOS 34.20210821.1.1 Tracker: https://github.com/coreos/fedora-coreos-tracker Discuss: https://discussion.fedoraproject.org/c/server/coreos/ Last login: Fri Sep 3 03:28:21 2021 from 192.168.127.1 ***@***.*** ~]$ sudo usermod --add-subuids 890000000-900000000 --add-subgids 1060000000-1070000000 core ***@***.*** ~]$ podman system migrate ***@***.*** ~]$ exit logout Connection to localhost closed. $ podman info ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ idMappings: gidmap: - container_id: 0 host_id: 1000 size: 1 - container_id: 1 host_id: 100000 size: 65536 - container_id: 65537 host_id: 1060000000 size: 10000001 uidmap: - container_id: 0 host_id: 1000 size: 1 - container_id: 1 host_id: 100000 size: 65536 - container_id: 65537 host_id: 890000000 size: 10000001 ~~~~~~~~~~~~~~~~~~~~~~~~~~~ but still got: $ podman build . Error: potentially insufficient UIDs or GIDs available in user namespace (requested 891479843:1066215063 for /var/tmp/libpod_builder063750535/build/Dockerfile): Check /etc/subuid and /etc/subgid: lchown /var/tmp/libpod_builder063750535/build/Dockerfile: invalid argument — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#11399 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB3AOCEDP5EICDN46GNWKVDUABAORANCNFSM5DHJPLPQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[hiyoko3m]

This is the output on my mac; I see where these number come from. Thanks!

$ id -u
891479843
$ id -g
1066215063

[jk-ulti]

It seems this wasn't just a one-off issue, I experienced this error and the inability to build any image as well. Editing the /etc/subuid and /etc/subgid files as above and running podman system migrate fixed the issue. I asked around my coworkers and so far everyone has high numbered IDs.

[mheon]

@nalind I recall a patch from you to change the build archiver to squash UIDs and GIDs down to 0 - would that fix this?

[nalind]

@mheon yeah, this error is happening while attempting to chown a path that podman's using to extract part of a build context (a temporary using "libpod_builder" as a prefix), so I expect #11473 would solve it.

[mheon]

Closing on that theory. Fix should be in 3.3.2, coming on Monday.

[deepforu47]

I am still getting below error with Podman "3.4.4" on Mac.

Error: writing blob: adding layer with blob "sha256:b4b03dbaa949daab471f94bcfd68cbe21c1147e8ec2acfe3f46f1520db48baeb": Error processing tar file(exit status 1): potentially insufficient UIDs or GIDs available in user namespace (requested 630384594:600260513 for /usr/bin/etcdctl): Check /etc/subuid and /etc/subgid: lchown /usr/bin/etcdctl: invalid argument

[baude]

can you open a new issue and put in your reproducer? Please give as much information as possible