[Regression] relative idmapped mounts prevent containers from starting in 4.4.0+ #17517

Syquel · 2023-02-15T19:06:05Z

Issue Description

If an idmapped mount is specified which is relative to the container user namespace the container cannot start in Podman 4.4.0+.
Example: test-volume:/mnt/test:idmap=uids=@0-1001-1;gids=@0-1001-1

The container startup is aborted with the following error message: Error: expected integer

This feature is documented in crun.1.md and worked in Podman 4.3.1.

Steps to reproduce the issue

Install Podman 4.4.0+
Run podman --debug run -it --rm --volume 'test-volume:/mnt/test:idmap=uids=@0-1001-1;gids=@0-1001-1' fedora:37

Describe the results you received

The container startup is aborted with the error message Error: expected integer.

Describe the results you expected

The container starts successfully and the volume is idmapped relatively to the container user namespace.

podman info output

host:
  arch: amd64
  buildahVersion: 1.29.0
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: /usr/bin/conmon is owned by conmon 1:2.1.6-1
    path: /usr/bin/conmon
    version: 'conmon version 2.1.6, commit: 158b5421dbac6bda96b1457955cf2e3c34af29bc'
  cpuUtilization:
    idlePercent: 97.42
    systemPercent: 0.75
    userPercent: 1.83
  cpus: 12
  distribution:
    distribution: arch
    version: unknown
  eventLogger: journald
  hostname: syquel.de
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.1.11-arch1-1
  linkmode: dynamic
  logDriver: journald
  memFree: 59464073216
  memTotal: 67230724096
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: /usr/bin/crun is owned by crun 1.8-1
    path: /usr/bin/crun
    version: |-
      crun version 1.8
      commit: 0356bf4aff9a133d655dc13b1d9ac9424706cac4
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: true
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: /usr/bin/slirp4netns is owned by slirp4netns 1.2.0-1
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 0
  swapTotal: 0
  uptime: 9h 35m 35.00s (Approximately 0.38 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries: {}
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 467645579264
  graphRootUsed: 23679283200
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 58
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.4.1
  Built: 1676117906
  BuiltTime: Sat Feb 11 13:18:26 2023
  GitCommit: 34e8f3933242f2e566bbbbf343cf69b7d506c1cf-dirty
  GoVersion: go1.20
  Os: linux
  OsArch: linux/amd64
  Version: 4.4.1

Podman in a container

No

Privileged Or Rootless

Privileged

Upstream Latest Release

Yes

Additional environment details

No response

Additional information

Downgrading podman to version 4.3.1 resolves the issue.
Normal idmapped mounts which are not relative to the container user namespace are working in version 4.4.0.

Full debug log:

# podman --debug run -it --rm --volume 'test-volume:/mnt/test:idmap=uids=@0-1001-1;gids=@0-1001-1' fedora:37
INFO[0000] podman filtering at log level debug
DEBU[0000] Called run.PersistentPreRunE(podman --debug run -it --rm --volume test-volume:/mnt/test:idmap=uids=@0-1001-1;gids=@0-1001-1 fedora:37)
DEBU[0000] Using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /var/lib/containers/storage
DEBU[0000] Using run root /run/containers/storage
DEBU[0000] Using static dir /var/lib/containers/storage/libpod
DEBU[0000] Using tmp dir /run/libpod
DEBU[0000] Using volume path /var/lib/containers/storage/volumes
DEBU[0000] Using transient store: false
DEBU[0000] Set libpod namespace to ""
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that metacopy is being used
DEBU[0000] Cached value indicated that native-diff is not being used
INFO[0000] Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=true
DEBU[0000] Initializing event backend journald
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument
DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument
DEBU[0000] Using OCI runtime "/usr/bin/crun"
INFO[0000] Setting parallel job count to 37
DEBU[0000] Successfully loaded network podman1: &{podman1 7ad3417bb4f87d1cf26b28bdecf6ab907ff3fd6cce65f2eca81e310c2752978e bridge podman1 2022-10-26 10:32:45.354703422 +0200 CEST [{{{10.89.0.0 ffffff00}} 10.89.0.1 <nil>} {{{fd05:a84b:c2aa:96:: ffffffffffffffff0000000000000000}} fd05:a84b:c2aa:96::1 <nil>}] true false true [] map[] map[] map[driver:host-local]}
DEBU[0000] Successfully loaded 1 networks
DEBU[0000] Pulling image fedora:37 (policy: missing)
DEBU[0000] Looking up image "fedora:37" in local containers storage
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf"
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/00-shortnames.conf"
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/01-mirror.conf"
DEBU[0000] Trying "registry.fedoraproject.org/fedora:37" ...
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev]@19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a"
DEBU[0000] Found image "fedora:37" as "registry.fedoraproject.org/fedora:37" in local containers storage
DEBU[0000] Found image "fedora:37" as "registry.fedoraproject.org/fedora:37" in local containers storage ([overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev]@19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a)
DEBU[0000] exporting opaque data as blob "sha256:19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a"
DEBU[0000] Looking up image "registry.fedoraproject.org/fedora:37" in local containers storage
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0000] Trying "registry.fedoraproject.org/fedora:37" ...
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev]@19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a"
DEBU[0000] Found image "registry.fedoraproject.org/fedora:37" as "registry.fedoraproject.org/fedora:37" in local containers storage
DEBU[0000] Found image "registry.fedoraproject.org/fedora:37" as "registry.fedoraproject.org/fedora:37" in local containers storage ([overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev]@19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a)
DEBU[0000] exporting opaque data as blob "sha256:19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a"
DEBU[0000] User mount test-volume:/mnt/test options [idmap=uids=@0-1001-1;gids=@0-1001-1]
DEBU[0000] Looking up image "fedora:37" in local containers storage
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0000] Trying "registry.fedoraproject.org/fedora:37" ...
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev]@19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a"
DEBU[0000] Found image "fedora:37" as "registry.fedoraproject.org/fedora:37" in local containers storage
DEBU[0000] Found image "fedora:37" as "registry.fedoraproject.org/fedora:37" in local containers storage ([overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev]@19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a)
DEBU[0000] exporting opaque data as blob "sha256:19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a"
DEBU[0000] Inspecting image 19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a
DEBU[0000] exporting opaque data as blob "sha256:19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a"
DEBU[0000] exporting opaque data as blob "sha256:19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a"
DEBU[0000] Inspecting image 19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a
DEBU[0000] Inspecting image 19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a
DEBU[0000] Inspecting image 19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a
DEBU[0000] Inspecting image 19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a
DEBU[0000] using systemd mode: false
DEBU[0000] No hostname set; container's hostname will default to runtime default
DEBU[0000] Found apparmor_parser binary in /sbin/apparmor_parser
DEBU[0000] Loading seccomp profile from "/etc/containers/seccomp.json"
DEBU[0000] Allocated lock 0 for container 1486532d5b931a7da2ffc3d996f4d0e6f295ea140c89219dc54d6655dbde1aba
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev]@19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a"
DEBU[0000] exporting opaque data as blob "sha256:19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a"
DEBU[0000] Cached value indicated that idmapped mounts for overlay are supported
DEBU[0000] overlay: mount_data=lowerdir=/var/lib/containers/storage/overlay/l/UHZ6JQHRA2VH24WJWU2U56WK4L,upperdir=/var/lib/containers/storage/overlay/52385078191a0824270ab2ad04e299e834e0090d8d5a1f17d59c3215ed58a1f6/diff,workdir=/var/lib/containers/storage/overlay/52385078191a0824270ab2ad04e299e834e0090d8d5a1f17d59c3215ed58a1f6/work,nodev
DEBU[0000] Created container "1486532d5b931a7da2ffc3d996f4d0e6f295ea140c89219dc54d6655dbde1aba"
DEBU[0000] Container "1486532d5b931a7da2ffc3d996f4d0e6f295ea140c89219dc54d6655dbde1aba" has work directory "/var/lib/containers/storage/overlay-containers/1486532d5b931a7da2ffc3d996f4d0e6f295ea140c89219dc54d6655dbde1aba/userdata"
DEBU[0000] Container "1486532d5b931a7da2ffc3d996f4d0e6f295ea140c89219dc54d6655dbde1aba" has run directory "/run/containers/storage/overlay-containers/1486532d5b931a7da2ffc3d996f4d0e6f295ea140c89219dc54d6655dbde1aba/userdata"
DEBU[0000] Creating new volume test-volume for container
DEBU[0000] Validating options for local driver
DEBU[0000] Handling terminal attach
DEBU[0000] Cached value indicated that volatile is being used
DEBU[0000] overlay: mount_data=lowerdir=/var/lib/containers/storage/overlay/ef39eb2b8fa009a5049949b623a220b3a5529a951e97fd8fd76d4b9acea8f478/mapped/0/l/UHZ6JQHRA2VH24WJWU2U56WK4L,upperdir=/var/lib/containers/storage/overlay/ef39eb2b8fa009a5049949b623a220b3a5529a951e97fd8fd76d4b9acea8f478/diff,workdir=/var/lib/containers/storage/overlay/ef39eb2b8fa009a5049949b623a220b3a5529a951e97fd8fd76d4b9acea8f478/work,nodev,volatile
DEBU[0000] Mounted container "1486532d5b931a7da2ffc3d996f4d0e6f295ea140c89219dc54d6655dbde1aba" at "/var/lib/containers/storage/overlay/ef39eb2b8fa009a5049949b623a220b3a5529a951e97fd8fd76d4b9acea8f478/merged"
DEBU[0000] Going to mount named volume test-volume
DEBU[0000] Copying up contents from container 1486532d5b931a7da2ffc3d996f4d0e6f295ea140c89219dc54d6655dbde1aba to volume test-volume
DEBU[0000] Created root filesystem for container 1486532d5b931a7da2ffc3d996f4d0e6f295ea140c89219dc54d6655dbde1aba at /var/lib/containers/storage/overlay/ef39eb2b8fa009a5049949b623a220b3a5529a951e97fd8fd76d4b9acea8f478/merged
DEBU[0000] Using /sbin/apparmor_parser binary
INFO[0000] Successfully loaded AppAmor profile "containers-default-0.51.0"
DEBU[0000] Skipping unrecognized mount in /etc/containers/mounts.conf: "# Configuration file for default mounts in containers (see man 5"
DEBU[0000] Skipping unrecognized mount in /etc/containers/mounts.conf: "# containers-mounts.conf for further information)"
DEBU[0000] Skipping unrecognized mount in /etc/containers/mounts.conf: ""
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode subscription
DEBU[0000] Cleaning up container 1486532d5b931a7da2ffc3d996f4d0e6f295ea140c89219dc54d6655dbde1aba
DEBU[0000] Network is already cleaned up, skipping...
DEBU[0000] Unmounted container "1486532d5b931a7da2ffc3d996f4d0e6f295ea140c89219dc54d6655dbde1aba"
DEBU[0000] Removing container 1486532d5b931a7da2ffc3d996f4d0e6f295ea140c89219dc54d6655dbde1aba
DEBU[0000] Cleaning up container 1486532d5b931a7da2ffc3d996f4d0e6f295ea140c89219dc54d6655dbde1aba
DEBU[0000] Network is already cleaned up, skipping...
DEBU[0000] Container 1486532d5b931a7da2ffc3d996f4d0e6f295ea140c89219dc54d6655dbde1aba storage is already unmounted, skipping...
DEBU[0000] Removing all exec sessions for container 1486532d5b931a7da2ffc3d996f4d0e6f295ea140c89219dc54d6655dbde1aba
DEBU[0000] Container 1486532d5b931a7da2ffc3d996f4d0e6f295ea140c89219dc54d6655dbde1aba storage is already unmounted, skipping...
DEBU[0000] ExitCode msg: "expected integer"
Error: expected integer
DEBU[0000] Shutting down engines

The text was updated successfully, but these errors were encountered:

Syquel · 2023-02-16T01:12:27Z

After a little bit of digging through the source code I think this is related to the change in fdcc225 which expects the idmap option argument to be %d-%d-%d and does not account for relative idmappings.

podman/libpod/container_internal_common.go

Lines 60 to 75 in fdcc225

    
           func parseOptionIDs(option string) ([]idtools.IDMap, error) { 
        
           	ranges := strings.Split(option, "#") 
        
           	ret := make([]idtools.IDMap, len(ranges)) 
        
           	for i, m := range ranges { 
        
           		var v idtools.IDMap 
        
           		_, err := fmt.Sscanf(m, "%d-%d-%d", &v.ContainerID, &v.HostID, &v.Size) 
        
           		if err != nil { 
        
           			return nil, err 
        
           		} 
        
           		if v.ContainerID < 0 || v.HostID < 0 || v.Size < 1 { 
        
           			return nil, fmt.Errorf("invalid value for %q", option) 
        
           		} 
        
           		ret[i] = v 
        
           	} 
        
           	return ret, nil 
        
           }

This function is called in parseIDMapMountOption which should probably handle the relative idmapping.

podman/libpod/container_internal_common.go

Lines 77 to 118 in fdcc225

    
           func parseIDMapMountOption(idMappings stypes.IDMappingOptions, option string) ([]spec.LinuxIDMapping, []spec.LinuxIDMapping, error) { 
        
           	uidMap := idMappings.UIDMap 
        
           	gidMap := idMappings.GIDMap 
        
           	if strings.HasPrefix(option, "idmap=") { 
        
           		var err error 
        
           		options := strings.Split(strings.SplitN(option, "=", 2)[1], ";") 
        
           		for _, i := range options { 
        
           			switch { 
        
           			case strings.HasPrefix(i, "uids="): 
        
           				uidMap, err = parseOptionIDs(strings.Replace(i, "uids=", "", 1)) 
        
           				if err != nil { 
        
           					return nil, nil, err 
        
           				} 
        
           			case strings.HasPrefix(i, "gids="): 
        
           				gidMap, err = parseOptionIDs(strings.Replace(i, "gids=", "", 1)) 
        
           				if err != nil { 
        
           					return nil, nil, err 
        
           				} 
        
           			default: 
        
           				return nil, nil, fmt.Errorf("unknown option %q", i) 
        
           			} 
        
           		} 
        
           	} 
        
           	uidMappings := make([]spec.LinuxIDMapping, len(uidMap)) 
        
           	gidMappings := make([]spec.LinuxIDMapping, len(gidMap)) 
        
           	for i, uidmap := range uidMap { 
        
           		uidMappings[i] = spec.LinuxIDMapping{ 
        
           			HostID:      uint32(uidmap.ContainerID), 
        
           			ContainerID: uint32(uidmap.HostID), 
        
           			Size:        uint32(uidmap.Size), 
        
           		} 
        
           	} 
        
           	for i, gidmap := range gidMap { 
        
           		gidMappings[i] = spec.LinuxIDMapping{ 
        
           			HostID:      uint32(gidmap.ContainerID), 
        
           			ContainerID: uint32(gidmap.HostID), 
        
           			Size:        uint32(gidmap.Size), 
        
           		} 
        
           	} 
        
           	return uidMappings, gidMappings, nil 
        
           }

giuseppe · 2023-02-16T09:19:31Z

before we were relying on an experimental feature in crun, since there was no support for it in OCI. Now that there is idmapping support in the OCI runtime specs, we are using that to pass down the information.

I'll take a look and see if it is possible to add the same feature to Podman

we were previously using an experimental feature in crun, but we lost this capability once we moved to using the OCI runtime spec to specify the volume mappings in fdcc225. Add the same feature to libpod, so that we can support relative positions for the idmaps. Closes: containers#17517 Signed-off-by: Giuseppe Scrivano <[email protected]>

giuseppe · 2023-02-16T09:36:08Z

PR here: #17522

Syquel · 2023-02-16T18:42:33Z

Maybe this should not be implemented then and mounts / volumes should always be idmapped relatively to the user namespace.

I would prefer that solution because I can not imagine any use case where I would want to do an absolute idmapping or where that would even work.

Syquel added the kind/bug Categorizes issue or PR as related to a bug. label Feb 15, 2023

Syquel changed the title ~~relative idmapped mounts prevent containers from starting in 4.4.0+~~ [Regression] relative idmapped mounts prevent containers from starting in 4.4.0+ Feb 15, 2023

giuseppe self-assigned this Feb 16, 2023

giuseppe mentioned this issue Feb 16, 2023

libpod: support relative positions for idmaps #17522

Merged

openshift-merge-robot closed this as completed in #17522 Feb 20, 2023

Syquel mentioned this issue Feb 27, 2023

securityContext.fsGroup #17583

Open

github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Aug 31, 2023

github-actions bot locked as resolved and limited conversation to collaborators Aug 31, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Regression] relative idmapped mounts prevent containers from starting in 4.4.0+ #17517

[Regression] relative idmapped mounts prevent containers from starting in 4.4.0+ #17517

Syquel commented Feb 15, 2023

Syquel commented Feb 16, 2023 •

edited

Loading

giuseppe commented Feb 16, 2023

giuseppe commented Feb 16, 2023

Syquel commented Feb 16, 2023

[Regression] relative idmapped mounts prevent containers from starting in 4.4.0+ #17517

[Regression] relative idmapped mounts prevent containers from starting in 4.4.0+ #17517

Comments

Syquel commented Feb 15, 2023

Issue Description

Steps to reproduce the issue

Describe the results you received

Describe the results you expected

podman info output

Podman in a container

Privileged Or Rootless

Upstream Latest Release

Additional environment details

Additional information

Syquel commented Feb 16, 2023 • edited Loading

giuseppe commented Feb 16, 2023

giuseppe commented Feb 16, 2023

Syquel commented Feb 16, 2023

Syquel commented Feb 16, 2023 •

edited

Loading