root can not write to file owned by root (and nogroup)

[bernt-matthias]

Issue Description

I'm still quite new to podman and containers and have a hard time to describe the issue

In podman run --rm --entrypoint /bin/bash -p 8888:8888 -u root -e NB_USER=root -e NB_UID=0 -e NB_GID=0 -e JUPYTER_ALLOW_INSECURE_WRITES=true -e NOTEBOOK_ARGS="--allow-root" -it glcr.b-data.ch/jupyterlab/r/base

• cd /usr/local/lib/R/etc
• ls -la Renviron.site: -rw-rw-r-- 1 root nogroup 21 Oct 31 10:38 Renviron.site
• echo >> Renviron.site gives me bash: Renviron.site: Permission denied
• I'm root and member of nogroup (id: uid=0(root) gid=0(root) groups=0(root),65534(nogroup))

I just do not understand why I'm not able to write here. Is there anything special about nogroup? Since I'm root I can chown root:root Renviron.site which then allows me to write.

For another file:

• ls -la Renviron: -rw-r--r-- 1 root root 1858 Oct 31 10:38 Renviron
• echo >> Renviron
• changes the execute bit: ls -la given -rwxr-xr-x 1 root root 1859 Jan 31 09:41 Renviron

lsattr does not work: lsattr: Operation not supported While reading flags on Renviron

xref b-data/jupyterlab-r-docker-stack#5 (comment)

Steps to reproduce the issue

See above

Describe the results you received

• Unexpected: Can not write to file.
• Unexpected: Permissions change.

Describe the results you expected

• Can write to file.
• Permissions do not change while writing.

podman info output

podman --version
podman version 5.2.2



host:
  arch: amd64
  buildahVersion: 1.37.5
  cgroupControllers:
  - cpuset
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.10-1.el9.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: 3ea3d7f99779af0fcd69ec16c211a7dc3b4efb60'
  cpuUtilization:
    idlePercent: 96.98
    systemPercent: 0.75
    userPercent: 2.28
  cpus: 56
  databaseBackend: sqlite
  distribution:
    distribution: rocky
    version: "9.4"
  eventLogger: file
  freeLocks: 2045
  hostname: bioinf3
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 4533
      size: 1
    uidmap:
    - container_id: 0
      host_id: 61715
      size: 1
  kernel: 5.14.0-427.13.1.el9_4.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 1537643773952
  memTotal: 1622256828416
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.10.0-3.el9_4.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.10.0
    package: netavark-1.10.3-1.el9.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.10.3
  ociRuntime:
    name: crun
    package: crun-1.14.3-1.el9.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.14.3
      commit: 1961d211ba98f532ea52d2e80f4c20359f241a98
      rundir: /run/user/61715/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240806.gee36266-2.el9.x86_64
    version: |
      pasta 0^20240806.gee36266-2.el9.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /run/user/61715/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.3-1.el9.x86_64
    version: |-
      slirp4netns version 1.2.3
      commit: c22fde291bb35b354e6ca44d13be181c76a0a432
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 0
  swapTotal: 0
  uptime: 1943h 56m 22.00s (Approximately 80.96 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /gpfs1/schlecker/home/songalax/.config/containers/storage.conf
  containerStore:
    number: 3
    paused: 0
    running: 0
    stopped: 3
  graphDriverName: overlay
  graphOptions:
    overlay.force_mask: "700"
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.13-1.el9.x86_64
      Version: |-
        fusermount3 version: 3.10.2
        fuse-overlayfs: version 1.13-dev
        FUSE library version 3.10.2
        using FUSE kernel interface version 7.31
  graphRoot: /home/songalax/.local/share/containers/storage
  graphRootAllocated: 32985348833280
  graphRootUsed: 19864584454144
  graphStatus:
    Backing Filesystem: gpfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 5
  runRoot: /run/user/61715/containers
  transientStore: false
  volumePath: /gpfs1/schlecker/home/songalax/.local/share/containers/storage/volumes
version:
  APIVersion: 5.2.2
  Built: 1731414899
  BuiltTime: Tue Nov 12 13:34:59 2024
  GitCommit: ""
  GoVersion: go1.22.7 (Red Hat 1.22.7-2.el9_5)
  Os: linux
  OsArch: linux/amd64
  Version: 5.2.2

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

No

Additional environment details

We use GPFS on our HPC.

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[benz0li]

@bernt-matthias Please confirm that everything (except starting RStudio) worked with Podman v4.9.4 on the very same HPC in August 2024.

[bernt-matthias]

I can confirm that it worked back then, but I can't be 100% sure if I or an admin changed anything else.

[benz0li]

Addendum: Group ownership of file /usr/local/lib/R/etc/Renviron.site is actually gid=100(users).

Cross references:

• https://github.com/b-data/jupyterlab-r-docker-stack/blob/97c0801128764fad77101a8e4673bd3fa35925c8/base/latest.Dockerfile#L457-L458
  • https://github.com/b-data/jupyterlab-r-docker-stack/blob/97c0801128764fad77101a8e4673bd3fa35925c8/base/latest.Dockerfile#L24

[benz0li]

@bernt-matthias This [strange group mapping 65534(nogroup)] may be the result of a security policy on your HPC.
ℹ Have a look at Container permission denied: How to diagnose this error.

[benz0li]

@bernt-matthias How does /etc/subuid and /etc/subgid look like for your user?

IMHO

  idMappings:
    gidmap:
    - container_id: 0
      host_id: 4533
      size: 1
    uidmap:
    - container_id: 0
      host_id: 61715
      size: 1

means, that uid/gid mappings are not OK. Cross reference:

• Docker incompatibility: Podman fails if user's uid/gid is mapped in subuid/subgid #5751

---

ℹ See also Basic Setup and Use of Podman in a Rootless environment

[bernt-matthias]

How does /etc/subuid and /etc/subgid look like for your user?

/etc/subuid:

songalax:427680:65536

/etc/subgid

proteom:427680:65536

Edit: id gives me uid=61715(songalax) gid=4533(proteom) groups=4533(proteom),4639(...),5005(...),5012(...),5016(...)

Stumbled over this in the liked docs:

NOTE: this is not currently supported with network installs; these files must be available locally to the host machine. It is not possible to configure this with LDAP or Active Directory.

This only means that subuid and subgid config can't be from LDAP, or? Because the user itself is from LDAP, i.e. no entry in passwd.

[bernt-matthias]

Just realized that you probably need the complete files:

/etc/subuid

evechkr:100000:65536
evetoha:165536:65536
evecoos:231072:65536
greenbone:296608:65536
acme:362144:65536
songalax:427680:65536

/etc/subgid

evechkr:100000:65536
evetoha:165536:65536
evecoos:231072:65536
greenbone:296608:65536
acme:362144:65536
proteom:427680:65536

[benz0li]

IMHO you need songalax:427680:65536 instead of proteom:427680:65536 in /etc/subgid.

Then, everything should work fine.

[bernt-matthias]

Thanks for the info. I will check with our admins regarding the subguid config file and report.

[bernt-matthias]

With the fixed subguid config we now see in the container:

-rw-rw-r--  1 root users   21 Oct 31 10:38 Renviron.site

Writing to this file also behaves normally.

So I guess we can close this issue. Thanks a lot for all your time and patience.