Instructions needed how to run podman under SLURM

[sampie]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind feature

Description

Steps to reproduce the issue:

1. $ srun --pty /bin/bash

2. $ podman run -it ubuntu /bin/bash

Describe the results you received:
Error message saying: 'standard_init_linux.go:211: exec user process caused "permission denied" '

Describe the results you expected:
Entering a container. Or at least an error message stating what is the problem and how to solve it.

Additional information you deem important (e.g. issue happens only occasionally):

Info gives a different error message. It looks that when running a SLURM session, there is no user's folder under /run/user/, but still podman is expecting to have it.

$ podman info
Error: could not get runtime: error generating default config from memory: cannot mkdir /run/user/16869/libpod: mkdir /run/user/16869/libpod: no such file or directory

Output of podman version:

$ podman version
Version:            1.6.4
RemoteAPI Version:  1
Go Version:         go1.12.12
OS/Arch:            linux/amd64

Output of podman info --debug:

$ podman info --debug
debug:
  compiler: gc
  git commit: ""
  go version: go1.12.12
  podman version: 1.6.4
host:
  BuildahVersion: 1.11.5
  CgroupVersion: v1
  Conmon:
    package: conmon-2.0.8-1.el7.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.8, commit: f85c8b1ce77b73bcd48b2d802396321217008762'
  Distribution:
    distribution: '"centos"'
    version: "7"
  IDMappings:
    gidmap:
    - container_id: 0
      host_id: 202
      size: 1
    - container_id: 1
      host_id: 1000000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 16869
      size: 1
    - container_id: 1
      host_id: 1000000
      size: 65536
  MemFree: 337595760640
  MemTotal: 404287807488
  OCIRuntime:
    name: runc
    package: runc-1.0.0-67.rc10.el7_8.x86_64
    path: /usr/bin/runc
    version: 'runc version spec: 1.0.1-dev'
  SwapFree: 4288409600
  SwapTotal: 4294963200
  arch: amd64
  cpus: 2
  eventlogger: file
  hostname: node01
  kernel: 3.10.0-1160.11.1.el7.x86_64
  os: linux
  rootless: true
  slirp4netns:
    Executable: /usr/bin/slirp4netns
    Package: slirp4netns-0.4.3-4.el7_8.x86_64
    Version: |-
      slirp4netns version 0.4.3
      commit: 2244b9b6461afeccad1678fac3d6e478c28b4ad6
  uptime: 2227h 2m 40.96s (Approximately 92.79 days)
registries:
  blocked: null
  insecure: null
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  ConfigFile: /home/tkoop/97/sampie/.config/containers/storage.conf
  ContainerStore:
    number: 15
  GraphDriverName: vfs
  GraphOptions: {}
  GraphRoot: /tmp/images-1001
  GraphStatus: {}
  ImageStore:
    number: 1
  RunRoot: /run/user/16869/containers
  VolumePath: /tmp/images-1001/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

podman-1.6.4-29.el7_9.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

No

This is a production environment and podman is coming from official production quality repositories. As far as I know the only package repository for more recent podman is Kubic project's, but it is stating that it is not a preferred for production environments: "These packages haven’t been through the official Red Hat QA process and may not be preferable for production environments."

Additional environment details (AWS, VirtualBox, physical, etc.):

[vrothberg]

Thanks for reaching out.

There are a couple of resource online that may be help in getting it to work:

• https://podman.io/blogs/2019/09/26/podman-in-hpc.html
• https://github.com/eriksjolund/slurm-container-cluster

There was a recent workshop by AWS that goes into the details how to run Podman on Slurm but https://pcluster-podman-gromacs.workshop.aws/ee/verify/slurm.html is not reachable anymore. @ChristianKniep, was is taken offline or is it temporary?

[ChristianKniep]

superseded by containers-on-pcluster.workshop.aws, I have yet to add Podman. ATM it uses AmazonLinux2 (CentOS7), the installation of Cloud9 on CentOS8 blocked me.
@sampie you are using CentOS7?
@vrothberg I can dig up my centos7 install script and add it as post-install...

[sampie]

@ChristianKniep Yes, the computers in this cluster have CentOS 7. I am wondering if I can somehow run podman in a SLURM job.

[eriksjolund]

I've tried to create a GitHub actions workflow to build Podman for CentOS 7 and CentOS 8.
The project is in a flux and experimental. (I haven't even tested the latest builds).
In the workflow there is a build matrix where you can specify software versions. Some of the components
are not built but just downloaded from GitHub and added to the resulting tar archive.

Anyway, maybe the project could be of interest to those that want to build Podman and then run it on a compute cluster.

    strategy:
      matrix:
        go-version: [1.15.3]
        podman-version: [ad1aaba8df96cb25e12fe28ec96f3c131e572e3e]
        conmon-version: [v2.0.27]
        centos-version: [8, 7]
        CNI-plugins-version: [v0.9.1]
        crun-version: [0.18]
        slirp4netns-version: [v1.1.9]
        fuse-overlayfs-version: [v1.4.0]
        installprefix: [/home/erik.sjolund/podman]

Another project of mine is
https://github.com/eriksjolund/slurm-container-cluster
That project is probably more useful in other situations, e.g. when you have some spare desktop computers and would like to boot them up with Fedora CoreOS USB sticks and then run a Slurm cluster on them. The Slurm software components run in containers and the Slurm jobs will execute as "Podman-in-Podman" (i.e. running a container in a container).

[sampie]

Did I understood correctly that not yet such solution exists, which allows clusters with existing SLURM installation to run podman as a job?

[kniec]

Podman also allows for distributed container runs using mpirun.
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/building_running_and_managing_containers/index
https://www.canopie-hpc.org/wp-content/uploads/2019/12/prace_azab.pdf
I need to get back to my podman chapter in https://containers-on-pcluster.workshop.aws/self-console/ to add it in. Currently I have a little blocker to update to CentOS8 and with CentOS7 is a bit manual labor - as I am time constrained I am hesitant to put that in.

[vrothberg]

Thanks a lot, @kniec! I leave the issue open for reference.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@vrothberg @eriksjolund What is the goal of this issue? Should it still be open? Should we add more documentation?

[vrothberg]

Documentation would be my goal. Something in the man pages and maybe a more verbose doc in the tutorials?

[sampie]

From user's point of view, it looks like podman does not work under slurm. Does it work? How? I think in this thread there are instructions for mpi, which appears not be be the same as running slurm jobs. Then there are some very special slurm setup cases, which might not match what users have.

What actually happens is that when trying to run podman under slurm, there will be 'standard_init_linux.go:211: exec user process caused "permission denied" ' error and no explanation what is wrong and how to fix it.

[eriksjolund]

I think it would be great if Podman had some instructions targeted for this type of HPC user

• the user does not have any root privileges
• the user's home directory resides in a network file system where subuids are not understood. (And maybe even XATTRS support could be missing)
• the user has write access to a directory on local disk (outside of the home directory). Podman could be configured to use that directory by setting rootless_storage_path in storage.conf.
• the Linux distribution is old (for instance CentOS 7).
• fusermount3 is missing
• podman is not installed

I have access to a CentOS 7 cluster that is also is running Slurm. It's configured like this:

$ cat /etc/sysctl.d/99-rootless.conf 
user.max_user_namespaces=28633
$ cat /proc/sys/user/max_user_namespaces
28633
$ 

/etc/subuid and /etc/subgid are configured to give my user a range of subuids and subgids. As I don't have root permissions, I needed to ask the system group to set up the files /etc/subuid, /etc/subgid and /etc/sysctl.d/99-rootless.conf .

My plan is to see if it is possible to run Podman in a compute job submitted with Slurm.

Today I made a minimal to test to see if could run Podman 3.2.2 (without Slurm):

$ podman --version
podman version 3.2.2
$ podman run --rm -ti docker.io/library/alpine:latest cat /etc/os-release
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.14.0
PRETTY_NAME="Alpine Linux v3.14"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://bugs.alpinelinux.org/"
$

That seems to work.

I used this storage.conf

[storage]
graphroot = "/tmp/erik.sjolund_test3"
rootless_storage_path = "/tmp/erik.sjolund_test4"
driver = "vfs"

with Podman downloaded from
https://github.com/eriksjolund/build-podman/actions/runs/975701693
that was built with a GitHub Actions workflow.

It seems the graphroot directory is empty (/tmp/erik.sjolund_test3) so I guess only the rootless_storage_path setting is needed.

I haven't performed any Slurm tests yet though. (The cluster is often heavily used by other people. I would need to find some good time to test it).

[rhatdan]

This seems like a rather specific use case to document in general documention. I think you should blog on your experience. But I don't see where the upstream would put something like this.

[eriksjolund]

@rhatdan yes, you have point. My use case is probably a bit too specific.

What if there would be a more general document about "surviving in the harshest conditions", that would spell out: What are the minimal requirements for running a container with Podman?

If I understand correctly this is always required:

The value of /proc/sys/user/max_user_namespaces would need to be a positive number (or a positive number that is large enough)

Running a container with single UID mapping might then be possible (podman run --uidmap $container_uid:0:1 --user $container_uid ...)

To be able to run a container without single UID mapping, you would additionally need
/usr/bin/newuidmap and have the files /etc/subuid and /etc/subgid configured.

I would guess there are only a few dependencies where access to the home directory and the /etc/containers directory is absolutely needed. At least over time it seems that Podman has become less and less dependent on those directories.

By setting environment variables, such as CONTAINERS_CONF and CONTAINERS_STORAGE_CONF,
you can for instance relax the dependencies on the home directory and the /etc/containers directory.

In my opinion it would be great if it would be possible to run a Podman container without any file access in the
the home directory and the /etc/containers directory.

How does my discussion relate to the topic of this GitHub issue (Slurm)?
I think running Slurm on compute clusters will be such a "harsh environment".
The comment about the missing user's folder under /run/user in
#10039 (comment)
shows an example of such a "harsh environment" condition.

[sampie]

@eriksjolund If you find a way to run podman under a SLURM job, I would be very interested testing the method / config.

[sampie]

Has it been identified which slurm and podman configuration options enable podman to work under a slurm job? Are there any requirements for podman versions, for example if a newer than 1.6.4 is needed?

[caramarc]

Running interactive as user on the system (submit nodes via ssh) podman runs just fine, under slurm it falls apart with:

Error: container_linux.go:349: starting container process caused "process_linux.go:297: applying cgroup configuration for process caused \"failed to write 31852 to cgroup.procs: write /sys/fs/cgroup/devices/slurm/uid_10749/job_416144/step_0/2dfbfdbe10f9a351dc7124a70a798c27ad2b0aeef054051c7953f3f83d096a7a/cgroup.procs: operation not permitted\"": OCI permission denied

Any hints on workarounds?
The ONLY way I was able to run it under slurm is by changing ProctrackType=proctrack/cgroup to ProctrackType=proctrack/linuxproc which is not recommended. (NOTE: "proctrack/linuxproc" and "proctrack/pgid" can fail to identify all processes associated with a job since processes can become a child of the init process (when the parent process terminates) or change their process group. To reliably track all processes, "proctrack/cgroup" is highly recommended. https://slurm.schedmd.com/slurm.conf.html#OPT_ProctrackType)
It looks like slurm cgroups and podman trying to work under it don't play nice.

[eriksjolund]

@caramarc Interesting results!

There are some command-line options for podman run that have names starting with cgroup

• --cgroupns=mode
• --cgroups=enabled|disabled|no-conmon|split
• --cgroup-parent=path
• --cgroup-conf=KEY=VALUE

If anyone wants to experiment, maybe something like --cgroups=disabled could be used together with ProctrackType=proctrack/cgroup ...

(Currently I have little time over, otherwise I would like to try it out)

About the lack of cgroups v2 support in Slurm

It looks like there is some ongoing development for introducing cgroupsv2 support in Slurm but it
does not look ready yet. The plugin cgroup/v2 was removed in June 2021 with the git commit message:

Remove cgroup/v2 plugin for now.

Still under active development.

I also found this quote:
Slurm is not compatible with cgroups v2 (quote from Archwiki)

[caramarc]

@eriksjolund I did a lot more testing after my post and this specific error is related to kernels 3.10.x. It works just fine in kernels 4.18.x. Whatever cgroup v1 changes happened between kernels 3.10.x and 4.18.x addressed this issue.

kernel 3.10x:
bash-4.2$ mkdir /sys/fs/cgroup/devices/slurm/uid_10000/job_418146/step_interactive/aaa
bash-4.2# sleep 10000 &
[1] 40939
bash-4.2# echo 40939 >> /sys/fs/cgroup/devices/slurm/uid_10000/job_418146/step_interactive/aaa/cgroup.procs
bash: echo: write error: Operation not permitted

kernel 4.18.x:
bash-4.4# mkdir /sys/fs/cgroup/devices/slurm/uid_10000/job_13/step_interactive/aaa
bash-4.4# sleep 10000 &
[1] 516560
bash-4.4# echo 516560 >> /sys/fs/cgroup/devices/slurm/uid_10000/job_13/step_interactive/aaa/cgroup.procs
bash-4.4#

[eriksjolund]

@caramarc I see, good news then.
Are you maybe using RHEL 7/CentOS 7 (kernel 3.10) and RHEL 8/CentOS 8 (kernel 4.18)?
(Just a guess)

I would be interested to hear some more details.
Did you use ProctrackType=proctrack/cgroup?
Was there a need to specify any extra command-line arguments to podman run?
Which Podman version did you use?

[caramarc]

@eriksjolund
Yes I am using one of the RHEL7 clone.
I tried multiple version of podman 1.x (whatever gets delivered by default RHEL7.9), 2.2 and 3.1. I also went trough various versions on conmon thinking that was the problem.

I didn't use/specify any extra command-line arguments and yes ProctrackType=proctrack/cgroup

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[NiccoBiondi]

Hello,
I recently installed slurm using the Nvidia https://github.com/NVIDIA/deepops repo.

I configured slurm and podman on the node and they both work fine on their own. I have some difficulties trying to use them together, particular the gpu resource filtering.

After a srun / sbatch on the node, nvidia-smi correctly displays the resources allocated by slurm (e.g. 2 out of 4 gpus), but when we start a container with podman, all the gpu of the machine are available (e.g. 4/4).
This does not happen with singularity / nvidia enroot (only 2/4 gpu are visible from the container).

Is there any way to fix this?

[schneidr]

Do you by chance have some notes on how you got it running? I don't even get Slurm to start containers, I have the same result as the OP.

Edit:

We managed to get it to work by setting the XDG_DATA_HOME variable. Now we have the same problem, with all GPUs being available in the container. Did you ever get this working?

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

This feels more like a discussion than an issue, I am moving to discussion.

[eriksjolund]

Maybe cgroups v2 support has arrived in Slurm?

Slurm provides support for systems with Control Group v2.
quote from
https://github.com/SchedMD/slurm/blob/c89975f03f6407957db12f73616ce0bedd369ee2/doc/html/cgroup_v2.shtml#L5

(I haven't tried it out)

[MrTomRod]

Slurm docs about Cgroup v2: https://slurm.schedmd.com/cgroup_v2.html

[sampie]

Hi,

Has there been any update for this issue? Is there a way to run podman (4.4.1) under a slurm (22.05) job? Is there documentation how to do this?

I have AlmaLinux9 environment with slurm 22.05. If I take an interactive shell as a slurm job and try podman, I get following error:

$ podman version
WARN[0000] Failed to get rootless runtime dir for DefaultAPIAddress: lstat /run/user/16869: no such file or directory
WARN[0000] RunRoot is pointing to a path (/run/user/16869/containers) which is not writable. Most likely podman will fail.
WARN[0000] Failed to get rootless runtime dir for DefaultAPIAddress: lstat /run/user/16869: no such file or directory
WARN[0000] RunRoot is pointing to a path (/run/user/16869/containers) which is not writable. Most likely podman will fail.
Error: mkdir /run/user/16869: permission denied

So, under slurm job there is no /run/user folder, which seems to be required by podman.

[MrTomRod]

This might be the docs you're looking for: https://slurm.schedmd.com/containers.html

You need Slurm-23.02+, cgroups v2 and special settings, I think.