cgroup mount detection is not robust to unusual cgroup configurations #19735
Labels
hcc/jira
stage/accepted
Confirmed, and intend to work on. No timeline committment though.
theme/cgroups
cgroups issues
type/bug
Comments
googol
changed the title
|
Hi @googol can you describe the Cgroups Mount output more? At a passive glance it seems like there are two mounts over |
|
What sort of info would be useful? Any useful commands to post the output of? It does look odd to me as well, but I don't know why it's set up that way, that's how it comes in the OS. Nomad 1.6.4 does manage to work with it though The snippet in the issue body is the relevant lines from |
|
The same error occurs with 1.7.3: Nomad 1.7.3 startup log
|
|
Yeah sorry @googol I don't think I'll be able to debug this until I have a chance to load up slackware in a VM and poke around. It's unclear what the cgroup mount configuration is and Nomad 1.7 makes some assumptions in how that should look. |
|
Ok, that piece of code clearly explains why I'm seeing this problem, and raises some new questions:
|
|
This looks like an unraid specific thing, from what I've looked up now. The init script # Mount Control Groups filesystem interface:
if grep -wq cgroup /proc/filesystems ; then
# Check if unraidcgroup1 is passed over in command line
if grep -wq unraidcgroup1 /proc/cmdline ; then
if [ -d /sys/fs/cgroup ]; then
# See linux-*/Documentation/cgroups/cgroups.txt (section 1.6)
# Check if we have some tools to autodetect the available cgroup controllers
if [ -x /bin/cut -a -x /bin/tail ]; then
# Mount a tmpfs as the cgroup filesystem root
mount -t tmpfs -o mode=0755,size=8M cgroup_root /sys/fs/cgroup
# Autodetect available controllers and mount them in subfolders
controllers="$(/bin/cut -f 1 /proc/cgroups | /bin/tail -n +2)"
for i in $controllers; do
mkdir /sys/fs/cgroup/$i
mount -t cgroup -o $i $i /sys/fs/cgroup/$i
done
unset i controllers
# Eric S. figured out this needs to go here...
echo 1 > /sys/fs/cgroup/memory/memory.use_hierarchy
else
# We can't use autodetection so fall back mounting them all together
mount -t cgroup cgroup /sys/fs/cgroup
fi
else
mkdir -p /dev/cgroup
mount -t cgroup cgroup /dev/cgroup
fi
else
if [ -d /sys/fs/cgroup ]; then
# See https://docs.kernel.org/admin-guide/cgroup-v2.html (section Mounting)
# Mount a tmpfs as the cgroup2 filesystem root
mount -t tmpfs -o mode=0755,size=8M cgroup_root /sys/fs/cgroup
mount -t cgroup2 none /sys/fs/cgroup
else
mkdir -p /dev/cgroup
mount -t cgroup2 none /dev/cgroup
fi
fi
fiThe upstream slackware64-15 sources seem to have a slightly simpler script though: # Mount Control Groups filesystem interface:
if [ -z "$container" ]; then
if grep -wq cgroup /proc/filesystems ; then
if [ -d /sys/fs/cgroup ]; then
# See linux-*/Documentation/cgroups/cgroups.txt (section 1.6)
# Check if we have some tools to autodetect the available cgroup controllers
if [ -x /bin/cut -a -x /bin/tail ]; then
# Mount a tmpfs as the cgroup filesystem root
mount -t tmpfs -o mode=0755,size=8M cgroup_root /sys/fs/cgroup
# Autodetect available controllers and mount them in subfolders
controllers="$(/bin/cut -f 1 /proc/cgroups | /bin/tail -n +2)"
for i in $controllers; do
mkdir /sys/fs/cgroup/$i
mount -t cgroup -o $i $i /sys/fs/cgroup/$i
done
unset i controllers
else
# We can't use autodetection so fall back mounting them all together
mount -t cgroup cgroup /sys/fs/cgroup
fi
else
mkdir -p /dev/cgroup
mount -t cgroup cgroup /dev/cgroup
fi
fi
fiI'll raise an issue with unraid to verify |
|
Update:
I think the cgroup detection logic should be changed from the current model to something a bit more robust. Since it is valid to mount cgroups v2 on top of a tmpfs, checking the first listed mount on the |
tgross
changed the title
tgross
added
theme/cgroups
|
Doing some issue board cleanup and noticed this got left in a bit of a limbo. I've re-titled it to reflect the current state and marked it for roadmapping. |
|
Thanks Tim! Looks like I've forgotten to report back like I said I would in my last comment, but unraid released their changes which fixed my problem as expected. So yeah my original immediate problem is solved (running nomad client on unraid), but of course this could come up in other systems for you. Thanks for the help on this, @shoenig pointing to the relevant bit of the code helped me get this fixed on unraid's side! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Nomad version
The affected client version:
This is also the version on the server.
Version details for v1.6.4 being used as comparison in logs below
Operating system and Environment details
Unraid Version 6.12.6 2023-12-01 (based on slackware-64 version 15). Kernel 6.1.64.
Using prebuilt nomad binary downloaded from hashicorp, with custom packaging & startup scripts required by unraid
Issue
All allocations fail with the following error messages:
Reproduction steps
Start nomad v1.7.2 client, try to run a job on it.
Expected Result
Job runs as normal
Actual Result
No jobs can get allocated
Nomad Client logs (if appropriate)
logs from startup of nomad client v1.7.2:
Nomad 1.6.4 log on the same machine
Node status
Interesting thing here is that nomad v1.7.2 reports cgroups v1 even though the system has cgroups v2 (and nomad 1.6.4 reports it correctly)
Nomad v1.6.4 on the same machine
Cgroups mount
Cgroup controllers
The text was updated successfully, but these errors were encountered: