Memory Usage of Allocation is always 0 bytes #9120
Comments
|
I am having the same problem - no memory stats for allocations - but only on one of my nomad instances of four. I'd also add a point of clarification that the allocation doesn't show any RAM allocated, but the node itself does. Node:
Allocation:
Common variable between our situation is running on ARM (Raspberry Pi). I will try to add relevant details as my OS and architectures vary from the original problem. Hopefully the details of my builds will be helpful for further research. I'm running Nomad/Consul across four Raspberry Pi's. The memory display is only working on three of them. WORKING: 2x RPI 3 running Hypriot OS 1.11.1 - armhf kernel. 1x RPI 4 running Hypriot OS 1.12.3 - arm64 kernel (32-bit userland, i.e. docker) Hypriot is a modified Raspbian build optimized for docker workloads. I've made no modifications to the original Hypriot image other than adding nomad/consul and jobs. NOT WORKING: 1x RPI 4 running Ubuntu 20.04 Focal ARM64. I manually installed Docker, docker-compose per docker.com and setup nomad/consul. Image was created from latest Raspberry Pi image builder tool to flash the card with Ubuntu 20.04 LTS ARM64 lite edition (i.e. did not build OS myself). |
|
Hrm, that's very interesting that Ubuntu 20.04 doesn't appear to work, but Hypriot OS does. I wonder if Hypriot does something differently by default? It does say that it's tuned for Docker by default. |
|
I thought it was weird as well. If that is the case, docker inspect doesn't show any memory used.
It seems docker may be the culprit here? Output of |
Yup, that's exactly how it's done for Docker containers. Do y'all see this same behavior with the |
|
@tgross: I've tried with I would imagine that any method used for Opening an issue upstream with Docker sounds like a good idea too, but I unsure if I'd be able to find the right place to report such an issue. I'm happy to test stuff though. |
|
Oh right. If I recall from a previous issue, the Raspberry Pi distros don't have the cgroups you need enabled by default. See the |
|
@tgross thanks for the reply. I see the problem now, but the link you've provided unfortunately does not contain any useful information about how to resolve it. If I understand it correctly, it tells me how I can check that cgroups are enabled, but not how to change the settings. |
|
@tgross Thanks for the tip - that was the problem. Memory status is now updating. Thanks! @sbrl I wasn't sure how to enable cgroups either, but the Resource Isolation documentation page linked to does mention "Some Linux distributions do not boot with all required cgroups enabled by default." I Googled cgroups on Ubuntu 20.04 (which is my distro in question), and it mentions modifying the cmdline.txt file used at boot. It should be found somewhere under /boot I'd guess. I think on a typical Raspberry debian distro it's directly in the /boot dir but YMMV. For Ubuntu 20.04 it's under Here was the output of the cmd from the Nomad docs that displays enabled cgroups: awk '{print $1 " " $4}' /proc/cgroups
#subsys_name enabled
cpuset 1
cpu 1
cpuacct 1
blkio 1
memory 0
devices 1
freezer 1
net_cls 1
perf_event 1
net_prio 1
pids 1
rdma 1As you can see "memory" was disabled. I simply added Full cmdline.txt for reference: I'd suggest this issue can be closed. Thanks for everyone's help and input. |
|
Glad to hear that's working and thanks for the assist there @futuralogic! |
|
I'm actually experiencing this now, since recently (can't tell exacty whem, but some weeks ago?). Was always working fine before. Not sure if it's related to an OS system upgrade or a Nomad upgrade. So see it across all clients, though, Debian 11. This is with the docker driver. Oddly enough, the cgroups seem to be in order: EDIT: Hold off, seems |
|
It's very delayed (sorry about that!)) but I've finally managed to find some time to try out the fix suggested above. On my Raspberry Pis, editing ...fixed the issue. Note that it has to be on the 1st line there - no line breaks ( I implemented the following bash function that applies the fix automatically: check_cgroups_memory() {
echo ">>> Checking memory cgroups";
cgroups_enabled="$(awk '/memory/ { print $2 }' < /proc/cgroups)";
if [[ "${cgroups_enabled}" -ne 0 ]]; then
echo ">>> memory cgroups already enabled";
return 0;
fi
filepath_cmdline="/boot/cmdline.txt";
if [[ ! -e "${filepath_cmdline}" ]]; then
filepath_cmdline="/boot/firmware/cmdline.txt";
fi
if [[ ! -e "${filepath_cmdline}" ]]; then
echo ">>> Failed to find cmdline.txt; can't check for cgroups";
return 1;
fi
if grep -q cgroup_enable=memory /boot/cmdline.txt; then
echo ">>> memory cgroups already present in cmdline.txt, a reboot is required to apply the update";
return 0;
fi
echo ">>> memory cgroups not present in cmdline.txt, enabling....";
(tr -d '\n' <"${filepath_cmdline}" && echo " cgroup_enable=memory cgroup_memory=1") | sudo tee "${filepath_cmdline}.new";
sudo mv "${filepath_cmdline}" "${filepath_cmdline}.old-$(date +"%Y-%m-%d")";
sudo mv "${filepath_cmdline}.new" "${filepath_cmdline}";
echo ">>> New contents of cmdline.txt:";
cat "${filepath_cmdline}";
echo ">>> A reboot is required to apply the changes.";
} |
|
I am having this issue now. I don’t remember when it appeared, but I remember that the problem was still in the versions: I have now updated to but the problem persists. Description of the problem - Memory Usage of any Allocation is always 0 bytes Host Resource Utilization is showing correctly docker stats cat /proc/cgroups Nomad logsConsul logs |
|
@AlekseyMelikov best to open a new issue. This issue is specifically to do with Raspberry Pis / ARM devices. You're using x86_64 there, so while the symptoms are the same the solution is likely to be very different. |
|
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Nomad version
Operating system and Environment details
4 x Raspberry Pi 4 w/4 GiB RAM (worker nodes), 1 x Raspberry Pi 4 w/2GiB RAM (server node, does not run tasks)
Issue
All allocated tasks (at least with the Docker driver) show a memory usage of 0 bytes. It never used to be this way, but then I checked one day and it showed 0 bytes for all my allocations and it hasn't fixed itself. Example screenshot from the web interface:
Reproduction steps
Job file (if appropriate)
Any job file will do, but here's one of mine:
Click to expand
Nomad Client logs (if appropriate)
Logs available upon request, but the logging feature (at least for allocations) isn't working either
Nomad Server logs (if appropriate)
The text was updated successfully, but these errors were encountered: