Trouble Running NVIDIA GPU Containers on Custom Yocto-Based Distro on HPE Server with NVIDIA A40 GPU

[Nauman3S]

I'm experiencing difficulties running NVIDIA GPU containers on a custom Yocto-based distribution tailored for an HPE server equipped with an NVIDIA A40 GPU. Despite having set up a custom meta-nvidia layer (mickledore branch), which includes recipes for NVIDIA drivers, libnvidia-container, libtirpc, and nvidia-container-toolkit (based on meta-tegra's recipes-containers layer at OE4T/meta-tegra), I encounter errors when attempting to run containers that utilize the GPU.

Distro Details:

Distro: poky Included Recipes and Layers: containerd, virtualization layers, NVIDIA drivers and kernel modules, systemd, kernel headers, etc.

Issue Reproduction Steps:

Configuring the container runtime:

sudo nvidia-ctk runtime configure --runtime=containerd
sudo systemctl restart containerd

Pulling images for testing:

sudo ctr images pull docker.io/nvidia/cuda:12.0.0-runtime-ubuntu20.04
sudo ctr images pull docker.io/nvidia/cuda:12.0.0-runtime-ubi8
sudo ctr images pull docker.io/nvidia/cuda:12.0.0-base-ubuntu20.04
sudo ctr images pull docker.io/nvidia/cuda:12.0.0-base-ubi8
Running a container with GPU:

sudo ctr run --rm --gpus 0 --runtime io.containerd.runc.v1 --privileged docker.io/nvidia/cuda:12.0.0-runtime-ubuntu20.04 test nvidia-smi

Error Message:

ctr: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: ldcache error: process /sbin/ldconfig.real failed with error code: 1: unknown
This error persists across all pulled NVIDIA images(non-ubuntu based images show the same error but with /sbin/ldconfig instead of /sbin/ldconfig.real. However, non-GPU containers (e.g., docker.io/macabees/neofetch:latest) work without issues.

Further Details:

Running ldconfig -p shows 264 libs found, including various NVIDIA libraries while running ldconfig outputs no error.

uname -a: Kernel version: Linux intel-corei7-64-02 6.1.38-intel-pk-standard #1 SMP PREEMPT_DYNAMIC Thu Jul 13 04:53:52 UTC 2023 x86_64 GNU/Linux

Output from sudo nvidia-container-cli -k -d /dev/tty info includes warnings about missing libraries and compat32 libraries, although nvidia-smi shows the GPU is recognized correctly.

Attempted Solutions:

Verifying all NVIDIA driver and toolkit components are correctly installed. Ensuring the ldconfig cache is current and includes paths to the NVIDIA libraries and /sbin/ldconfig.real is a symlink to /sbin/ldconfig.

Despite these efforts, the error persists, and GPU containers fail to start. I'm seeking advice on resolving this ldcache and container initialization error to run NVIDIA GPU containers on this custom Yocto-based distribution.

[elezar]

@Nauman3S which verison of the NVIDIA Container Toolkit are you using?

The issue is most likely that the ldconfig entry in the /etc/nvidia-container-runtime/config.toml file is incorrect. We should be resolving the proper ldconfig path and only using ldconfig.real if it actually exists (e.g. on ubuntu systems).

Could you also provide the content of the config file?

[Nauman3S]

@elezar my /etc/nvidia-container-runtime/config.toml file:

disable-require = false
[nvidia-container-cli]
#root = "/run/nvidia/driver"
#path = "/usr/bin/nvidia-container-cli"
environment = []
debug = "/var/log/nvidia-container-toolkit.log"
ldcache = "/etc/ld.so.cache"
load-kmods = true
no-cgroups = true
#user = "root:video"
ldconfig = "/sbin/ldconfig"
#alpha-merge-visible-devices-envvars = false

[nvidia-container-runtime]
debug = "/var/log/nvidia-container-runtime.log"
log-level = "debug"

# Specify the runtimes to consider. This list is processed in order and the PATH
# searched for matching executables unless the entry is an absolute path.
runtimes = [
        "runc",
]

mode = "auto"

[nvidia-container-runtime.modes.csv]
mount-spec-path = "/etc/nvidia-container-runtime/host-files-for-container.d"

I also tried it with ldconfig = "@/sbin/ldconfig" and ldconfig = "@/sbin/ldconfig.real" but the result is same.
And the nvidia-ctk -v outputs

nvidia-ctk -v 
NVIDIA Container Toolkit CLI version 1.14.6
commit: 4668c511de4b311c96bc3dd0310bff40b75083bd

[elezar]

Could you replace

ldconfig = "/sbin/ldconfig"

with

ldconfig = "@/sbin/ldconfig"

The @ indicates that the path is on the host -- meaning that the ldconfig on the host is used to update the ldcache in the container.

[Nauman3S]

Thank you. I already changed it to use @ prefix but still get the same error. I'm wondering is there any way to get more verbose logs from the cli?

[elezar]

@Nauman3S unfortunately there aren't too many logs available for that specific part of the code. One thing that you could try is whether the error perists when the v1.15.0-rc.4 release candidate is used. We have made some changes to the normalization of the ldconfig path. Here it is still recommended to use @/sbin/ldconfig in the config file.

I will have a look to see if there's anything obvious that's amis with how things are being handled.