Fix nvidia container toolkit docker contamination

[ereslibre]

Description of changes

This PR implements three main changes.

• Do not shadow containerd, crio or docker binaries with tools that are meant to set up the host with the NVIDIA container toolkit (https://github.com/NVIDIA/nvidia-container-toolkit/tree/a470818ba7d9166be282cd0039dd2fc9b0a34d73/tools/container)
• Fix the Docker runtime wrapper for Nvidia GPU's. This makes the --runtime=nvidia --gpus= use case work as expected.
• Remove the Nvidia wrapper for podman: according to the Nvidia documentation (https://web.archive.org/web/20240729183805/https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html#configuring-podman), the CDI approach is recommended for podman, and it has had support on NixOS for CDI since 22.05 (podman 4.1.0).

Things done

Tested, with the NixOS options:

virtualisation = {
  containers.enable = true;
  podman.enable = true;
  docker = {
    enable = true;
    # Allows the CDI implementation, while also makes it possible to use the nvidia runtime wrappers.
    package = pkgs.docker_25;
    # For the runtime wrappers/non-CDI case...
    enableNvidia = true;
  };
};
hardware = {
  # Required by the runtime wrappers
  graphics.enable32Bit = true;
  # CDI generation
  nvidia-container-toolkit.enable = true;
};

The following works as expected:

• Podman

# CDI
❯ podman run --rm --device=nvidia.com/gpu=all ubuntu:latest nvidia-smi
Wed Jul 31 08:27:17 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 555.58.02              Driver Version: 555.58.02      CUDA Version: 12.5     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4090        On  |   00000000:41:00.0 Off |                  Off |
| 61%   49C    P8             17W /  450W |       2MiB /  24564MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA GeForce RTX 2080 ...    On  |   00000000:61:00.0 Off |                  N/A |
|  0%   53C    P8             35W /  250W |       1MiB /   8192MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

# Wrapper
# Does not apply. CDI is recommended.

• Docker

# CDI
❯ docker run --rm --device=nvidia.com/gpu=all ubuntu:latest nvidia-smi
Wed Jul 31 08:28:24 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 555.58.02              Driver Version: 555.58.02      CUDA Version: 12.5     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4090        On  |   00000000:41:00.0 Off |                  Off |
| 48%   50C    P8             17W /  450W |       2MiB /  24564MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA GeForce RTX 2080 ...    On  |   00000000:61:00.0 Off |                  N/A |
|  0%   54C    P8             35W /  250W |       1MiB /   8192MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

# Wrapper
❯ docker run --rm -it --runtime=nvidia --gpus=all ubuntu:latest nvidia-smi
Wed Jul 31 08:28:43 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 555.58.02              Driver Version: 555.58.02      CUDA Version: 12.5     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4090        On  |   00000000:41:00.0 Off |                  Off |
| 43%   49C    P8             16W /  450W |       2MiB /  24564MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA GeForce RTX 2080 ...    On  |   00000000:61:00.0 Off |                  N/A |
|  0%   54C    P8             35W /  250W |       1MiB /   8192MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

Fixes: #293857
Fixes: #322400

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

---

Add a 👍 reaction to pull requests you find important.

[SomeoneSerge]

@ofborg build nvidia-docker

[ereslibre]

@SomeoneSerge: is this good to merge on its current status?

[github-actions]

Backport failed for release-24.05, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin release-24.05
git worktree add -d .worktree/backport-331071-to-release-24.05 origin/release-24.05
cd .worktree/backport-331071-to-release-24.05
git switch --create backport-331071-to-release-24.05
git cherry-pick -x df2df4c3a61bb110694e00cde38d871c7761bc08 f7b4d57421d67baf096e0b46168699e774067812 058e8f5ef11cd5291cf03aa4a886e475cf7bdd71