Requesting "None" GPU Accelerator on NERC OpenShift AI (RHOAI) allocates all 4 GPUs

[Milstein]

Motivation

Using RHOAI user can use GPU resource on their workbench Jupyter Lab setup. There is option to request either GPU based on their Allocation Quota specified by "OpenShift Request on GPU Quota" OpenShift allocation quota attribute on their ColdFront Allocation or None for their workload as shown here:

But when selected None, that allow user to use all available GPUs without getting billed based on their usage:

Completion Criteria

When set to GPU "None" option then the workbench should have set:

  limits:
    ...
    nvidia.com/gpu: "0"
  requests:
    ...
    nvidia.com/gpu: "0"

With a resource request for nvidia.com/gpu: 0 this environment variable should be set automatically.

Also, when setting the GPU Accelerator to a specific count (Number of accelerators), ensure it aligns with the currently available GPU quota for the user namespace. It's possible that some GPUs are already in use by other workloads.

Description

• First step to resolve the issue

Workaround Ref: NVIDIA/k8s-device-plugin#61

Completion dates

Desired - 2024-08-21
Required - 2024-09-11

[naved001]

But when selected None, that allow user to use all available GPUs without getting billed based on their usage:

I looked at the pod definition, and there was no requests/limits for a gpu device at all. This means that it will not be restricted by the resourcequota either or can be billed for.

After some looking around I came across the following from the nvidia k8s device plugin readme and according to that this maybe the expected behavior.

WARNING: if you don't request GPUs when using the device plugin with NVIDIA images all the GPUs on the machine will be exposed inside your container.

A workaround would be to not schedule pods that don't request GPUs on nodes with GPUs. I am not sure how far we got with that.

[knikolla]

Also, when setting the GPU Accelerator to a specific count (Number of accelerators), ensure it aligns with the currently available GPU quota for the user namespace. It's possible that some GPUs are already in use by other workloads.

If all notebooks for all users continue to be put into the rhods-notebooks namespace there's no way to set limits based only on the user allocation quota.

[joachimweyl]

@naved001 to confirm are you referring to this issue?

[naved001]

@joachimweyl ah, yes.

[msdisme]

Is there an issue to check all existing projects to make sure they are set to 0?

[gagansk]

@gagansk is checking with RHOAI dev team.

[joachimweyl]

@gagansk any update from RHOAI?

[gagansk]

@dystewart has raised the issue with the RHOAI team on 08/28/2024. I am following up with the RHOAI IDE PM (Kezia Cook).

[gagansk]

@Milstein Some folks running internal AI cluster at Red Hat have run into this problem. I am relaying the messages I received here for your information.

accelerator profiles don't do anything WRT admission. If your GPU nodes are untainted, the scheduler can deem them as "free real estate" and then workloads can consume against them even without requesting quota.

Here's the fix: NVIDIA/gpu-operator#421 (comment)

it can be made in the:

apiVersion: nvidia.com/v1
kind: ClusterPolicy
metadata:
  name: gpu-cluster-policy

Object under .spec.toolkit.env

[joachimweyl]

@dystewart would you be the one to implement the fix Gagan found?

[dystewart]

@joachimweyl Yes! I'll test this fix