epic: Implement Cortex Hardware API for Nvidia

[vansangpfiev]

Implementation for #1165

• Scope to Nvidia first (AMD, Intel, Qualcomm to subsequent sprints)

Tasklist

(Will fill in details when implement each task)

Hardware API

• Hardware Detection
  • Implement platform-specific detection
  • Implement GPU-specific detection
  • Implement hardware detection function #1591
  • Develop separate modules for Windows, Linux, and macOS to gather hardware information
  • Use appropriate system calls and libraries
• Implement CLI for hardware activation
• Implement RAM check for CPU fallback
• Create API for hardware activation (Nvidia) #1603
• Documentation

/engines

• Modify engine initialization
• Implement hardware passing to engines

/model/start

• Handle ngl settings (in /models/start)
• Implement RAM and VRAM detection
• Implement fallback logic

Jan

• System Monitor
• Model Start
• Engines Settings?

Bugs to Address

• bug: Inference with NVIDIA GPU stop working after resuming from sleep on Linux #1459

Related bugs:

• bug: Models loading onto wrong GPU in multi-GPU setup #1468
• bug: some models failed to load if many GPU are selected #1458
• bug: Cortex-cpp continues to have 1 layer offload to CPU while using GPU #1104
• bug: Failed to Load Model with AMD GPU and Vulkan #1525

Out-of-scope

• AMD
• Intel
• Qualcomm/Snapdragon

[vansangpfiev]

Hardware API Documentation

Get hardware information

GET /v1/hardware

Response:

{
  "cpu": {
    "arch": "string",
    "cores": number,
    "model": "string",
    "instructions": ["string"]
  },
  "os": {
    "version": "string",
    "name": "string"
  },
  "ram": {
    "total": number,
    "available": number,
    "type": "string"
  },
  "storage": {
    "total": number,
    "available": number,
    "type": "string"
  },
  "gpus": [
    {
      "model": "string",
      "vram": "string",
      "driver_version": "string"
    }
  ],
  "power": {
    "battery_life": number,
    "charging_status": "string",
    "is_power_saving": boolean
  },
  "monitors": [
    {
      "resolution": "string",
      "refresh_rate": number,
      "resolution":"string"
    }
  ]
}

Hardware Activation

POST /v1/hardware/activate

{
"gpus": [0, 1]
}

[dan-menlo]

Thanks @vansangpfiev. Will we be implementing deactivate this sprint?

[vansangpfiev]

Thanks @vansangpfiev. Will we be implementing deactivate this sprint?

Since we have /activate endpoint, I think it is redundant to add /deactivate.
By default, we activate all the GPUs. We deactivate all GPUs that are not in request for /activate.

[dan-menlo]

A few notes from our quick call:

Hardware Support

We will need to work with multiple hardware providers, but these can be dealt with in separate sprints:

• For Intel, can we detect iGPUs, NPUs and CPUs? (i.e. Lunar Lake)
• For AMD, can we detect iGPUs
• For Qualcomm/ARM, can we detect Adreno etc

ngl settings

• We detect hardware to recommend ngl setting to users
• /models/start API will infer hardware info from database, and then recommend ngl
• This is not part of Hardware API, but /models/start is using hwinfo

[vansangpfiev]

CLI Documentation:

Get hardware information

cortex hardware list --cpu --os --ram --storage --gpu --power --monitors

If no flag is specified, display all hardware information

Activate hardware

cortex hardware activate --gpus [gpu_list]

gpu_list is required, [] means deactivate all GPUs

Start model

cortex start [model_id] --gpus [gpu_list]

--gpus is optional, if not specified use all activated GPUs

Run

cortex run [model_id] --gpus [gpu_list]

--gpus is optional, if not specified use all activated GPUs

[gabrielle-ong]

Nicely done @vansangpfiev! Testing it out now - 2 quick questions:

1. I cant seem to deactivate the GPU to test without GPU -

cortex-nightly hardware activate --gpus []
Invalid GPU index provided.

2. GPU information has Index=1, ID=0 for the same GPU, which is confusing - can we standardize to using Index like the other fields?
   

[vansangpfiev]

Nicely done @vansangpfiev! Testing it out now - 2 quick questions:

1. I cant seem to deactivate the GPU to test without GPU -

cortex-nightly hardware activate --gpus []
Invalid GPU index provided.

2. GPU information has Index=1, ID=0 for the same GPU, which is confusing - can we standardize to using Index like the other fields?
   

Thanks @gabrielle-ong

1. Let me take a look. Would you mind sharing the cortex.log and cortex-cli.log?
2. Sure, let me fix it. Actually, the ID is the GPU ID that nvidia-smi reports, it can be different from #index.

[gabrielle-ong]

Thanks Sang!
2 - I see, understand. then it'll help to make it clear its the nvidia-smi ID through the help command

1- it just takes in the empty array, no error logs.
cortex-cli.log

20241114 06:32:23.404000 UTC 13784 INFO  CUDA Version: 12.4 - utils/system_info_utils.h:141
20241114 06:32:23.404000 UTC 18228 INFO  Will check for new update, time from last check: 2531 seconds - cortex_upd_cmd.cc:127
20241114 06:32:23.404000 UTC 18228 INFO  Engine release path: https://delta.jan.ai/cortex/latest/version.json - cortex_upd_cmd.cc:138
20241114 06:32:23.545000 UTC 18228 INFO  Got the latest release, update to the config file: v1.0.2-235 - cortex_upd_cmd.cc:175

cortex.log:

20241114 05:38:18.970000 UTC 3728 INFO  Origin:  - main.cc:160
20241114 05:38:19.139000 UTC 12684 INFO  Gpu Driver Version: 551.76 - utils/system_info_utils.h:116
20241114 05:38:19.279000 UTC 12684 INFO  CUDA Version: 12.4 - utils/system_info_utils.h:141
20241114 05:38:19.484000 UTC 12684 INFO  CUDA Version: 12.4 - utils/system_info_utils.h:141
20241114 05:38:19.531000 UTC 12684 INFO  Origin:  - main.cc:160
20241114 05:49:51.989000 UTC 7484 INFO  Origin:  - main.cc:160
20241114 05:49:51.989000 UTC 16792 INFO  activate: {
	"gpus" : 
	[
		0
	]
}
 - hardware.cc:38
20241114 05:49:51.989000 UTC 16792 INFO  No hardware activation changes -> No need to update - hardware_service.cc:211
20241114 05:49:51.989000 UTC 16792 INFO  Origin:  - main.cc:160
20241114 05:50:00.401000 UTC 1384 INFO  Origin:  - main.cc:160
20241114 05:50:00.542000 UTC 4276 INFO  Gpu Driver Version: 551.76 - utils/system_info_utils.h:116
20241114 05:50:00.682000 UTC 4276 INFO  CUDA Version: 12.4 - utils/system_info_utils.h:141
20241114 05:50:00.870000 UTC 4276 INFO  CUDA Version: 12.4 - utils/system_info_utils.h:141
20241114 05:50:00.964000 UTC 4276 INFO  Origin:  - main.cc:160
20241114 05:50:12.567000 UTC 17776 INFO  Origin:  - main.cc:160
20241114 05:50:12.567000 UTC 16092 INFO  activate: {
	"gpus" : 
	[
		1
	]
}
 - hardware.cc:38
20241114 05:50:12.567000 UTC 16092 INFO  Origin:  - main.cc:160
20241114 05:50:37.058000 UTC 15972 INFO  Origin:  - main.cc:160
20241114 05:50:37.058000 UTC 11104 INFO  activate: {
	"gpus" : []
}
 - hardware.cc:38
20241114 05:50:37.058000 UTC 11104 INFO  Origin:  - main.cc:160
20241114 06:32:23.404000 UTC 11156 INFO  Origin:  - main.cc:160
20241114 06:32:23.404000 UTC 6656 INFO  activate: {
	"gpus" : []
}
 - hardware.cc:38
20241114 06:32:23.404000 UTC 6656 INFO  Origin:  - main.cc:160

[vansangpfiev]

@gabrielle-ong Can you please try again with nightly 236?

[gabrielle-ong]

Thanks Sang! Successfully activate and deactivated GPUs with CLI and API, marking as complete

Using GPU

Using CPU