bug: Cortex-cpp continues to have 1 layer offload to CPU while using GPU

[Van-QA]

Describe the bug
When generating responses using a local llm, cortex-cpp still seems to use CPU.
https://discord.com/channels/1107178041848909847/1149558035971321886/1253148982188838954

To Reproduce

1. Install cortex-cpp and the CUDA toolkit locally.
2. Turn on GPU acceler‌‌atio‌n
3. Generate responses using a local llm.
4. Observe high CPU usage.

Expected behavior
Since cortex-cpp is using a local llm and the CUDA toolkit, it should primarily use the GPU for processing and not consume as much CPU.

Desktop

• OS: Linux

Additional context
The logs indicate that 32 out of 33 layers are offloaded to the GPU, but 1 layer is still processed on the CPU. This behavior will be investigated further.

[dan-menlo]

@nguyenhoangthuan99 I am linking this to #1151 as a sub-issue. Please let me know if already solved

[vansangpfiev]

This should be resolved by changing ngl in model.yml

[gabrielle-ong]

Edit: set to the maximum ngl

[dan-menlo]

@hahuyhoang411 @gabrielle-ong I recommend we create a separate issue in Models repo, for our model.yaml to include num_layers:

• Allow for user to use the slider in Jan to offload to GPU (i.e. "Max")
• Allow for simple checking to prevent int out of bounds etc

This would solve for the issue where people are unsure how many layers are in a model, which results in slow inference due to layers being left on CPU instead of being fully offloaded to GPU

[hahuyhoang411]

Ah yes in the model.yml for each member in a family I always include the max value for ngl E.g. Qwen2.5

https://huggingface.co/cortexso/qwen2.5/blob/main/model.yml

[gabrielle-ong]

Putting in investigating first - cc @imtuyethan / @louis-jan to update what is needed for Jan after Cortex's hardware API

My naive understanding, please cmiiw / add on

• Pending Cortex hardware detection api: feat: Hardware API #1593
• Cortex hardware detection API detects the GPU, layers to offset
• Cortex reads model.yaml for max ngl for a model
• [UI] Jan slider in engine parameters: sets ngl < max ngl (model.yaml) && ngl < detected hardware layers

[gabrielle-ong]

Marking as complete since ngl can be configured on model.yaml
Linking to separate story #1108 for Cortex then Jan to recommend ngl based on hardware