bug: Inference with NVIDIA GPU stop working after resuming from sleep on Linux

[barbicane]

Describe the bug
After resuming from sleep, inference with nvidia GPU doesn't work until restarting the system.

Steps to reproduce
Steps to reproduce the behavior:

1. Setup GPU Acceleration.
2. Do some chat.
3. Suspend system.
4. Resume system.
5. Model (Trinity in my case) isn't loaded in NVRam anymore even if Jan is closed/reopened until the system is restarted.

Expected behavior
the model is loaded again in NVRam and GPU is used.

Environment details

• Operating System: Linux - ZorinOS 17
• Jan Version: 0.4.9
• Processor: AMD Ryzen 7 7745HX with Radeon Graphics
• RAM: 16GB
• Graphic Card: Nvidia RTX 4070m

[Van-QA]

It seems like there is an issue where if a user does not stop the model, the system goes into a "sleep" state. When the system is turned back on, the nitro process is killed.
However, the app still indicates that the model is starting. This leads to the error where the model is stuck in a starting state.

[imtuyethan]

Hmm what is the decision for this case? @Van-QA

It seems like there is an issue where if a user does not stop the model, the system goes into a "sleep" state. When the system is turned back on, the nitro process is killed. However, the app still indicates that the model is starting. This leads to the error where the model is stuck in a starting state.

[hiento09]

I think this bug needs to be handled in the code and cannot be resolved from the config in CI. What I understand here is that the app needs a mechanism to detect the state of the machine, save the app's state to local storage, and add logic to restore the app's state when the machine's state changes, such as transitioning from sleep to start. For example, to check the machine's state, you can use https://www.electronjs.org/docs/latest/api/power-monitor. cc @louis-jan

[hiento09]

Added @namchuai and @vansangpfiev for state handling in cortex.cpp

[hiento09]

cc @dan-homebrew @0xSage

[dan-menlo]

Ok - I am tagging this to @louis-jan @namchuai @vansangpfiev.

• We should be able to handle system sleep state
• Does this happen in Windows or Mac?

[namchuai]

Hmm, for cortex.cpp, I haven't experienced this issue. However, will do the retesting.

[dan-menlo]

I am marking this for Sprint 25, as I would like to stabilize our Hardware and Engine APIs first.

[gabrielle-ong]

May be linked to #1741