[Bug]: 0.15 dev build has occasional issues with Intel GPU stats #14479

leccelecce · 2024-10-21T13:56:37Z

leccelecce
Oct 21, 2024

Checklist

I have updated to the latest available Frigate version.
I have cleared the cache of my browser.
I have tried a different browser to see if it is related to my browser.
I have tried reproducing the issue in incognito mode to rule out problems with any third party extensions or plugins I have installed.

Describe the problem you are having

Since upgrading from 0.14.1 to 0.15 dev, I get intermittent issues with Intel iGPU stats not appearing. Log error below.

This is on an Intel i7-10700T with Intel UHD Graphics 630

The System Metrics page still seems to display though the GPU sections seems to disappear, possibly just at first launch.

Steps to reproduce

Run 0.15
Start Frigate

Version

0.15.0-8fcb8e5

In which browser(s) are you experiencing the issue with?

No response

Frigate config file

...
ffmpeg:
  hwaccel_args: preset-intel-qsv-h264
...

docker-compose file or Docker CLI command

services:
  frigate:
    container_name: frigate
    image: ghcr.io/blakeblackshear/frigate:8fcb8e5 # 0.15 dev build 21/10/2024
    privileged: true # this may not be necessary for all setups
    restart: unless-stopped
    shm_size: "512mb"
...
    volumes:
...
    ports:
...
    devices:
      - /dev/bus/usb/002:/dev/bus/usb/002
      - /dev/dri/renderD128:/dev/dri/renderD128

Relevant Frigate log output

2024-10-21 14:40:42.507751696  [2024-10-21 14:40:42] frigate.api.fastapi_app        INFO    : Starting FastAPI app
2024-10-21 14:40:44.438189595  [2024-10-21 14:40:41] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as usb
2024-10-21 14:40:44.442010913  [2024-10-21 14:40:44] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found
2024-10-21 14:41:22.552066445  [2024-10-21 14:41:22] watchdog.camera_test           INFO    : camera_test exceeded fps limit. Exiting ffmpeg...
2024-10-21 14:41:22.552328852  [2024-10-21 14:41:22] watchdog.camera_test           INFO    : Waiting for ffmpeg to exit gracefully...
2024-10-21 14:41:22.720460151  [2024-10-21 14:41:22] frigate.video                  ERROR   : camera_test: Unable to read frames from ffmpeg process.
2024-10-21 14:41:22.720762933  [2024-10-21 14:41:22] frigate.video                  ERROR   : camera_test: Unable to read frames from ffmpeg process.
2024-10-21 14:41:22.720823486  [2024-10-21 14:41:22] frigate.video                  ERROR   : camera_test: ffmpeg process is not running. exiting capture thread...
2024-10-21 14:41:32.728532060  [2024-10-21 14:41:32] watchdog.camera_test           ERROR   : Ffmpeg process crashed unexpectedly for camera_test.
2024-10-21 14:41:32.728610138  [2024-10-21 14:41:32] watchdog.camera_test           ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2024-10-21 14:41:32.728665139  [2024-10-21 14:41:32] ffmpeg.camera_test.detect      ERROR   : libva info: VA-API version 1.22.0
2024-10-21 14:41:32.728722375  [2024-10-21 14:41:32] ffmpeg.camera_test.detect      ERROR   : libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
2024-10-21 14:41:32.728763311  [2024-10-21 14:41:32] ffmpeg.camera_test.detect      ERROR   : libva info: Found init function __vaDriverInit_1_22
2024-10-21 14:41:32.728809788  [2024-10-21 14:41:32] ffmpeg.camera_test.detect      ERROR   : libva info: va_openDriver() returns 0
2024-10-21 14:42:35.074333781  [2024-10-21 14:42:35] asyncio                        ERROR   : Task exception was never retrieved
2024-10-21 14:42:35.074342645  future: <Task finished name='Task-443' coro=<set_gpu_stats() done, defined at /opt/frigate/frigate/stats/util.py:138> exception=ZeroDivisionError('division by zero')>
2024-10-21 14:42:35.074348555  Traceback (most recent call last):
2024-10-21 14:42:35.074351957    File "/opt/frigate/frigate/stats/util.py", line 198, in set_gpu_stats
2024-10-21 14:42:35.074355955      intel_usage = get_intel_gpu_stats()
2024-10-21 14:42:35.074359359    File "/opt/frigate/frigate/util/services.py", line 335, in get_intel_gpu_stats
2024-10-21 14:42:35.074390885      f"{round(((sum(render[key]) / len(render[key])) + (sum(video[key]) / len(video[key]))) / 2, 2)}%"
2024-10-21 14:42:35.074393880  ZeroDivisionError: division by zero
2024-10-21 14:52:42.873467026  [2024-10-21 14:52:42] asyncio                        ERROR   : Task exception was never retrieved
2024-10-21 14:52:42.873472773  future: <Task finished name='Task-4611' coro=<set_gpu_stats() done, defined at /opt/frigate/frigate/stats/util.py:138> exception=ZeroDivisionError('division by zero')>
2024-10-21 14:52:42.873475155  Traceback (most recent call last):
2024-10-21 14:52:42.873477456    File "/opt/frigate/frigate/stats/util.py", line 198, in set_gpu_stats
2024-10-21 14:52:42.873479523      intel_usage = get_intel_gpu_stats()
2024-10-21 14:52:42.873481778    File "/opt/frigate/frigate/util/services.py", line 335, in get_intel_gpu_stats
2024-10-21 14:52:42.873484120      f"{round(((sum(render[key]) / len(render[key])) + (sum(video[key]) / len(video[key]))) / 2, 2)}%"
2024-10-21 14:52:42.873485969  ZeroDivisionError: division by zero

Relevant go2rtc log output

n/a

Operating system

Debian

Install method

Docker Compose

Network connection

Wired

Camera make and model

n/a

Screenshots of the Frigate UI's System metrics pages

Any other information that may be helpful

No response

Answered by leccelecce

Oct 25, 2024

Fixed in 0.15.0-eca504c albeit I do seem to get gaps in the metrics that I don't think were there before.

View full answer

leccelecce · 2024-10-21T14:05:33Z

leccelecce
Oct 21, 2024
Author

Also just saw this new item appear:

0 replies

leccelecce · 2024-10-21T14:13:49Z

leccelecce
Oct 21, 2024
Author

Hmmm, this might be related to one particular cheap Revotech i-706 mini camera I have that doesn't have the best RTSP implementation (camera_test) in the logs above. If I disable it, the error goes away even if I restart several times. I'm wondering if it's upsetting the driver briefly somehow. It was never a problem before though.

0 replies

leccelecce · 2024-10-21T14:15:05Z

leccelecce
Oct 21, 2024
Author

Output of GPU "hardware info":

Vainfo Output
Return Code: 0

Process Output:

vainfo: VA-API version: 1.22 (libva 2.10.0)
vainfo: Driver version: Intel iHD driver for Intel(R) Gen Graphics - 24.3.3 ()
vainfo: Supported profile and entrypoints
VAProfileNone : VAEntrypointVideoProc
VAProfileNone : VAEntrypointStats
VAProfileMPEG2Simple : VAEntrypointVLD
VAProfileMPEG2Simple : VAEntrypointEncSlice
VAProfileMPEG2Main : VAEntrypointVLD
VAProfileMPEG2Main : VAEntrypointEncSlice
VAProfileH264Main : VAEntrypointVLD
VAProfileH264Main : VAEntrypointEncSlice
VAProfileH264Main : VAEntrypointFEI
VAProfileH264Main : VAEntrypointEncSliceLP
VAProfileH264High : VAEntrypointVLD
VAProfileH264High : VAEntrypointEncSlice
VAProfileH264High : VAEntrypointFEI
VAProfileH264High : VAEntrypointEncSliceLP
VAProfileVC1Simple : VAEntrypointVLD
VAProfileVC1Main : VAEntrypointVLD
VAProfileVC1Advanced : VAEntrypointVLD
VAProfileJPEGBaseline : VAEntrypointVLD
VAProfileJPEGBaseline : VAEntrypointEncPicture
VAProfileH264ConstrainedBaseline: VAEntrypointVLD
VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
VAProfileH264ConstrainedBaseline: VAEntrypointFEI
VAProfileH264ConstrainedBaseline: VAEntrypointEncSliceLP
VAProfileVP8Version0_3 : VAEntrypointVLD
VAProfileVP8Version0_3 : VAEntrypointEncSlice
VAProfileHEVCMain : VAEntrypointVLD
VAProfileHEVCMain : VAEntrypointEncSlice
VAProfileHEVCMain : VAEntrypointFEI
VAProfileHEVCMain10 : VAEntrypointVLD
VAProfileHEVCMain10 : VAEntrypointEncSlice
VAProfileVP9Profile0 : VAEntrypointVLD
VAProfileVP9Profile2 : VAEntrypointVLD

0 replies

NickM-27 · 2024-10-21T14:23:31Z

NickM-27
Oct 21, 2024
Collaborator Sponsor

Fixed in #14480

10 replies

leccelecce Oct 25, 2024
Author

Unfortunately no change in latest dev build.

"gpu_usages": {
        "error-gpu": {
            "gpu": "",
            "mem": ""
        }
    },

NickM-27 Oct 25, 2024
Collaborator Sponsor

there should be errors in the logs then, I don't see how that would occur otherwise since it is just using the old method for gpu stats if the new one fails

leccelecce Oct 25, 2024
Author

Nothing in web console logs or Frigate backend logs. I'll debug locally and find out where it's going wrong, just need to get the devcontainer working on Windows 11 again...

leccelecce Oct 25, 2024
Author

Fixed in 0.15.0-eca504c albeit I do seem to get gaps in the metrics that I don't think were there before.

Answer selected by leccelecce

NickM-27 Oct 25, 2024
Collaborator Sponsor

great, that is what I expected it would look like, for now that should be fine

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug]: 0.15 dev build has occasional issues with Intel GPU stats #14479

{{title}}

Replies: 4 comments 10 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

[Bug]: 0.15 dev build has occasional issues with Intel GPU stats #14479

leccelecce Oct 21, 2024

Checklist

Describe the problem you are having

Steps to reproduce

Version

In which browser(s) are you experiencing the issue with?

Frigate config file

docker-compose file or Docker CLI command

Relevant Frigate log output

Relevant go2rtc log output

Operating system

Install method

Network connection

Camera make and model

Screenshots of the Frigate UI's System metrics pages

Any other information that may be helpful

Replies: 4 comments · 10 replies

leccelecce Oct 21, 2024 Author

leccelecce Oct 21, 2024 Author

leccelecce Oct 21, 2024 Author

NickM-27 Oct 21, 2024 Collaborator Sponsor

leccelecce Oct 25, 2024 Author

NickM-27 Oct 25, 2024 Collaborator Sponsor

leccelecce Oct 25, 2024 Author

leccelecce Oct 25, 2024 Author

NickM-27 Oct 25, 2024 Collaborator Sponsor

leccelecce
Oct 21, 2024

Replies: 4 comments 10 replies

leccelecce
Oct 21, 2024
Author

leccelecce
Oct 21, 2024
Author

leccelecce
Oct 21, 2024
Author

NickM-27
Oct 21, 2024
Collaborator Sponsor

leccelecce Oct 25, 2024
Author

NickM-27 Oct 25, 2024
Collaborator Sponsor

leccelecce Oct 25, 2024
Author

leccelecce Oct 25, 2024
Author

NickM-27 Oct 25, 2024
Collaborator Sponsor