[Support]: Possible bug causing metrics to stop running

[aav7fl]

Describe the problem you are having

I think I've run into a bug with Frigate when it is unable to write data to my disk (temporarily). When this happens, my metrics page (and through the API) no long update. e.g. the metrics for my system and cameras seen here /system#general.

I'm running Frigate through Docker on my NAS. Frigate doesn't know this, but the folder it's saving to is an SMB mount from my NAS. When I reboot my NAS, Frigate is temporarily unable to write to the disk. When this happens, the metrics in Frigate no longer update. Everything else on Frigate seems to work like snapshots and recordings. I've only noticed that the metrics can't recover.

You can see in my logs that it's failing to write to /media/frigate/recordings, which is a volume path shared through Docker, which is an SMB folder mounted on my host machine.

The attached logs occurred at the exact moment that my metrics stopped updating in Frigate. If I restart Frigate, the metrics start working again.

Note that the last_updated field in my /api/stats was from a few days ago (the exact moment my logs display an exception). Even though I captured it today.

Version

0.15.0-6b12a45

What browser(s) are you using?

No response

Frigate config file

Not related

Relevant Frigate log output

2024-12-13 10:31:35.035333521  Exception in thread frigate_stats_emitter:
2024-12-13 10:31:35.035400962  Traceback (most recent call last):
2024-12-13 10:31:35.035408957    File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
2024-12-13 10:31:35.051125732      self.run()
2024-12-13 10:31:35.051210162    File "/opt/frigate/frigate/stats/emitter.py", line 79, in run
2024-12-13 10:31:35.051798988      stats = stats_snapshot(
2024-12-13 10:31:35.051861217    File "/opt/frigate/frigate/stats/util.py", line 295, in stats_snapshot
2024-12-13 10:31:35.052659962      storage_stats = shutil.disk_usage(path)
2024-12-13 10:31:35.052716750    File "/usr/lib/python3.9/shutil.py", line 1262, in disk_usage
2024-12-13 10:31:35.054655247      st = os.statvfs(path)
2024-12-13 10:31:35.054727439  OSError: [Errno 112] Host is down: '/media/frigate/recordings'

Relevant go2rtc log output

Unrelated

FFprobe output from your camera

Unrelated

Frigate stats

{
  "cameras": {
    "camera_0": {
      "camera_fps": 5.1,
      "process_fps": 5.1,
      "skipped_fps": 0,
      "detection_fps": 0,
      "detection_enabled": true,
      "pid": 415,
      "capture_pid": 458,
      "ffmpeg_pid": 462,
      "audio_rms": 0,
      "audio_dBFS": 0
    },
    "camera_1": {
      "camera_fps": 5,
      "process_fps": 5.2,
      "skipped_fps": 0,
      "detection_fps": 0.3,
      "detection_enabled": true,
      "pid": 421,
      "capture_pid": 463,
      "ffmpeg_pid": 467,
      "audio_rms": 0,
      "audio_dBFS": 0
    },
    "camera_2": {
      "camera_fps": 5,
      "process_fps": 5.1,
      "skipped_fps": 0,
      "detection_fps": 0,
      "detection_enabled": true,
      "pid": 431,
      "capture_pid": 469,
      "ffmpeg_pid": 472,
      "audio_rms": 0,
      "audio_dBFS": 0
    }
  },
  "detectors": {
    "coral": {
      "inference_speed": 27.43,
      "detection_start": 1734103876.52442,
      "pid": 364
    }
  },
  "detection_fps": 14.1,
  "gpu_usages": {
    "intel-vaapi": {
      "gpu": "1.2%",
      "mem": "-%",
      "clients": {
        "3858": "23.48%",
        "467": "6.18%",
        "457": "5.74%",
        "462": "5.19%",
        "472": "7.96%"
      }
    }
  },
  "cpu_usages": {
    "frigate.full_system": {
      "cpu": "25.8",
      "mem": "13.5"
    },
    "1": {
      "cpu": "0.0",
      "cpu_average": "0",
      "mem": "0.0",
      "cmdline": "/package/admin/s6/command/s6-svscan -d4 -- /run/service"
    },
    "14": {
      "cpu": "0.0",
      "cpu_average": "0",
      "mem": "0.0",
      "cmdline": "s6-supervise s6-linux-init-shutdownd"
    },
    "15": {
      "cpu": "0.0",
      "cpu_average": "0",
      "mem": "0.0",
      "cmdline": "/package/admin/s6-linux-init/command/s6-linux-init-shutdownd -c /run/s6/basedir -g 3000 -C -B"
    },
    "23": {
      "cpu": "0.0",
      "cpu_average": "0",
      "mem": "0.0",
      "cmdline": "s6-supervise frigate-log"
    },
    "24": {
      "cpu": "0.0",
      "cpu_average": "0",
      "mem": "0.0",
      "cmdline": "s6-supervise go2rtc-log"
    },
    "25": {
      "cpu": "0.0",
      "cpu_average": "0",
      "mem": "0.0",
      "cmdline": "s6-supervise go2rtc"
    },
    "26": {
      "cpu": "0.0",
      "cpu_average": "0",
      "mem": "0.0",
      "cmdline": "s6-supervise s6rc-fdholder"
    },
    "27": {
      "cpu": "0.0",
      "cpu_average": "0",
      "mem": "0.0",
      "cmdline": "s6-supervise certsync"
    },
    "28": {
      "cpu": "0.0",
      "cpu_average": "0",
      "mem": "0.0",
      "cmdline": "s6-supervise go2rtc-healthcheck"
    },
    "29": {
      "cpu": "0.0",
      "cpu_average": "0",
      "mem": "0.0",
      "cmdline": "s6-supervise nginx"
    },
    "30": {
      "cpu": "0.0",
      "cpu_average": "0",
      "mem": "0.0",
      "cmdline": "s6-supervise nginx-log"
    },
    "31": {
      "cpu": "0.0",
      "cpu_average": "0",
      "mem": "0.0",
      "cmdline": "s6-supervise certsync-log"
    },
    "32": {
      "cpu": "0.0",
      "cpu_average": "0",
      "mem": "0.0",
      "cmdline": "s6-supervise s6rc-oneshot-runner"
    },
    "33": {
      "cpu": "0.0",
      "cpu_average": "0",
      "mem": "0.0",
      "cmdline": "s6-supervise frigate"
    },
    "43": {
      "cpu": "0.0",
      "cpu_average": "0",
      "mem": "0.0",
      "cmdline": "/package/admin/s6-2.11.3.2/command/s6-fdholderd -1 -i data/rules"
    },
    "48": {
      "cpu": "0.0",
      "cpu_average": "0",
      "mem": "0.0",
      "cmdline": "/package/admin/s6/command/s6-ipcserverd -1 -- /package/admin/s6/command/s6-ipcserver-access -v0 -E -l0 -i data/rules -- /package/admin/s6/command/s6-sudod -t 30000 -- /package/admin/s6-rc/command/s6-rc-oneshot-run -l ../.. --"
    },
    "82": {
      "cpu": "0.0",
      "cpu_average": "0",
      "mem": "0.0",
      "cmdline": "s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/nginx"
    },
    "87": {
      "cpu": "0.0",
      "cpu_average": "0",
      "mem": "0.0",
      "cmdline": "s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/go2rtc"
    },
    "89": {
      "cpu": "0.0",
      "cpu_average": "0",
      "mem": "0.0",
      "cmdline": "s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/certsync"
    },
    "91": {
      "cpu": "0.0",
      "cpu_average": "0",
      "mem": "0.0",
      "cmdline": "s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/frigate"
    },
    "97": {
      "cpu": "17.0",
      "cpu_average": "17",
      "mem": "0.1",
      "cmdline": "/usr/local/go2rtc/bin/go2rtc -config=/dev/shm/go2rtc.yaml"
    },
    "110": {
      "cpu": "4.9",
      "cpu_average": "5",
      "mem": "1.0",
      "cmdline": "python3 -u -m frigate"
    },
    "113": {
      "cpu": "0.0",
      "cpu_average": "0",
      "mem": "0.0",
      "cmdline": "bash ./run.user go2rtc-healthcheck"
    },
    "130": {
      "cpu": "0.0",
      "cpu_average": "0",
      "mem": "0.0",
      "cmdline": "nginx: master process nginx"
    },
    "160": {
      "cpu": "0.0",
      "cpu_average": "0",
      "mem": "0.0",
      "cmdline": "nginx: worker process"
    },
    "161": {
      "cpu": "0.0",
      "cpu_average": "0",
      "mem": "0.0",
      "cmdline": "nginx: worker process"
    },
    "226": {
      "cpu": "0.0",
      "cpu_average": "0",
      "mem": "0.0",
      "cmdline": "nginx: worker process"
    },
    "227": {
      "cpu": "0.0",
      "cpu_average": "0",
      "mem": "0.0",
      "cmdline": "nginx: worker process"
    },
    "228": {
      "cpu": "0.0",
      "cpu_average": "0",
      "mem": "0.0",
      "cmdline": "nginx: cache manager process"
    },
    "304": {
      "cpu": "0.0",
      "cpu_average": "0",
      "mem": "0.0",
      "cmdline": "bash ./run.user certsync"
    },
    "348": {
      "cpu": "2.4",
      "cpu_average": "3",
      "mem": "0.6",
      "cmdline": "frigate.recording_manager"
    },
    "350": {
      "cpu": "1.3",
      "cpu_average": "1",
      "mem": "0.5",
      "cmdline": "frigate.review_segment_manager"
    },
    "364": {
      "cpu": "23.7",
      "cpu_average": "6",
      "mem": "0.6",
      "cmdline": "frigate.detector.coral"
    },
    "383": {
      "cpu": "5.0",
      "cpu_average": "3",
      "mem": "0.6",
      "cmdline": "frigate.output       "
    },
    "415": {
      "cpu": "3.7",
      "cpu_average": "4",
      "mem": "0.7",
      "cmdline": "frigate.process:camera_0"
    },
    "421": {
      "cpu": "3.2",
      "cpu_average": "3",
      "mem": "0.7",
      "cmdline": "frigate.process:camera_1"
    },
    "431": {
      "cpu": "2.9",
      "cpu_average": "3",
      "mem": "0.7",
      "cmdline": "frigate.process:camera_2"
    },
    "458": {
      "cpu": "3.4",
      "cpu_average": "2",
      "mem": "0.6",
      "cmdline": "frigate.capture:camera_0"
    },
    "462": {
      "cpu": "6.2",
      "cpu_average": "5",
      "mem": "0.2",
      "cmdline": "/usr/lib/ffmpeg/7.0/bin/ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel_flags allow_profile_mismatch -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format vaapi -user_agent FFmpeg Frigate/0.15.0-6b12a45 -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/camera_0 -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy /tmp/cache/camera_0@%Y%m%d%H%M%S%z.mp4 -r 5 -vf fps=5,scale_vaapi=w=1280:h=720,hwdownload,format=nv12,eq=gamma=1.4:gamma_weight=0.5 -threads 2 -f rawvideo -pix_fmt yuv420p pipe:"
    },
    "463": {
      "cpu": "2.6",
      "cpu_average": "2",
      "mem": "0.6",
      "cmdline": "frigate.capture:camera_1"
    },
    "467": {
      "cpu": "5.8",
      "cpu_average": "6",
      "mem": "0.3",
      "cmdline": "/usr/lib/ffmpeg/7.0/bin/ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel_flags allow_profile_mismatch -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format vaapi -user_agent FFmpeg Frigate/0.15.0-6b12a45 -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/camera_1 -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy /tmp/cache/camera_1@%Y%m%d%H%M%S%z.mp4 -r 5 -vf fps=5,scale_vaapi=w=1280:h=720,hwdownload,format=nv12,eq=gamma=1.4:gamma_weight=0.5 -threads 2 -f rawvideo -pix_fmt yuv420p pipe:"
    },
    "469": {
      "cpu": "2.5",
      "cpu_average": "2",
      "mem": "0.6",
      "cmdline": "frigate.capture:camera_2"
    },
    "472": {
      "cpu": "5.6",
      "cpu_average": "6",
      "mem": "0.2",
      "cmdline": "/usr/lib/ffmpeg/7.0/bin/ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel_flags allow_profile_mismatch -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format vaapi -user_agent FFmpeg Frigate/0.15.0-6b12a45 -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/camera_2 -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy /tmp/cache/camera_2@%Y%m%d%H%M%S%z.mp4 -r 5 -vf fps=5,scale_vaapi=w=1280:h=720,hwdownload,format=nv12,eq=gamma=1.4:gamma_weight=0.5 -threads 2 -f rawvideo -pix_fmt yuv420p pipe:"
    },
    "535": {
      "cpu": "0.0",
      "cpu_average": "0",
      "mem": "0.0",
      "cmdline": "/usr/lib/ffmpeg/7.0/bin/ffmpeg -threads 1 -f rawvideo -pix_fmt yuv420p -video_size 1280x720 -i pipe: -threads 1 -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:"
    },
    "542": {
      "cpu": "0.0",
      "cpu_average": "0",
      "mem": "0.0",
      "cmdline": "/usr/lib/ffmpeg/7.0/bin/ffmpeg -threads 1 -f rawvideo -pix_fmt yuv420p -video_size 1280x720 -i pipe: -threads 1 -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:"
    },
    "549": {
      "cpu": "0.0",
      "cpu_average": "0",
      "mem": "0.0",
      "cmdline": "/usr/lib/ffmpeg/7.0/bin/ffmpeg -threads 1 -f rawvideo -pix_fmt yuv420p -video_size 1280x720 -i pipe: -threads 1 -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:"
    },
    "556": {
      "cpu": "0.0",
      "cpu_average": "0",
      "mem": "0.0",
      "cmdline": "/usr/lib/ffmpeg/7.0/bin/ffmpeg -threads 1 -f rawvideo -pix_fmt yuv420p -video_size 1280x720 -i pipe: -threads 1 -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:"
    },
    "38330": {
      "cpu": "0.0",
      "cpu_average": "0",
      "mem": "0.0",
      "cmdline": "sleep 60"
    },
    "38382": {
      "cpu": "0.0",
      "cpu_average": "0",
      "mem": "0.0",
      "cmdline": "sleep 30s"
    }
  },
  "service": {
    "uptime": 7762,
    "version": "0.15.0-6b12a45",
    "latest_version": "0.14.1",
    "storage": {
      "/media/frigate/recordings": {
        "total": 16642232.1,
        "used": 12183391.4,
        "free": 4458840.6,
        "mount_type": "cifs"
      },
      "/media/frigate/clips": {
        "total": 16642232.1,
        "used": 12183391.4,
        "free": 4458840.6,
        "mount_type": "cifs"
      },
      "/tmp/cache": {
        "total": 479117.9,
        "used": 153673.4,
        "free": 301035,
        "mount_type": "overlay"
      },
      "/dev/shm": {
        "total": 512,
        "used": 332.9,
        "free": 179.1,
        "mount_type": "tmpfs"
      }
    },
    "temperatures": {

    },
    "last_updated": 1734103877
  },
  "processes": {
    "recording": {
      "pid": 348
    },
    "review_segment": {
      "pid": 350
    },
    "go2rtc": {
      "pid": 97
    }
  }
}

Install method

HassOS Addon

docker-compose file or Docker CLI command

frigate:
  container_name: frigate
  image: ghcr.io/blakeblackshear/frigate:0.15.0-beta3
  restart: always
  labels:
    autoheal: true
  privileged: true
  shm_size: '512mb'
  volumes:
    - /dev/bus/usb:/dev/bus/usb
    - /etc/localtime:/etc/localtime:ro
    - /srv/frigate/config:/config
    - /mnt/smb/frigate:/media/frigate
  devices:
    - /dev/dri/renderD128:/dev/dri/renderD128
  ports:
    - 5000:5000
    - 1935:1935
    - 8554:8554
    - 8555:8555

Object Detector

Coral

Network connection

Wired

Camera make and model

Unrelated

Screenshots of the Frigate UI's System metrics pages

Any other information that may be helpful

I don't think this is related, but Coral is attached to my machine over USB 2.0 and not USB 3.0.

Some time ago, there was a change in the Kernel (or OS) that made the USB 3.0 connection to my Coral TPU lock up my entire machine when being used by Frigate. It's been happening for years and I've given up trying to solve it.

[NickM-27]

yeah looks like we need to catch that