[Bug]: Fails to run/Crashes when disk space runs out - fails to clear space

[doug62]

Checklist

• I have updated to the latest available Frigate version.
• I have cleared the cache of my browser.
• I have tried a different browser to see if it is related to my browser.
• I have tried reproducing the issue in incognito mode to rule out problems with any third party extensions or plugins I have installed.

Describe the problem you are having

While there seems to be a mechanism to clear disk space to to continue recording when disk space runs out - the mechanism fails, this failure causes Frigate to terminate and then fail to launch.

https://github.dev/blakeblackshear/frigate

This file does not have try/catch in the main event loop, seems likely that this is the cause. I use Kubernetes PVC, it is also possible that Frigate disk usage might not be correctly reported to your app as well.

Steps to reproduce

Run App for a few days, when disk fills up, program terminates.

Version

V14.1

In which browser(s) are you experiencing the issue with?

Any

Frigate config file

mqtt:
logger:
  logs:
    frigate.record.maintainer: debug
birdseye:
  mode: motion
record:
  enabled: True
  expire_interval: 7
  retain:
    days: 7
    mode: all
  events:
    pre_capture: 1
    post_capture: 1
    retain:
      default: 7
      mode: motion
  export:
    timelapse_args: "-vf setpts=PTS/60 -r 25"


snapshots:
  enabled: True
  timestamp: True
  retain:
    default: 7

go2rtc:
  streams:
    mouse:
      -X
    lobby:
      - X
    garage-wide:
      -X
    garage:
      - X
    front-door: 
      - X
    front-higha:
      - X
    family:
      - X

cameras:
  mouse:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/mouse
          roles:
            - detect
            - record
    detect:
      width: 1280
      height: 720
      fps: 1
    objects:
      track:
        - mouse
  lobby:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/lobby
          roles:
            - detect
            - record
    detect:
      width: 1920
      height: 1080
      fps: 5
    objects:
      track:
        - person
        - dog
        - cat
    motion:
      threshold: 50
      contour_area: 50
      mask:
        - 0,1080,1920,1080,1829,343,1713,327,1683,688,1552,696,1569,316,50,366
  Garage:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/garage
          roles:
            - detect
            - record
    detect:
      width: 1920
      height: 1080
      fps: 5
    objects:
      track:
        - person
        - dog
        - cat
        - mouse
    motion:
      threshold: 50
      contour_area: 50
  GarageWide:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/garage-wide
          # input_args: preset-rtsp-generic
          roles:
            - detect
            - record
    detect:
      width: 4608
      height: 1728
      fps: 10
    objects:
      track:
        - person
  FrontDoor:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/front-door
          roles:
            - detect
            - record
    detect:
      width: 1920
      height: 1080
      fps: 10
    objects:
      track:
        - person
        - car
      filters:
        car:
          mask: 
            - 1594,853,1920,897,1920,31,307,25,296,402,841,466,1349,605,1432,663
    motion:
      threshold: 50
      contour_area: 50
      mask:
        - 1594,853,1920,897,1920,31,307,25,296,402,841,466,1349,605,1432,663
  FrontHigha:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/front-higha
          roles:
            - detect
            - record
    detect:
      width: 4608
      height: 1728
      fps: 10
    objects:
      track:
        - person
        - car
      filters:
        car:
          mask:
            - 2193,307,2582,299,3103,546,4608,1728,4608,0,3065,0,2186,41                
        person:
          mask: 
            - 2193,307,2582,299,3103,546,4608,1728,4608,0,3065,0,2186,41
    motion:
      threshold: 60
      contour_area: 50
      mask:
        - 2193,307,2582,299,3103,546,4608,1728,4608,0,3065,0,2186,41

  Family:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/family
          roles:
            - detect
            - record
    detect:
      width: 1920
      height: 1080
      fps: 5
    objects:
      track:
        - person
        - dog
        - cat
objects:
  # Optional: list of objects to track from labelmap.txt (default: shown below)
  # track:
  #   - cat
  #   - dog
  #   - car
  #   - person
  #   - mouse
  # Optional: mask to prevent all object types from being detected in certain areas (default: no mask)
  # Checks based on the bottom center of the bounding box of the object.
  # NOTE: This mask is COMBINED with the object type specific mask below
  # mask: 0,0,1000,0,1000,200,0,200
  # Optional: filters to reduce false positives for specific object types
  filters:
    person:
      # Optional: minimum width*height of the bounding box for the detected object (default: 0)
      min_area: 0
      # Optional: maximum width*height of the bounding box for the detected object (default: 24000000)
      max_area: 24000000
      # Optional: minimum score for the object to initiate tracking (default: shown below)
      min_score: 0.5
      # Optional: minimum decimal percentage for tracked object's computed score to be considered a true positive (default: shown below)
      threshold: 0.7
      # Optional: mask to prevent this object type from being detected in certain areas (default: no mask)
      # Checks based on the bottom center of the bounding box of the object
      # mask: 0,0,1000,0,1000,200,0,200
  # Optional: Motion configuration
  # NOTE: Can be overridden at the camera level
motion:
  threshold: 30
  contour_area: 10
  frame_alpha: 0.01
  frame_height: 100
  improve_contrast: True

docker-compose file or Docker CLI command

Running in Kubernetes.... 8CPU, 300G disk

Relevant Frigate log output

Program fails to launch

Relevant go2rtc log output

aaa

Operating system

HassOS

Install method

HassOS Addon

Network connection

Wired

Camera make and model

aaaa

Screenshots of the Frigate UI's System metrics pages

xxx

Any other information that may be helpful

xxx

[NickM-27]

We need to see logs and preferably output of du -h and df -h

[doug62]

I reset the deployment so it will take 9 more days for it to fill and then report the errors - please keep this ticket open, it has been fairly consistent :)