[Bug]: 14.0b2 - No Space Left on Device - Cleanup doesnt seem to be running (well)

[vertex-github]

Describe the problem you are having

Im seeing these errors in the logs:
2024-06-15 09:39:50.611503449 [2024-06-15 09:39:50] ffmpeg.outdoor_front.record ERROR : Could not write header for output file #0 (incorrect codec parameters ?): No space left on device

Ok, so I figured that would be an out of disk. However, the storage pool is currently at 45% usage - 2TB free. So there's something odd here.
As an enhancement it would be helpful if that log line above also included the location/path of the directory it encountered the error on.
My "local" drive to the container has 20GB free, the NFS mount has 2TB free so something doesnt make sense here.

It also seems that the periodic cleanup process doesnt seem to be running / running-effectively when storage constraints get tight. It would be a really nice feature if I could set a usage threshold (say 80% to mirror what ZFS storage prefers) where Frigate would kick in the cleanup operation. Even in 0.12/0.13 I had to go in occasionally and manually delete old recordings - it seems some directories would get orphaned and then simply ignored in the subsequent cleanup runs - eg. a few days ago I deleted a handful of recording directories from March and April 2024. Ive got retain recordings set to 7d, and 7d on events (which I changed to 5d last night - reflected in config below). On every restart Id expect Frigate to scan everything in its recording store and cleanup old stuff up.

Fully understand this is beta.

Steps to reproduce

While the new UI in 0.14 is really nice (thanks for that), the general stability of the platform seems significantly lower.
Reviewing the "Review" web page often results in black squares for 10s of seconds on end before anything loads (on my cell phone I couldnt get a page loaded this morning for review). The main page often has "no frames" or white boxes for cameras. The no frames could well be related to this odd no-storage-space issue. This did improve considerably from 0.14b1 to 0.14b2 but Frigate is not as reliable as 0.13 with regards to viewing footage on desktop browser or Android browser (both connecting on local LAN, one wired, one WiFi 5/6).

Version

ghcr.io/blakeblackshear/frigate:0.14.0-beta2 - I cant access this page now because Frigate is unresponsive.

Frigate config file

All paths removed since they have embedded PWDs (still).

# yaml-language-server: $schema=http://frigate_host:5000/api/config/schema.json
#ui:
#  use_experimental: true
mqtt:
  host: 192.168.66.3
  user:
  password:
  topic_prefix: frigate
  client_id: frigate
  port: 1883

detectors:
  coral1:
    type: edgetpu
    device: pci:0

detect:
  stationary:
# num frames to wait before rechecking the object is there
# You cant set 0 here otherwise Frigate wont start
    interval: 5
# num frames to be considered stationary
    threshold: 50

objects:
  track:
    - person
    - dog

ffmpeg:
  output_args:
# -aac re-encodes the audio into AAC
    record: preset-record-generic-audio-aac
#    record: preset-record-generic-audio-copy

birdseye:
  enabled: true
  # 1 is the highest quality, and 31 is the lowest. Lower quality feeds utilize less CPU resources.
  quality: 1
  mode: continuous

record:
  enabled: true
  # Optional: Number of minutes to wait between cleanup runs (default: shown below)
  # This can be used to reduce the frequency of deleting recording segments from disk if you want to minimize i/o
  expire_interval: 15
#  20231205
  retain:
    days: 5
    mode: all
  events:
    # Optional: Number of seconds before the event to include (default: shown below)
    pre_capture: 5
    # Optional: Number of seconds after the event to include (default: shown below)
    post_capture: 10
    objects:
      - person
      - car
      - dog
    # Optional: Restrict recordings to objects that entered any of the listed zones (default: no required zones)
#    required_zones: []
    retain:
      default: 7
      mode: all

snapshots:
  # Optional: Enable writing jpg snapshot to /media/frigate/clips (default: shown below)
  # This value can be set via MQTT and will be updated in startup based on retained value
  enabled: true
  timestamp: true
  bounding_box: true
  crop: false
  required_zones: []
  retain:
    default: 5

motion:
  # Optional: The threshold passed to cv2.threshold to determine if a pixel is different enough to be counted as motion. (default: shown below)
  # Increasing this value will make motion detection less sensitive and decreasing it will make motion detection more sensitive.
  # The value should be between 1 and 255.
  threshold: 25
  # Optional: Minimum size in pixels in the resized motion image that counts as motion (default: 30)
  # Increasing this value will prevent smaller areas of motion from being detected. Decreasing will
  # make motion detection more sensitive to smaller moving objects.
  # As a rule of thumb:
  #  - 15 - high sensitivity
  #  - 30 - medium sensitivity
  #  - 50 - low sensitivity
  contour_area: 30
  # Optional: Alpha value passed to cv2.accumulateWeighted when averaging the motion delta across multiple frames (default: shown below)
  # Higher values mean the current frame impacts the delta a lot, and a single raindrop may register as motion.
  # Too low and a fast moving person wont be detected as motion.
  delta_alpha: 0.2
  # Optional: Alpha value passed to cv2.accumulateWeighted when averaging frames to determine the background (default: shown below)
  # Higher values mean the current frame impacts the average a lot, and a new object will be averaged into the background faster.
  # Low values will cause things like moving shadows to be detected as motion for longer.
  # https://www.geeksforgeeks.org/background-subtraction-in-an-image-using-concept-of-running-average/
  frame_alpha: 0.2
  # Optional: Height of the resized motion frame  (default: 50)
  # This operates as an efficient blur alternative. Higher values will result in more granular motion detection at the expense
  # of higher CPU usage. Lower values result in less CPU, but small changes may not register as motion.
  frame_height: 50
  # Optional: motion mask
  # NOTE: see docs for more detailed info on creating masks
#  mask: 0,900,1080,900,1080,1920,0,1920
  # Optional: improve contrast (default: shown below)
  # Enables dynamic contrast improvement. This should help improve night detections at the cost of making motion detection more sensitive
  # for daytime.
  improve_contrast: true
  # Optional: Delay when updating camera motion through MQTT from ON -> OFF (default: shown below).
  mqtt_off_delay: 30

cameras:
  outdoor3:
    ffmpeg:
      inputs:
          roles:
            - record
          roles:
            - detect
#    record:
#      retain:
#        days: 7
#        mode: motion
    objects:
      track:
        - person
        - dog
    motion:
      mask:
        - 540,476,615,476,610,448,533,450
        - 546,290,562,289,562,275,546,276
  basement:
    ffmpeg:
      inputs:
          roles:
            - record
          roles:
            - detect
#    record:
#      retain:
#        days: 14
#        mode: motion
    motion:
      mask:
        - 103,479,181,480,187,454,102,452
  office2:
    ffmpeg:
      inputs:
          roles:
            - record
          roles:
            - detect
#    record:
#      retain:
#        days: 7
#        mode: all
    motion:
      mask:
        - 104,474,180,472,181,451,98,449
  outdoor_front:
    ffmpeg:
      input_args:
        - -avoid_negative_ts
        - make_zero
        - -analyzeduration
        - 1000M
        - -probesize
        - 1000M
      inputs:
          roles:
            - record
          roles:
            - detect
    objects:
      track:
        - person
        - dog
        - car
        - bicycle
#    record:
#      retain:
#        days: 5
#        mode: motion
    motion:
      mask:
        - 108,473,182,473,184,452,109,452
  driveway:
    ffmpeg:
      input_args:
        - -avoid_negative_ts
        - make_zero
        - -analyzeduration
        - 1000M
        - -probesize
        - 1000M
      output_args:
        record: preset-record-generic-audio-copy
      inputs:
          roles:
            - record
          roles:
            - detect
#    record:
#      retain:
#        days: 7
#        mode: motion
    objects:
      track:
        - person
        - dog
        - car
        - bicycle
    zones:
      car_1:
        coordinates: 447,360,168,360,178,0,364,0
        #coordinates: 321,251,59,337,38,0,194,0
        objects:
          - car
        # filters:
        #   car:
        #     threshold: 0.5
        #     min_score: 0.4
      car_2:
        coordinates: 482,0,640,287,417,360,310,0
        objects:
          - car
        filters:
          car:
            threshold: 0.5
            min_score: 0.3
    motion:
      mask:
        - 100,360,162,360,166,333,98,333
  garage_1:
    ffmpeg:
      input_args:
        - -avoid_negative_ts
        - make_zero
        - -fflags
        - nobuffer+genpts+discardcorrupt
        - -flags
        - low_delay
        - -strict
        - experimental
        - -analyzeduration
        - 1000M
        - -probesize
        - 1000M
        - -rw_timeout
        - '5000000'
      inputs:
        - path:
          roles:
            - record
        - path:
          roles:
            - detect
    detect:
      width: 640
      height: 480
#    record:
#      retain:
#        days: 7
#        mode: motion
    motion:
      mask:
        - 138,469,228,471,232,446,134,445
  garage_2:
    ffmpeg:
      inputs:
          roles:
            - record
          roles:
            - detect
#    record:
#      retain:
#        days: 7
#        mode: motion
    motion:
      mask:
        - 112,472,182,471,182,448,106,450
  garage_3:
    ffmpeg:
      inputs:
          roles:
            - detect
#    record:
#      retain:
#        days: 7
#        mode: motion
    motion:
      mask:
        - 78,480,128,480,130,456,70,459
  hallway:
    ffmpeg:
      inputs:
          roles:
            - detect
#    record:
#      retain:
#        days: 7
#        mode: motion
    motion:
      mask:
        - 111,714,202,713,207,681,110,680
version: 0.14

Relevant log output

n/a

Operating system

Other Linux

Install method

Docker Compose

Network connection

Wired

Camera make and model

Axis, Reolink, Amcrest

Any other information that may be helpful

No response

[NickM-27]

As an enhancement it would be helpful if that log line above also included the location/path of the directory it encountered the error on.

the log is controlled by ffmpeg

in this case this looks like /tmp/cache is full, not the disk itself. Also there have been many fixes for things since beta2 so most likely would be advisable to run newer commits

[vertex-github]

/tmp/cache from inside the container or on the host? Will try a new commit shortly.

[NickM-27]

/tmp/cache inside of the container.

[NickM-27]

On every restart Id expect Frigate to scan everything in its recording store and cleanup old stuff up.

that only happens when you have sync_recordings enabled. This is extremely CPU intensive and not really worth running every time / enabling by default

[vertex-github]

Im curious why its that expensive - its just a dir scan and then a pattern match on the directory names found. If we dont do this, then Frigate gradually uses up all space on the device and runs out of storage space - that doesnt seem like a good outcome. There's a code flow that is allowing old recordings to not be cleaned up.

…

On Sat, Jun 15, 2024 at 10:18 AM 'Nicolas Mowen' via github < ***@***.***> wrote: On every restart Id expect Frigate to scan everything in its recording store and cleanup old stuff up. that only happens when you have sync_recordings enabled. This is extremely CPU intensive and not really worth running every time / enabling by default — Reply to this email directly, view it on GitHub <#11974 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABRJVJIODSHOCXMSJ6K2653ZHREJXAVCNFSM6AAAAABJLXER5OVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TOOBSGA4DK> . You are receiving this because you authored the thread.Message ID: ***@***.*** com>

[NickM-27]

it is not just that, you have to compare with the database to know which files are missing in the DB as well as what is in the DB but no longer on the file system.

Also, we have not had wide reports of recordings not being cleaned up or storage issues, I checked mine and everything is in sync, so it seems to be an edge case / something that is not fully reproducible.

[vertex-github]

Ive got 840MB in there right now. According to df: tmpfs 1953128 1037896 915232 54% /tmp/cache Im looking on the github container page and see several new images released within the last hour, but is there a "latest-dev" branch tag or something I can pull? If not, would it be possible to add such a tag to the latest dev container? "latest-dev-unstable" or something like that?

…

On Sat, Jun 15, 2024 at 10:14 AM 'Nicolas Mowen' via github < ***@***.***> wrote: /tmp/cache inside of the container. — Reply to this email directly, view it on GitHub <#11974 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABRJVJKI5WCRORKDHCEAIBDZHRD57AVCNFSM6AAAAABJLXER5OVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TOOBSGA3TA> . You are receiving this because you authored the thread.Message ID: ***@***.*** com>

[NickM-27]

there was one at one point but it led to too many issues, we are not looking to have a latest dev tag at this time. beta3 will likely be released soon so could also just wait for that

[vertex-github]

Fair enough - can I provide more info on this since (or how can I get more) - I've been running into stale recordings for many months now - this is not a new 0.14 issue. I recall cleaning stuff up from Dec 2023 also a few months back. Its not a lot of stuff - sometimes an occasional day. Last night I cleaned up about 5 consecutive days from April, and a late March directory.

…

On Sat, Jun 15, 2024 at 10:24 AM 'Nicolas Mowen' via github < ***@***.***> wrote: it is not just that, you have to compare with the database to know which files are missing in the DB as well as what is in the DB but no longer on the file system. Also, we have not had wide reports of recordings not being cleaned up or storage issues, I checked mine and everything is in sync, so it seems to be an edge case / something that is not fully reproducible. — Reply to this email directly, view it on GitHub <#11974 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABRJVJMB4V6X37VBWMORKCDZHRFADAVCNFSM6AAAAABJLXER5OVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TOOBSGEZTM> . You are receiving this because you authored the thread.Message ID: ***@***.*** com>

[NickM-27]

did you check how many files are actually in that day though? that would be good to know.

[vertex-github]

I did not - I just cleaned the old directories - sometimes it seems they're full based on how long the rm -rf command takes to execute though. Sometimes it's pretty quick

[vertex-github]

I pulled the latest dev- build from about an hour ago and rebuilt/restarted the container. Still seeing the same black squares on the review page and missing camera feeds on the home page. Reloading the browser page results in a blank main frame - left sidebar is visible. It eventually came up (partially) after 30s + though.

…

On Sat, Jun 15, 2024 at 10:26 AM 'Nicolas Mowen' via github < ***@***.***> wrote: there was one at one point but it led to too many issues, we are not looking to have a latest dev tag at this time. beta3 will likely be released soon so could also just wait for that — Reply to this email directly, view it on GitHub <#11974 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABRJVJOUVBWHEVJMSCX5PJDZHRFITAVCNFSM6AAAAABJLXER5OVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TOOBSGE2TA> . You are receiving this because you authored the thread.Message ID: ***@***.*** com>

[vertex-github]

I see this in the logs - anything to be concerned about:
2024-06-15 10:41:11.821179679 [ERROR] Missing fingerprint from current nginx TLS cert

[NickM-27]

What is your docker compose? This behavior would indicate very slow storage as these files are very small

[nicholasmparker]

Im having the same behavior, and also getting this TLS error

[nicholasmparker]

Also - at first attempt it appears that beta2 does not have the same issue. I'll post more later tonight, wont be able to look for several hours

[NickM-27]

you need to disable tls, it is enabled by default

[vertex-github]

I just switched back to 0.14b2 - latest dev seemed worse than b2 docker container stats: (one desktop browser open to home page) ``` CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS 99520882c6f5 frigate 245.98% 1.615GiB / 15.61GiB 10.34% 711MB / 15.3MB 315MB / 7.04MB 768 ``` ``` frigate: # New ffmpeg borks on hwaccel args when you dont have HW. Older verisons ignored the settings. # image: ghcr.io/blakeblackshear/frigate:dev-4e5a6eb image: ghcr.io/blakeblackshear/frigate:0.14.0-beta2 # image: ghcr.io/blakeblackshear/frigate:0.13.2 container_name: frigate privileged: true restart: unless-stopped environment: TZ: America/New_York FRIGATE_RTSP_PASSWORD: "XXXXXXXXX" shm_size: "1024mb" # update for your cameras based on calculation above devices: # - /dev/bus/usb:/dev/bus/usb # passes the USB Coral, needs to be modified for other versions - /dev/apex_0:/dev/apex_0 # passes a PCIe Coral, follow driver instructions here https://coral.ai/docs/m2/get-started/#2a-on-linux # - /dev/dri/renderD128 # for intel hwaccel, needs to be updated for your hardware - /dev/dri/card0 # for intel hwaccel, needs to be updated for your hardware volumes: - /etc/localtime:/etc/localtime:ro - ./volumes/frigate/config/:/config/ - ./volumes/frigate/storage:/media/frigate - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear target: /tmp/cache tmpfs: # Approx 1.8GB size: 2000000000 ports: - "5000:5000" - "1935:1935" # RTMP feeds ```