[Support]: /tmp/cache 100% usage

[sshaikh]

Describe the problem you are having

Over time (around 12 hours) available swap (checked via free -h) steadily decreases till it's exhausted. Memory seems stable.

Version

0.13.2-6476f8a

Frigate config file

mqtt:
  host: mqtt
  user: frigate
  password: "{FRIGATE_MQTT_PASSWORD}"

ffmpeg:
  hwaccel_args: preset-vaapi

record:
  enabled: True
  retain:
    days: 7

snapshots:
  enabled: True

detect:
  enabled: True

objects:
  track:
    - person
    - bicycle
    - car
    - motocycle
    - bird
    - cat
    - dog
    - backpack
    - umbrella
    - suitcase

detectors:
  coral1:
    type: edgetpu
    device: pci:0
  coral2:
    type: edgetpu
    device: pci:1

go2rtc:
  streams:
    camera:
      - rtsp://cctv:{FRIGATE_RTSP_PASSWORD}@camera:554/h264Preview_01_main
    camera-sub:
      - rtsp://cctv:{FRIGATE_RTSP_PASSWORD}@camera:554/h264Preview_01_sub

cameras:
  cam:
    enabled: true
    ui:
      order: 1
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/camera
          input_args: preset-rtsp-restream
          roles:
            - record
            - audio
        - path: rtsp://127.0.0.1:8554/camera-sub
          input_args: preset-rtsp-restream
          roles:
            - detect
    audio:
      enabled: True
    detect:
      width: 640
      height: 480
      fps: 7
    motion:
      mask:
        - 0,30,200,30,200,0,0,0
    zones:
      driveway-zone:
        coordinates: 0,480,640,480,640,319,467,306,0,305
        filters:
          person:
            min_area: 3600
    record:
      events:
        required_zones:
          - driveway-zone
    snapshots:
      required_zones:
        - driveway-zone

Relevant log output

n/a

FFprobe output from your camera

Input #0, rtsp, from 'rtsp://camera:554/h264Preview_01_main':
  Metadata:
    title           : Session streamed by "preview"
  Duration: N/A, start: 0.000125, bitrate: N/A
  Stream #0:0: Video: h264 (High), yuv420p(progressive), 2560x1920, 15 fps, 120 tbr, 90k tbn
  Stream #0:1: Audio: aac (LC), 16000 Hz, mono, fltp

Frigate stats

No response

Operating system

Debian

Install method

Docker Compose

Coral version

M.2

Network connection

Wired

Camera make and model

Reolink

Any other information that may be helpful

Config is truncated - running eight cameras of various makes and models but with similar settings.

Box has 16GB of ram. shm is set to 200bm, tmpfs to 1 000 000 000. Only frigate is running and swap is freed on frigate restart.

[NickM-27]

will need info from top, the system page, etc. to know what processes are using memory.

[sshaikh]

How can I provide this information?

When sorting by memory, the first 35 or so entries in htop are python3 -u -m frigate, then 4 frigate.recording_manager, then around 24 ffmpegs. Each camera has around 20 frigate.process:camera.

[sshaikh]

Isn't it 1GB? If not how can I measure how much space it is using?

[NickM-27]

that's the maximum, not how much is always being used. Run du -H or df -H /tmp/cache/* in the container

[sshaikh]

$ docker exec frigate df -H /tmp/cache
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           1.1G   13M  988M   2% /tmp/cache

Although I think something is a bit odd after having restarted my box. Swap is reading 0% usage at the moment.

$ free -h
               total        used        free      shared  buff/cache   available
Mem:            15Gi       2.1Gi       3.1Gi        25Mi        10Gi        12Gi
Swap:          979Mi          0B       979Mi

I'll leave it overnight and report any change.

[sshaikh]

Swap is back up, but little change with /tmp/cache:

$ docker exec frigate df -H /tmp/cache
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           1.1G   16M  985M   2% /tmp/cache

$ free -h
               total        used        free      shared  buff/cache   available
Mem:            15Gi       1.8Gi       159Mi        25Mi        13Gi        13Gi
Swap:          979Mi       841Mi       138Mi

[sshaikh]

I used the following from https://www.cyberciti.biz/faq/linux-which-process-is-using-swap/ to list swap usage:

for file in /proc/*/status
do 
  awk '/VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file 
done

The big hitters seem to be:

python3 135856 kB
nginx 4092 kB
nginx 4564 kB
nginx 4564 kB
nginx 4564 kB
nginx 4564 kB
nginx 4564 kB
nginx 4564 kB
nginx 4564 kB
nginx 4564 kB
kworker/15:0H-events_highpri
nginx 4564 kB
nginx 4564 kB
nginx 4564 kB
nginx 4564 kB
nginx 4564 kB
nginx 4564 kB
nginx 4564 kB
nginx 4564 kB
nginx 4236 kB
frigate.logger 60036 kB
frigate.recordi 23096 kB
python3 4912 kB
frigate.detecto 71072 kB
frigate.detecto 71036 kB
frigate.output 74732 kB
ffmpeg 7008 kB
ffmpeg 9052 kB
ffmpeg 9028 kB
ffmpeg 11076 kB
ffmpeg 9052 kB
frigate.process 32016 kB
ffmpeg 9036 kB
frigate.process 72544 kB
frigate.process 70512 kB
frigate.process 72768 kB
frigate.process 71392 kB
frigate.process 69812 kB
frigate.process 70072 kB
frigate.capture 75600 kB
frigate.capture 75696 kB
frigate.capture 75704 kB
ffmpeg 9048 kB
ffmpeg 26688 kB
frigate.capture 75680 kB
ffmpeg 19224 kB
ffmpeg 11128 kB
ffmpeg 9048 kB
frigate.capture 75688 kB
ffmpeg 24196 kB
ffmpeg 10920 kB
frigate.capture 75568 kB
ffmpeg 26824 kB
ffmpeg 9052 kB
frigate.capture 75568 kB
ffmpeg 8920 kB
ffmpeg 26036 kB
ffmpeg 11608 kB
frigate.audio_m 72916 kB
ffmpeg 63988 kB
ffmpeg 9556 kB
ffmpeg 8896 kB
ffmpeg 11716 kB
ffmpeg 11272 kB
ffmpeg 13164 kB

[sshaikh]

Setting swappiness to 10 seems to have helped, so it looks like this is a Debian "issue" rather than Frigate. I'll probably switch it back and adjust my monitoring to stop alerting on it instead.

[sshaikh]

No dice I'm afraid. Swap is now 99% used:

$ free -h
               total        used        free      shared  buff/cache   available
Mem:            15Gi       9.8Gi       203Mi       710Mi       6.0Gi       5.2Gi
Swap:          979Mi       979Mi          0B

and:

$ docker exec frigate df -H /tmp/cache
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           1.1G  1.1G     0 100% /tmp/cache

Which I think is causing failures:

2024-03-03 15:36:44.011138292  [2024-03-03 15:36:44] ffmpeg.cam.record           ERROR   : Could not write header for output file #0 (incorrect codec parameters ?): No space left on device
2024-03-03 15:36:44.011234603  [2024-03-03 15:36:44] ffmpeg.cam.record           ERROR   : Error initializing output stream 0:0 -- 
2024-03-03 15:36:44.011290756  [2024-03-03 15:36:44] ffmpeg.cam.record           ERROR   : 
2024-03-03 15:36:44.011373588  [2024-03-03 15:36:44] watchdog.cam                INFO    : Terminating the existing ffmpeg process...
2024-03-03 15:36:44.011415982  [2024-03-03 15:36:44] watchdog.cam                INFO    : Waiting for ffmpeg to exit gracefully...

Apart from regular restarts are there any resolutions to this?

[sshaikh]

The logs in the web ui are truncated. Is there a way to get the full logs via the command line?

[sshaikh]

I've emailed the log (that is left) and directory listing. Hope that's acceptable!

[NickM-27]

Thanks, it does appear that the original cause is missing though based on the logs seems like the recording maintainer may have crashed given that the segments were not deleted in cache

[sshaikh]

I'm now running out of memory - looking at the logs it seemed to have started increasing steadily from midnight local time - which coincides with when the recordings stopped getting deleted. I suspect this is related. It seems to be the python3 -u -m frigate processes.

I'll have to restart the server now I'm afraid - is there anything I can do to anticipate a better log for next time?

[sshaikh]

Just saw these in the log while getting multiple entries in /tmp/cache. Not sure if relevant but object detection seems to be down:

2024-03-20 00:24:23.871229808  Exception in thread detected_frames_processor:
2024-03-20 00:24:23.871246108  Traceback (most recent call last):
2024-03-20 00:24:23.871254344    File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
2024-03-20 00:24:23.873463085      self.run()
2024-03-20 00:24:23.873466221    File "/opt/frigate/frigate/object_processing.py", line 1075, in run
2024-03-20 00:24:23.873850954      camera_state.update(
2024-03-20 00:24:23.873856635    File "/opt/frigate/frigate/object_processing.py", line 636, in update
2024-03-20 00:24:23.874061360      thumb_update, significant_update, autotracker_update = updated_obj.update(
2024-03-20 00:24:23.874066670    File "/opt/frigate/frigate/object_processing.py", line 245, in update
2024-03-20 00:24:23.874173009      if self.obj_data["frame_time"] - self.previous["frame_time"] > 60:
2024-03-20 00:24:23.874199389  KeyError: 'frame_time'
2024-03-20 00:25:25.826666381  [2024-03-20 00:25:25] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for cam1. Keeping the 6 most recent segments out of 7 and discarding the rest...

That last line is repeated every 5 seconds or so for each camera.

[andregoncalvespires]

I have the same problem here. Frigate crashes with high swap usage. I was unable to observe whether the cache is also high because I cannot access the log or interface.

Did you have a solution?