[Support]: Crash due to out of memory on Unraid since 0.13 #10199
-
Describe the problem you are havingEver since 0.13, I have been experience crashes due to high memory use in unraid. I cannot seem to pin point exactly what sets it off, as it has happened a few times to day when changes were made (most recently, birdseye was not functioning even when enabled, and it occurred when I disabled and reenabled it), though in the past weeks it has also sporadically occurred with no user input. Suddenly the container will RAPIDLY use memory, were talking 128 GB in a few seconds. I brought my crashing issues up on the Unraid forums and worked through several steps over the past few weeks to ensure there was not a hardware issue, yet it never sprang up again until I started using Frigate again. Version0.13.2-6476F8A Frigate config filemqtt:
host: 192.168.0.12
port: 1883
topic_prefix: frigate
client_id: frigate
user: mqtt
password: "!hponhftT13"
stats_interval: 60
detectors:
coral:
type: edgetpu
device: usb
ffmpeg:
hwaccel_args: preset-nvidia-h264
cameras:
doorbell:
ffmpeg:
inputs:
- path: rtsp://admin:[email protected]:554/h264Preview_01_sub
roles:
- detect
- path: rtsp://0.0.0.0:8554/doorbell
input_args: preset-rtsp-restream
roles:
- record
detect:
width: 640
height: 480
fps: 7
record:
enabled: true
retain:
days: 14
motion:
mask:
- 422,0,421,26,208,29,207,0
- 222,112,293,45,436,51,453,123,442,279,218,302
objects:
filters:
person:
mask:
- 301,0,0,0,0,480,230,480,229,100,265,99
- 438,40,640,0,640,480,501,480,438,437
- 234,102,305,21,440,52,445,187,228,185
snapshots:
enabled: true
backyard:
ffmpeg:
inputs:
- path: rtsp://admin:[email protected]:554/h264Preview_01_sub
roles:
- detect
- path: rtsp://0.0.0.0:8554/backyard
input_args: preset-rtsp-restream
roles:
- record
detect:
width: 896
height: 512
fps: 8
motion:
mask:
- 231,36,649,40,650,0,267,0,236,0
- 456,0,460,217,374,258,347,0
- 126,0,203,273,309,222,263,0
- 587,0,578,138,508,167,509,0
- 896,0,896,114,800,49,811,0
- 114,41,174,49,174,84,38,80,39,43
record:
enabled: true
retain:
days: 14
snapshots:
enabled: true
onvif:
host: 192.168.5.2
user: admin
password: BW5ynKNVLDhQyS
zones:
zone_0:
coordinates: 801,77,896,124,896,512,0,512,0,0
backzoom:
ffmpeg:
inputs:
- path: rtsp://0.0.0.0:8554/backzoom
input_args: preset-rtsp-restream
roles:
- record
record:
enabled: true
retain:
days: 0
mode: all
events:
retain:
default: 14
mode: motion
required_zones:
- zone_0
snapshots:
enabled: false
motion:
mask:
- 1259,0,1263,56,617,56,620,0
- 356,78,361,166,54,159,59,68
zones:
zone_0:
coordinates: 1920,0,1920,1080,0,1080,0,0
garage:
ffmpeg:
inputs:
- path: rtsp://0.0.0.0:8554/garage
input_args: preset-rtsp-restream
roles:
- record
- detect
detect:
width: 1920
height: 1080
fps: 5
record:
enabled: true
retain:
days: 7
mode: motion
events:
retain:
default: 7
mode: active_objects
objects:
person: 14
go2rtc:
streams:
doorbell:
- rtsp://admin:[email protected]:554/h264Preview_01_main
- "ffmpeg:doorbell#audio=opus"
backyard:
- ffmpeg:rtsp://admin:[email protected]:554/h264Preview_01_main#video=h264#audio=opus
backzoom:
- ffmpeg:rtsp://admin:[email protected]:554/h264Preview_02_main#video=h264
garage:
- rtsp://jpfeifer14:[email protected]/live
webrtc:
candidates:
- 192.168.0.11:8555
- stun:8555
birdseye:
enabled: true
restream: true
mode: continuousRelevant log outputError never appears in unraid log, nothing out of ordinary in frigate logFFprobe output from your cameranot camera specificFrigate stats{"cameras":{"backyard":{"audio_dBFS":0.0,"audio_rms":0.0,"camera_fps":8.1,"capture_pid":795,"detection_enabled":1,"detection_fps":2.1,"ffmpeg_pid":800,"pid":775,"process_fps":8.1,"skipped_fps":0.0},"backzoom":{"audio_dBFS":0.0,"audio_rms":0.0,"camera_fps":5.1,"capture_pid":799,"detection_enabled":1,"detection_fps":4.7,"ffmpeg_pid":807,"pid":777,"process_fps":5.0,"skipped_fps":0.0},"doorbell":{"audio_dBFS":0.0,"audio_rms":0.0,"camera_fps":7.2,"capture_pid":791,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":796,"pid":774,"process_fps":7.2,"skipped_fps":0.0},"garage":{"audio_dBFS":0.0,"audio_rms":0.0,"camera_fps":5.1,"capture_pid":803,"detection_enabled":1,"detection_fps":3.7,"ffmpeg_pid":814,"pid":780,"process_fps":5.1,"skipped_fps":0.0}},"cpu_usages":{"1":{"cmdline":"/package/admin/s6/command/s6-svscan -d4 -- /run/service","cpu":"0.0","cpu_average":"0","mem":"0.0"},"15":{"cmdline":"s6-supervise s6-linux-init-shutdownd","cpu":"0.0","cpu_average":"0","mem":"0.0"},"16":{"cmdline":"/package/admin/s6-linux-init/command/s6-linux-init-shutdownd -c /run/s6/basedir -g 3000 -C -B","cpu":"0.0","cpu_average":"0","mem":"0.0"},"24":{"cmdline":"s6-supervise s6rc-oneshot-runner","cpu":"0.0","cpu_average":"0","mem":"0.0"},"25":{"cmdline":"s6-supervise s6rc-fdholder","cpu":"0.0","cpu_average":"0","mem":"0.0"},"26":{"cmdline":"s6-supervise frigate","cpu":"0.0","cpu_average":"0","mem":"0.0"},"27":{"cmdline":"s6-supervise frigate-log","cpu":"0.0","cpu_average":"0","mem":"0.0"},"28":{"cmdline":"s6-supervise go2rtc","cpu":"0.0","cpu_average":"0","mem":"0.0"},"29":{"cmdline":"s6-supervise go2rtc-healthcheck","cpu":"0.0","cpu_average":"0","mem":"0.0"},"30":{"cmdline":"s6-supervise go2rtc-log","cpu":"0.0","cpu_average":"0","mem":"0.0"},"31":{"cmdline":"s6-supervise nginx","cpu":"0.0","cpu_average":"0","mem":"0.0"},"32":{"cmdline":"s6-supervise nginx-log","cpu":"0.0","cpu_average":"0","mem":"0.0"},"41":{"cmdline":"/package/admin/s6/command/s6-ipcserverd -1 -- /package/admin/s6/command/s6-ipcserver-access -v0 -E -l0 -i data/rules -- /package/admin/s6/command/s6-sudod -t 30000 -- /package/admin/s6-rc/command/s6-rc-oneshot-run -l ../.. --","cpu":"0.0","cpu_average":"0","mem":"0.0"},"42":{"cmdline":"/package/admin/s6-2.11.3.2/command/s6-fdholderd -1 -i data/rules","cpu":"0.0","cpu_average":"0","mem":"0.0"},"80":{"cmdline":"s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/nginx","cpu":"0.0","cpu_average":"0","mem":"0.0"},"81":{"cmdline":"s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/go2rtc","cpu":"0.0","cpu_average":"0","mem":"0.0"},"82":{"cmdline":"s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/frigate","cpu":"0.0","cpu_average":"0","mem":"0.0"},"89":{"cmdline":"/config/go2rtc -config=/dev/shm/go2rtc.yaml","cpu":"3.7","cpu_average":"3","mem":"0.0"},"96":{"cmdline":"bash ./run.user go2rtc-healthcheck","cpu":"0.0","cpu_average":"0","mem":"0.0"},"104":{"cmdline":"python3 -u -m frigate","cpu":"1.0","cpu_average":"1","mem":"0.2"},"119":{"cmdline":"nginx: master process nginx","cpu":"0.0","cpu_average":"0","mem":"0.0"},"145":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"146":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"147":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"151":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"168":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"195":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"222":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"250":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"285":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"316":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"343":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"373":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"408":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"451":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"491":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"528":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"585":{"cmdline":"nginx: cache manager process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"730":{"cmdline":"frigate.logger ","cpu":"0.3","cpu_average":"0","mem":"0.1"},"752":{"cmdline":"frigate.recording_manager","cpu":"0.0","cpu_average":"0","mem":"0.1"},"761":{"cmdline":"/usr/bin/python3 -c from multiprocessing.resource_tracker import main;main(53)","cpu":"0.0","cpu_average":"0","mem":"0.0"},"762":{"cmdline":"frigate.detector.coral","cpu":"2.2","cpu_average":"2","mem":"0.1"},"764":{"cmdline":"frigate.output ","cpu":"0.6","cpu_average":"0","mem":"0.1"},"770":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 640x480 -i pipe: -f mpegts -s 960x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"771":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 896x512 -i pipe: -f mpegts -s 1260x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"773":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1920x1080 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"774":{"cmdline":"frigate.process:doorbell","cpu":"0.5","cpu_average":"0","mem":"0.1"},"775":{"cmdline":"frigate.process:backyard","cpu":"2.1","cpu_average":"2","mem":"0.1"},"777":{"cmdline":"frigate.process:backzoom","cpu":"3.2","cpu_average":"6","mem":"0.1"},"778":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1920x1080 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"780":{"cmdline":"frigate.process:garage","cpu":"2.7","cpu_average":"2","mem":"0.1"},"791":{"cmdline":"frigate.capture:doorbell","cpu":"0.3","cpu_average":"0","mem":"0.1"},"795":{"cmdline":"frigate.capture:backyard","cpu":"0.5","cpu_average":"0","mem":"0.1"},"796":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.2-6476f8a -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://*:*@192.168.5.3:554/h264Preview_01_sub -r 7 -vf fps=7,scale_cuda=w=640:h=480:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"0.8","cpu_average":"0","mem":"0.1"},"799":{"cmdline":"frigate.capture:backzoom","cpu":"1.3","cpu_average":"1","mem":"0.1"},"800":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.2-6476f8a -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://*:*@192.168.5.2:554/h264Preview_01_sub -r 8 -vf fps=8,scale_cuda=w=896:h=512:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"1.0","cpu_average":"0","mem":"0.1"},"803":{"cmdline":"frigate.capture:garage","cpu":"1.3","cpu_average":"1","mem":"0.1"},"807":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.2-6476f8a -rtsp_transport tcp -timeout 5000000 -i rtsp://0.0.0.0:8554/backzoom -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an /tmp/cache/backzoom@%Y%m%d%H%M%S%z.mp4 -r 5 -vf fps=5,scale_cuda=w=1920:h=1080:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"1.8","cpu_average":"1","mem":"0.1"},"812":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.2-6476f8a -rtsp_transport tcp -timeout 5000000 -i rtsp://0.0.0.0:8554/doorbell -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an /tmp/cache/doorbell@%Y%m%d%H%M%S%z.mp4","cpu":"0.3","cpu_average":"0","mem":"0.0"},"813":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.2-6476f8a -rtsp_transport tcp -timeout 5000000 -i rtsp://0.0.0.0:8554/backyard -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an /tmp/cache/backyard@%Y%m%d%H%M%S%z.mp4","cpu":"0.3","cpu_average":"0","mem":"0.0"},"814":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.2-6476f8a -rtsp_transport tcp -timeout 5000000 -i rtsp://0.0.0.0:8554/garage -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an /tmp/cache/garage@%Y%m%d%H%M%S%z.mp4 -r 5 -vf fps=5,scale_cuda=w=1920:h=1080:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"1.4","cpu_average":"1","mem":"0.1"},"6331":{"cmdline":"sleep 30s","cpu":"0.0","cpu_average":"0","mem":"0.0"}},"detection_fps":10.5,"detectors":{"coral":{"detection_start":0.0,"inference_speed":7.43,"pid":762}},"gpu_usages":{"NVIDIA GeForce GTX 1660 SUPER":{"dec":"4.0%","enc":"0.0%","gpu":"1.0%","mem":"12.48%"}},"processes":{"go2rtc":{"pid":89},"logger":{"pid":730},"recording":{"pid":752}},"service":{"last_updated":1709414443,"latest_version":"0.13.2","storage":{"/dev/shm":{"free":246.5,"mount_type":"tmpfs","total":256.0,"used":9.5},"/media/frigate/clips":{"free":2375865.7,"mount_type":"fuse.shfs","total":3815447.8,"used":1433519.1},"/media/frigate/recordings":{"free":2375865.7,"mount_type":"fuse.shfs","total":3815447.8,"used":1433519.1},"/tmp/cache":{"free":935.4,"mount_type":"tmpfs","total":953.7,"used":18.2}},"temperatures":{},"uptime":714,"version":"0.13.2-6476f8a"}}Operating systemUNRAID Install methodDocker Compose Coral versionUSB Network connectionWired Camera make and modelReolink, Wyze Any other information that may be helpfulNo response |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 27 replies
-
|
you should really be using |
Beta Was this translation helpful? Give feedback.
-
|
Trying to get any help from AlexxIT at go2RTC is like puling teeth. I send him a copy of the error produced by HIS PROGRAM and I cannot get a simple response as to what the error even means, just a "fix it yourself" message: "Well. If you have problems only with hardware - this is related to your GPU, drivers and ffmpeg args for your environment. |
Beta Was this translation helpful? Give feedback.
-
|
Was finally able to generate logs decribing the error, although I have no Idea how to interpret them 2024-03-05 13:19:47.726954103 Metadata: |
Beta Was this translation helpful? Give feedback.
-
|
Tried manually specifying dimensions and ended up with this, Im not really familiar enough with ffmpeg to resolve the filter issue Impossible to convert between the formats supported by the filter 'Parsed_scale_cuda_0' and the filter 'auto_scale_0' |
Beta Was this translation helpful? Give feedback.
-
|
The solution to this was to pass "#raw=-vf hwdownload,format=nv12" in the ffmpeg command. |
Beta Was this translation helpful? Give feedback.
-
|
Can you share the working configuration, at leaste the modified part? Thanks |
Beta Was this translation helpful? Give feedback.
-
|
go2rtc: |
Beta Was this translation helpful? Give feedback.
The solution to this was to pass "#raw=-vf hwdownload,format=nv12" in the ffmpeg command.