[Support]: Clearing space while there is still plenty of space (10TB of space left)

[andrecastillo]

Checklist

• I have updated to the latest available Frigate version.
• I have cleared the cache of my browser.
• I have tried a different browser to see if it is related to my browser.
• I have tried reproducing the issue in incognito mode to rule out problems with any third party extensions or plugins I have installed.

Describe the problem you are having

Frigate started deleting all my files till it got down to 0 used bytes.

I have a 10 TB drive dedicated to Frigate as my NVR on unRAID, it's not used for anything else, and is in it's pool. I had no issues for weeks, it was fine, I had about 3 tb used. This morning i noticed as I was on my main page for unraid that the used space was dropping, significantly, 3 tb, then 2, then gbs, etc... I realized that files were being deleted. I could not figure out what was wrong. I stopped the container, deleted the sqllite db, resrated, and it happened again.

Then I noticed after another restart it stopped. So I let it do it's thing. It filled to about 9 GB (gigabytes not terrabytes) and then started again, and gave me another one of the errors that it included above. abotu needing to clear space. I couldn't figure out what was going on.

After some GPT, I updated the config from 14 days all and 30 days motion to 7 days all and 15 days motion. Since then, it hasn't happened again, or yet at least.

I did update the config yesterday to make it 14 days and 30 days motion, but after that I hadn't noticed the problem.

The only other thing worth mentionig is that I had to shut down my server today to replace a battery on the UPS. unraid didn't shut down all the way I think, so I had to hard power off. Not sure if that couold have caused this or not.

Steps to reproduce

1. Leave frigate running...

...

Version

0.14.1-f4f3cfa

In which browser(s) are you experiencing the issue with?

not a browser issue

Frigate config file

mqtt:
enabled: false

detectors:
coral:
type: edgetpu
device: usb

audio:
enabled: true

detect:
enabled: true

record:
enabled: true
retain:
days: 14
mode: all
events:
retain:
default: 30
mode: motion

cameras:
outdoor-garage-ne:
enabled: true
ffmpeg:
inputs:
- path: rtsp://admin:[password]@[ip-prefix].10/live
roles:
- audio
- record
- path: rtsp://admin:[password]@[ip-prefix].10/cam/realmonitor?channel=1&subtype=1
roles:
- detect

outdoor-frontdoor-doorbell:
enabled: true
ffmpeg:
inputs:
- path: rtsp://admin:[password]@[ip-prefix].11/Preview_01_main
roles:
- audio
- record
- path: rtsp://admin:[password]@[ip-prefix].11/Preview_01_sub
roles:
- detect

indoor-livingroom:
enabled: true
ffmpeg:
inputs:
- path: rtsp://admin:[password]@[ip-prefix].12/live
roles:
- audio
- record
- path: rtsp://admin:[password]@[ip-prefix].12/cam/realmonitor?channel=1&subtype=1
roles:
- detect

indoor-garage:
enabled: true
ffmpeg:
inputs:
- path: rtsp://admin:[password]@[ip-prefix].13/live
roles:
- audio
- record
- path: rtsp://admin:[password]@[ip-prefix].13/cam/realmonitor?channel=1&subtype=1
roles:
- detect

indoor-backdoor:
enabled: true
ffmpeg:
inputs:
- path: rtsp://admin:[password]@[ip-prefix].14/live
roles:
- audio
- record
- path: rtsp://admin:[password]@[ip-prefix].14/cam/realmonitor?channel=1&subtype=1
roles:
- detect

indoor-frontdoor:
enabled: true
ffmpeg:
inputs:
- path: rtsp://admin:[password]@[ip-prefix].15/live
roles:
- audio
- record
- path: rtsp://admin:[password]@[ip-prefix].15/cam/realmonitor?channel=1&subtype=1
roles:
- detect

version: 0.14
camera_groups:
Indoor:
order: 1
icon: LuLogIn
cameras:
- indoor-backdoor
- indoor-frontdoor
- indoor-garage
- indoor-livingroom
Outdoor:
order: 2
icon: LuLogOut
cameras:
- outdoor-frontdoor-doorbell
- outdoor-garage-ne

docker-compose file or Docker CLI command

HostConfig": {
"Binds": [
"/mnt/user/appdata/frigate:/config:rw",
"/mnt/user/nvr/frigate:/media/frigate:rw",
"/etc/localtime:/etc/localtime:rw"
],
"ContainerIDFile": "",
"LogConfig": {
"Type": "json-file",
"Config": {
"max-file": "2",
"max-size": "50m"
}
},
"NetworkMode": "br0",
"PortBindings": {},
"RestartPolicy": {
"Name": "unless-stopped",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"ConsoleSize": [
0,
0
],
"CapAdd": null,
"CapDrop": null,
"CgroupnsMode": "private",
"Dns": [],
"DnsOptions": [],
"DnsSearch": [],
"ExtraHosts": null,
"GroupAdd": null,
"IpcMode": "private",
"Cgroup": "",
"Links": null,
"OomScoreAdj": 0,
"PidMode": "",
"Privileged": true,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": [
"label=disable"
],
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 536870912,
"Runtime": "runc",
"Isolation": "",
"CpuShares": 0,
"Memory": 0,
"NanoCpus": 0,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": [],
"BlkioDeviceReadBps": [],
"BlkioDeviceWriteBps": [],
"BlkioDeviceReadIOps": [],
"BlkioDeviceWriteIOps": [],
"CpuPeriod": 0,
"CpuQuota": 0,
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": [
{
"PathOnHost": "/dev/dri",
"PathInContainer": "/dev/dri",
"CgroupPermissions": "rwm"
}
],
"DeviceCgroupRules": null,
"DeviceRequests": null,
"MemoryReservation": 0,
"MemorySwap": 0,
"MemorySwappiness": null,
"OomKillDisable": null,
"PidsLimit": null,
"Ulimits": null,
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0,
"Mounts": [
{
"Type": "tmpfs",
"Target": "/tmp/cache",
"TmpfsOptions": {
"SizeBytes": 1000000000
}
}
],
"MaskedPaths": null,
"ReadonlyPaths": null
},
"GraphDriver": {
"Data": null,
"Name": "btrfs"
},
"Mounts": [
{
"Type": "bind",
"Source": "/mnt/user/appdata/frigate",
"Destination": "/config",
"Mode": "rw",
"RW": true,
"Propagation": "rprivate"
},
{
"Type": "bind",
"Source": "/etc/localtime",
"Destination": "/etc/localtime",
"Mode": "rw",
"RW": true,
"Propagation": "rprivate"
},
{
"Type": "bind",
"Source": "/mnt/user/nvr/frigate",
"Destination": "/media/frigate",
"Mode": "rw",
"RW": true,
"Propagation": "rprivate"
},
{
"Type": "tmpfs",
"Source": "",
"Destination": "/tmp/cache",
"Mode": "",
"RW": true,
"Propagation": ""
}
],
"Config": {
"Hostname": "c5301211e83b",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"ExposedPorts": {
"5000/tcp": {},
"8554/tcp": {},
"8555/tcp": {},
"8555/udp": {}
},
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"LIBVA_DRIVER_NAME=iHD",
"TCP_PORT_5000=5000",
"TCP_PORT_8555=8555",
"TZ=America/New_York",
"HOST_OS=Unraid",
"HOST_CONTAINERNAME=frigate",
"TCP_PORT_8554=8554",
"PLUS_API_KEY=",
"UDP_PORT_8555=8555",
"HOST_HOSTNAME=unraid_server",
"TCP_PORT_8971=8971",
"FRIGATE_RTSP_PASSWORD=apasswordgoeshere",
"PATH=/usr/lib/btbn-ffmpeg/bin:/usr/local/go2rtc/bin:/usr/local/tempio/bin:/usr/local/nginx/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"NVIDIA_VISIBLE_DEVICES=all",
"NVIDIA_DRIVER_CAPABILITIES=compute,video,utility",
"S6_LOGGING_SCRIPT=T 1 n0 s10000000 T",
"S6_CMD_WAIT_FOR_SERVICES_MAXTIME=0"
],
"Cmd": null,
"Healthcheck": {
"Test": [
"CMD-SHELL",
"curl --fail --silent --show-error http://127.0.0.1:5000/api/version || exit 1"
],
"Interval": 15000000000,
"Timeout": 5000000000,
"StartPeriod": 120000000000,
"Retries": 3
},
"Image": "ghcr.io/blakeblackshear/frigate:stable",
"Volumes": null,
"WorkingDir": "/opt/frigate",
"Entrypoint": [
"/init"
],
"OnBuild": null,
"Labels": {
"net.unraid.docker.icon": "https://raw.githubusercontent.com/yayitazale/unraid-templates/main/frigate.png",
"net.unraid.docker.managed": "dockerman",
"net.unraid.docker.webui": "http://[IP]:[PORT:8971]"
}
},
"NetworkSettings": {
"Bridge": "",
"SandboxID": "a856d89bf6602e19afbc82059c6ef6b29d2fd165c9ef20089aff715ab47def9f",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": {},
"SandboxKey": "/var/run/docker/netns/a856d89bf660",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "",
"Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"MacAddress": "",
"Networks": {
"br0": {
"IPAMConfig": {
"IPv4Address": "[ipaddress]"
},
"Links": null,
"Aliases": [
"c5301211e83b"
],
"NetworkID": "e6dfb1280c20ca9b12bb75597ad99ce16c2e570cfbbe0c2bf3f745c85f3d08fa",
"EndpointID": "8b37a1fde2058096f7ab6122a267bc2eebed0c462fe9ac4ce1643c26d1199d79",
"Gateway": "",
"IPAddress": "[ipaddress]",
"IPPrefixLen": 24,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "",
"DriverOpts": null
}
}
}

Relevant Frigate log output

2024-11-13 10:34:32.509056260 [2024-11-13 10:34:32] frigate.storage INFO : Less than 1 hour of recording space left, running storage maintenance...
2024-11-13 10:34:32.760608263 [2024-11-13 10:34:32] frigate.storage ERROR : Could not clear 61871243.220000006 MB, currently 700.6000000000003 MB have been cleared. Retained recordings must be deleted.

Relevant go2rtc log output

2024-11-13 12:12:19.160841687 [INFO] Preparing new go2rtc config...
2024-11-13 12:12:19.377859015 [INFO] Starting go2rtc...
2024-11-13 12:12:19.465203134 12:12:19.465 INF go2rtc platform=linux/amd64 revision=b2399f3 version=1.9.2
2024-11-13 12:12:19.465218596 12:12:19.465 INF config path=/dev/shm/go2rtc.yaml
2024-11-13 12:12:19.465559968 12:12:19.465 INF [rtsp] listen addr=:8554
2024-11-13 12:12:19.465651657 12:12:19.465 INF [webrtc] listen addr=:8555/tcp
2024-11-13 12:12:19.466105485 12:12:19.466 INF [api] listen addr=:1984
2024-11-13 12:12:29.158940825 [INFO] Starting go2rtc healthcheck service...

Operating system

UNRAID

Install method

Docker CLI

Network connection

Wired

Camera make and model

amcrest, reolink

Screenshots of the Frigate UI's System metrics pages

Any other information that may be helpful

I deleted my container this morning on unraid and reinstalled it. I changed it so that the container was using the 10tb directly and not through a share this time, thinking maybe it had something to do with the share. That didn't change anything. Same issue this morning.

2024-11-14 07:24:01.195724786 [2024-11-14 07:24:01] frigate.storage INFO : Less than 1 hour of recording space left, running storage maintenance...
2024-11-14 07:24:01.398152030 [2024-11-14 07:24:01] frigate.storage ERROR : Could not clear 70109745.00999999 MB, currently 963.9999999999998 MB have been cleared. Retained recordings must be deleted.

[NickM-27]

We'll need to see debug logs for frigate.storage when this occurs

[andrecastillo]

Sorry, where can I find the debug logs? Is it just all the logs?

[NickM-27]

https://docs.frigate.video/configuration/advanced#logger

[andrecastillo]

filtered_frigatelog.txt

Log attached.

[NickM-27]

so indoor-garage is having issues, the bandwidth is crazy high. We will need to see some samples of the segments in /media/frigate/recordings for that camera. Don't need the file itself just need to know the duration of the mp4 clip and then the size of the file

[andrecastillo]

indoor-garage recordings.

[andrecastillo]

all others except outdoor-garage and doorbell are the same as indoor-garage.

[NickM-27]

the main urls should be - path: rtsp://admin:[password]@[ip-prefix].14/cam/realmonitor?channel=1&subtype=0

[andrecastillo]

Updated all the main urls, restarted the continer waited a few minutes, and whammo. Same issue. Frigate wont' record past 1 gb even. What's weird is that it's saying it cannot clear 69.70885 TB's (69708850.23 MB) which makes no sense.

debug
2024-11-14 10:25:25
peewee.sqliteq
received query INSERT INTO "recordings" ("id", "camera", "path", "start_time", "end_time", "duration", "motion", "objects", "dBFS", "segment_size", "regions") VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
debug
2024-11-14 10:25:25
peewee
('INSERT INTO "recordings" ("id", "camera", "path", "start_time", "end_time", "duration", "motion", "objects", "dBFS", "segment_size", "regions") VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)', ['1731597920.0-vilvxs', 'indoor-garage', '/media/frigate/recordings/2024-11-14/15/indoor-garage/25.20.mp4', 1731597920.0, 1731597920.000011, 1.1111111111111112e-05, 0, 0, 0, 0.3, 0])
debug
2024-11-14 10:25:25
frigate.storage
Storage cleanup check: 69708850.23 hourly with remaining storage: 9532599.8.
info
2024-11-14 10:25:25
frigate.storage
Less than 1 hour of recording space left, running storage maintenance...
debug
2024-11-14 10:25:25
frigate.storage
Starting storage cleanup.
debug
2024-11-14 10:25:25
peewee
('SELECT "t1"."id", "t1"."start_time", "t1"."end_time", "t1"."segment_size", "t1"."path" FROM "recordings" AS "t1" ORDER BY "t1"."start_time" ASC', [])
debug
2024-11-14 10:25:25
peewee
('SELECT "t1"."start_time", "t1"."end_time" FROM "event" AS "t1" WHERE (("t1"."retain_indefinitely" = ?) AND "t1"."has_clip") ORDER BY "t1"."start_time" ASC', [True])
error
2024-11-14 10:25:26
frigate.storage
Could not clear 69708850.23 MB, currently 754.7000000000003 MB have been cleared. Retained recordings must be deleted.
debug
2024-11-14 10:25:26
peewee
('SELECT "t1"."id", "t1"."path", "t1"."segment_size" FROM "recordings" AS "t1" ORDER BY "t1"."start_time" ASC', [])
debug
2024-11-14 10:25:26
frigate.storage
Expiring 181 recordings
debug
2024-11-14 10:25:26
peewee.sqliteq
received query DELETE FROM "recordings" WHERE ("recordings"."id" IN (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?))
debug
2024-11-14 10:25:26
peewee
('DELETE FROM "recordings" WHERE ("recordings"."id" IN (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?,

[NickM-27]

it's saying that because the camera data is corrupted, so it is reporting files sizes for very small amount of time which is meaning the average bitrate is very high. We will want to see the settings for that camera in the cameras webui

[andrecastillo]

Screenshots of video and audio settings for the indoor garage cam.

[andrecastillo]

Quick update. I've removed the camera mentioned above as having a super high bitrate and since then it seems to eb working fine. I still don't understand why though. All my cameras are configured the same.

[andrecastillo]

Added the camera back in last night and the problem started again. I'm not sure what to do.

[NickM-27]

perhaps try using go2rtc for that camera