[Bug]: /dev/shm fills up and freezes Frigate #12338

rwlove · 2024-07-08T12:55:18Z

rwlove
Jul 8, 2024

Describe the problem you are having

I'm not sure what triggers it, but occasionally /dev/shm fills up and Frigate freezes. I have to kill and restart Frigate to get it back to a working state.

Steps to reproduce

Run Frigate
Problem happens sporadically

Version

0.14.0-beta3

Frigate config file

---
mqtt:
  host: emqx.home.svc.cluster.local
  topic_prefix: frigate
  user: "{FRIGATE_MQTT_USERNAME}"
  password: "{FRIGATE_MQTT_PASSWORD}"

ffmpeg:
  global_args: ["-hide_banner", "-loglevel", "info"]
  hwaccel_args: preset-vaapi

detectors:
  coral:
    type: edgetpu
    device: usb

auth:
  enabled: false

tls:
  enabled: false
  
logger:
  default: info

database:
  path: /data/frigate.db

objects:
  track:
    - person
    - dog
    - car

record:
  enabled: true
  retain:
    days: 182
    mode: all
  events:
    retain:
      default: 365
      mode: active_objects

snapshots:
  enabled: true
  timestamp: true
  bounding_box: true
  crop: false

go2rtc:
  webrtc:
    candidates:
      - 10.10.40.102:1984
      - stun:1984
  streams:
    #backyard:
    #  - rtsp://user:{FRIGATE_AMCREST_RTSP_PASSWORD}@amcrest-backyard:554/cam/realmonitor?channel=1?subtype=1
    #backyard_substream:
    #  - rtsp://user:{FRIGATE_AMCREST_RTSP_PASSWORD}@amcrest-backyard:554/cam/realmonitor?channel=1?subtype=0
    pool:
      - rtsp://user:{FRIGATE_AMCREST_RTSP_PASSWORD}@amcrest-pool:554/cam/realmonitor?channel=1?subtype=0
    pool_substream:
      - rtsp://user:{FRIGATE_AMCREST_RTSP_PASSWORD}@amcrest-pool:554/cam/realmonitor?channel=1?subtype=1
    frontyard:
      - rtsp://user:{FRIGATE_AMCREST_RTSP_PASSWORD}@amcrest-frontyard:554/cam/realmonitor?channel=1?subtype=0
    frontyard_substream:
      - rtsp://user:{FRIGATE_AMCREST_RTSP_PASSWORD}@amcrest-frontyard:554/cam/realmonitor?channel=1?subtype=1
    #office:
    #  - rtsp://user:{FRIGATE_AMCREST_RTSP_PASSWORD}@amcrest-office:554/cam/realmonitor?channel=1?subtype=1
    #office_substream:
    #  - rtsp://user:{FRIGATE_AMCREST_RTSP_PASSWORD}@amcrest-office:554/cam/realmonitor?channel=1?subtype=0
    #turret:
    #  - rtsp://admin:{FRIGATE_AMCREST_RTSP_PASSWORD}@amcrest-turret:554/cam/realmonitor?channel=1?subtype=1
    #turret_substream:
    #  - rtsp://admin:{FRIGATE_AMCREST_RTSP_PASSWORD}@amcrest-turret:554/cam/realmonitor?channel=1?subtype=0
    #bookshelf:
    #  - rtsp://wyze:{FRIGATE_WYZECAM_RTSP_PASSWORD}@wyzecam-2/live
    #familyroom:
    #  - rtsp://wyze:{FRIGATE_WYZECAM_RTSP_PASSWORD}@wyzecam-1/live
    #kitchen:
    #  - rtsp://wyze:{FRIGATE_WYZECAM_RTSP_PASSWORD}@wyzecam-3/live
    #backyard2:
    #  - rtsp://user:{FRIGATE_AMCREST_RTSP_PASSWORD}@loryta-backyard:554/cam/realmonitor?channel=1?subtype=1
    #backyard2_substream:
    #  - rtsp://user:{FRIGATE_AMCREST_RTSP_PASSWORD}@loryta-backyard:554/cam/realmonitor?channel=1?subtype=0
    frontdoor:
      - rtsp://user:{FRIGATE_REOLINK_RTSP_PASSWORD}@reolink-frontdoor:554/h264Preview_01_main
    frontdoor_substream:
      - rtsp://user:{FRIGATE_REOLINK_RTSP_PASSWORD}@reolink-frontdoor:554/h264Preview_01_sub
    bush:
      - rtsp://user:{FRIGATE_REOLINK_RTSP_PASSWORD}@reolink-bush:554/h264Preview_01_main
    bush_substream:
      - rtsp://user:{FRIGATE_REOLINK_RTSP_PASSWORD}@reolink-bush:554/h264Preview_01_sub
    backyard:
      - "ffmpeg:rtsp://user:{FRIGATE_REOLINK_RTSP_PASSWORD}@reolink-backyard:554/h264Preview_01_main#video=h264#audio=copy#audio=opus"
    backyard_substream:
      - rtsp://user:{FRIGATE_REOLINK_RTSP_PASSWORD}@reolink-backyard:554/h264Preview_01_sub

cameras:
  reolink-backyard:
    ffmpeg:
      output_args:
        record: preset-record-generic-audio-copy
      inputs:
        - path: rtsp://127.0.0.1:8554/backyard
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/backyard_substream
          input_args: preset-rtsp-restream
          roles:
            - detect
    detect:
      fps: 7
      width: 1536
      height: 576
    live:
      stream_name: backyard

  reolink-frontdoor:
    ffmpeg:
      output_args:
        record: preset-record-generic-audio-copy
      inputs:
        - path: rtsp://127.0.0.1:8554/frontdoor
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/frontdoor_substream
          input_args: preset-rtsp-restream
          roles:
            - detect
    detect:
      fps: 7
      width: 640
      height: 480
    zones:
      driveway:
        coordinates: 240,224,288,222,323,221,385,220,435,218,468,219,499,222,487,224,424,231,394,235,359,236,315,236,282,239,242,237
      street:
        coordinates: 640,228,640,246,596,237,555,229,512,227,488,226,456,224,397,215,410,207,478,208,566,217
      walkway:
        coordinates: 199,312,277,313,239,277,310,294,360,319,404,341,358,365,249,364,142,352,142,304
    review:
      alerts:
        required_zones:
          - walkway
          - driveway
          - street
      detections:
        required_zones:
          - walkway
          - driveway
          - street
    snapshots:
      required_zones:
        - walkway
        - driveway
        - street
    objects:
      filters:
        person:
          max_area: 35000
          mask:
            - 418,253,417,281,395,291,383,313,396,329,417,349,454,354,521,372,573,383,597,347,588,277,518,248,478,238,448,235
            - 554,240,538,233,540,212,551,210
        dog:
          max_area: 35000
          mask:
            - 418,253,417,281,395,291,383,313,396,329,417,349,454,354,521,372,573,383,597,347,588,277,518,248,478,238,448,235
    live:
      stream_name: frontdoor

  reolink-bush:
    ffmpeg:
      output_args:
        record: preset-record-generic-audio-copy
      inputs:
        - path: rtsp://127.0.0.1:8554/bush
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/bush_substream
          input_args: preset-rtsp-restream
          roles:
            - detect
    detect:
      fps: 7
      width: 640
      height: 480
    zones:
      greenway:
        coordinates: 222,168,124,151,66,148,61,193,94,211,123,226,164,244,206,247,248,266,283,280,302,288,316,297,343,302,354,314,368,319,400,330,444,346,499,362,608,390,640,398,640,247,536,230,388,198
        objects:
          - person
          - car
      pool:
        coordinates: 73,201,33,170,0,165,0,219,0,288,49,249
    review:
      alerts:
        required_zones:
          - greenway
          - pool
      detections:
        required_zones:
          - greenway
          - pool
    snapshots:
      required_zones:
        - greenway
        - pool
    objects:
      filters:
        car:
          mask:
            - 640,480,640,386,574,368,488,339,402,312,319,283,256,260,209,241,164,223,127,212,96,199,72,188,0,180,0,480
    live:
      stream_name: bush

  amcrest-frontyard:
    ffmpeg:
      output_args:
        record: preset-record-generic-audio-copy
      inputs:
        - path: rtsp://127.0.0.1:8554/frontyard
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/frontyard_substream
          input_args: preset-rtsp-restream
          roles:
            - detect
    detect:
      fps: 5
      width: 640
      height: 480
    motion:
      mask:
        - 510,480,640,480,640,329,640,221,609,222,577,256,572,284,581,321,564,334,530,344,502,326,470,324,416,331,382,348,365,394,370,480,422,480
    zones:
      walkway:
        coordinates: 577,366,640,346,640,251,590,265,540,282,460,304,378,326,296,340,259,350,218,351,159,352,108,357,59,334,0,312,0,377,162,417,304,411,326,480,372,480,360,404,510,372
        objects:
          - person
      driveway:
        coordinates: 49,174,93,140,136,115,101,123,64,147,21,181,0,197,0,216
        objects:
          - person
          - car
      street:
        coordinates: 301,56,392,59,469,63,568,78,640,95,640,142,547,123,478,117,415,114,340,114,242,113,151,124,0,168,0,110,57,96,128,79,170,73,227,68
        objects:
          - person
          - car
    review:
      alerts:
        required_zones:
          - walkway
          - driveway
          - street
      detections:
        required_zones:
          - walkway
          - driveway
          - street
    snapshots:
      required_zones:
        - walkway
        - driveway
        - street
    objects:
      filters:
        person:
          mask:
            - 406,314,423,311,419,285,404,286
            - 148,117,177,112,175,80,147,90
            - 358,100,341,102,342,79,352,80
            - 557,123,583,127,583,83,554,82
    live:
      stream_name: frontyard

  amcrest-pool:
    ffmpeg:
      output_args:
        record: preset-record-generic-audio-copy
      inputs:
        - path: rtsp://127.0.0.1:8554/pool
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/pool_substream
          input_args: preset-rtsp-restream
          roles:
            - detect
    detect:
      fps: 5
      width: 640
      height: 480
    objects:
      filters:
        person:
          threshold: 0.75
          mask:
            - 186,207,201,203,204,186,188,185
            - 580,240,563,233,570,209,585,218
            - 516,240,539,240,536,195,507,198
            - 69,227,97,228,97,169,67,172
            - 305,209,331,209,324,142,306,144
    zones:
      pool:
        coordinates: 539,480,640,480,640,188,556,176,479,172,388,168,321,157,257,152,133,147,74,147,0,135,0,245,0,480,124,480,198,480,266,480,357,480,430,480
        objects:
          - person
          - dog
      greenway:
        coordinates: 150,198,306,198,640,240,640,162,447,141,234,131,67,137,68,215
        objects:
          - person
          - car
    review:
      alerts:
        required_zones:
          - pool
      detections:
        required_zones:
          - pool
          - greenway
    snapshots:
      required_zones:
        - pool
    live:
      stream_name: pool

  #amcrest-office:
  #  ffmpeg:
  #    output_args:
  #      record: preset-record-generic-audio-copy
  #    inputs:
  #      - path: rtsp://127.0.0.1:8554/office
  #        input_args: preset-rtsp-restream
  #        roles:
  #          - record
  #      - path: rtsp://127.0.0.1:8554/office_substream
  #        input_args: preset-rtsp-restream
  #        roles:
  #          - detect
  #  detect:
  #    fps: 15
  #    width: 640
  #    height: 480

  #amcrest-backyard:
  #  ffmpeg:
  #    output_args:
  #      record: preset-record-generic-audio-copy
  #    inputs:
  #      - path: rtsp://127.0.0.1:8554/backyard
  #        input_args: preset-rtsp-restream
  #        roles:
  #          - record
  #      - path: rtsp://127.0.0.1:8554/backyard_substream
  #        input_args: preset-rtsp-restream
  #        roles:
  #          - detect
  #  detect:
  #    fps: 15
  #    width: 640
  #    height: 480
  #  zones:
  #    yard:
  #      coordinates: 640,270,640,480,0,480,0,365,0,337,119,326
  #      objects:
  #        - dog
  #  record:
  #    events:
  #      required_zones:
  #        - yard
  #  snapshots:
  #    required_zones:
  #      - yard

  #loryta-backyard:
  #  ffmpeg:
  #    output_args:
  #      record: preset-record-generic-audio-copy
  #    inputs:
  #      - path: rtsp://127.0.0.1:8554/backyard2
  #        input_args: preset-rtsp-restream
  #        roles:
  #          - record
  #      - path: rtsp://127.0.0.1:8554/backyard2_substream
  #        input_args: preset-rtsp-restream
  #        roles:
  #          - detect
  #  detect:
  #    fps: 15
  #    width: 640
  #    height: 480

Relevant log output

2024-07-08 12:26:49.182621867  ::ffff:10.42.5.84 - - [08/Jul/2024:08:26:49 -0400] "GET /api/review?limit=10&severity=alert HTTP/1.1" 200 3 "https://frigate.foo.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" "192.168.1.1"
2024-07-08 12:26:53.813790491  ::ffff:10.42.4.100 - - [08/Jul/2024:08:26:53 -0400] "GET /live/mse/api/ws?src=frontyard HTTP/1.1" 101 14979657 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" "192.168.1.1"
2024-07-08 12:26:53.813827571  ::ffff:10.42.4.100 - - [08/Jul/2024:08:26:53 -0400] "GET /live/mse/api/ws?src=bush HTTP/1.1" 101 14658032 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" "192.168.1.1"
2024-07-08 12:26:53.814106780  ::ffff:10.42.3.107 - - [08/Jul/2024:08:26:53 -0400] "GET /live/mse/api/ws?src=pool HTTP/1.1" 101 15837518 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" "192.168.1.1"
2024-07-08 12:26:53.814204017  ::ffff:10.42.3.107 - - [08/Jul/2024:08:26:53 -0400] "GET /live/mse/api/ws?src=frontdoor HTTP/1.1" 101 15508005 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" "192.168.1.1"
2024-07-08 12:26:53.898958010  ::ffff:10.42.4.100 - - [08/Jul/2024:08:26:53 -0400] "GET /ws HTTP/1.1" 101 5609 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" "192.168.1.1"
2024-07-08 12:28:41.335153233  Fatal Python error: Bus error
2024-07-08 12:28:41.335156961  
2024-07-08 12:28:41.335158653  Thread 0x00007f68eb7fe6c0 (most recent call first):
2024-07-08 12:28:41.335182036    File "/usr/lib/python3.9/threading.py", line 312 in wait
2024-07-08 12:28:41.335255103    File "/usr/lib/python3.9/multiprocessing/queues.py", line 233 in _feed
2024-07-08 12:28:41.335306435    File "/usr/lib/python3.9/threading.py", line 892 in run
2024-07-08 12:28:41.335366018    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
2024-07-08 12:28:41.335439428    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
2024-07-08 12:28:41.335441208  
2024-07-08 12:28:41.335450787  Thread 0x00007f68eaffd6c0 (most recent call first):
2024-07-08 12:28:41.335495214    File "/opt/frigate/frigate/log.py", line 95 in run
2024-07-08 12:28:41.335561677    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
2024-07-08 12:28:41.335624787    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
2024-07-08 12:28:41.335626192  
2024-07-08 12:28:41.335635374  Current thread 0x00007f68ea7fc6c0 (most recent call first):
2024-07-08 12:28:41.335721647    File "/opt/frigate/frigate/video.py", line 119 in capture_frames
2024-07-08 12:28:41.335790453    File "/opt/frigate/frigate/video.py", line 346 in run
2024-07-08 12:28:41.335869880    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
2024-07-08 12:28:41.335929807    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
2024-07-08 12:28:41.335931600  
2024-07-08 12:28:41.335941660  Thread 0x00007f68e9ffb6c0 (most recent call first):
2024-07-08 12:28:41.336028694    File "/usr/lib/python3.9/multiprocessing/synchronize.py", line 261 in wait
2024-07-08 12:28:41.336100219    File "/usr/lib/python3.9/multiprocessing/synchronize.py", line 349 in wait
2024-07-08 12:28:41.336151624    File "/opt/frigate/frigate/video.py", line 195 in run
2024-07-08 12:28:41.336236135    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
2024-07-08 12:28:41.336305345    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
2024-07-08 12:28:41.336306370  
2024-07-08 12:28:41.336315125  Thread 0x00007f68e97fa6c0 (most recent call first):
2024-07-08 12:28:41.336349625    File "/opt/frigate/frigate/log.py", line 95 in run
2024-07-08 12:28:41.336408516    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
2024-07-08 12:28:41.336457601    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
2024-07-08 12:28:41.336466500  
2024-07-08 12:28:41.336467889  Thread 0x00007f695d26e740 (most recent call first):
2024-07-08 12:28:41.336511534    File "/usr/lib/python3.9/threading.py", line 1049 in _wait_for_tstate_lock
2024-07-08 12:28:41.336549397    File "/usr/lib/python3.9/threading.py", line 1033 in join
2024-07-08 12:28:41.336602616    File "/opt/frigate/frigate/video.py", line 383 in capture_camera
2024-07-08 12:28:41.336677096    File "/usr/lib/python3.9/multiprocessing/process.py", line 108 in run
2024-07-08 12:28:41.336743996    File "/usr/lib/python3.9/multiprocessing/process.py", line 315 in _bootstrap
2024-07-08 12:28:41.336809865    File "/usr/lib/python3.9/multiprocessing/popen_fork.py", line 71 in _launch
2024-07-08 12:28:41.336889600    File "/usr/lib/python3.9/multiprocessing/popen_fork.py", line 19 in __init__
2024-07-08 12:28:41.336973856    File "/usr/lib/python3.9/multiprocessing/context.py", line 277 in _Popen
2024-07-08 12:28:41.337032875    File "/usr/lib/python3.9/multiprocessing/context.py", line 224 in _Popen
2024-07-08 12:28:41.337092599    File "/usr/lib/python3.9/multiprocessing/process.py", line 121 in start
2024-07-08 12:28:41.337168602    File "/opt/frigate/frigate/app.py", line 520 in start_camera_capture_processes
2024-07-08 12:28:41.337215274    File "/opt/frigate/frigate/app.py", line 696 in start
s6-log: warning: unable to write to /dev/shm/logs/frigate/current: No space left on device
s6-log: warning: unable to write to /dev/shm/logs/frigate/current: No space left on device
2024-07-08 12:28:45.341041957    File "/opt/frigate/frigate/__main__.py", line 17 in <module>
2024-07-08 12:28:45.341044282    File "/usr/lib/python3.9/runpy.py", line 87 in _run_code
2024-07-08 12:28:45.341045439    File "/usr/lib/python3.9/runpy.py", line 197 in _run_module_as_main
2024-07-08 12:28:45.341046340  Fatal Python error: Bus error
2024-07-08 12:28:45.341047192  
2024-07-08 12:28:45.341048133  Thread 0x00007f68eb7fe6c0 (most recent call first):
2024-07-08 12:28:45.341049069    File "/usr/lib/python3.9/threading.py", line 312 in wait
2024-07-08 12:28:45.341050023    File "/usr/lib/python3.9/multiprocessing/queues.py", line 233 in _feed
2024-07-08 12:28:45.341050959    File "/usr/lib/python3.9/threading.py", line 892 in run
2024-07-08 12:28:45.341052854    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
2024-07-08 12:28:45.341053812    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
2024-07-08 12:28:45.341054588  
2024-07-08 12:28:45.341080797  Thread 0x00007f68eaffd6c0 (most recent call first):
2024-07-08 12:28:45.341081840    File "/opt/frigate/frigate/log.py", line 95 in run
2024-07-08 12:28:45.341082920    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
2024-07-08 12:28:45.341083852    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
2024-07-08 12:28:45.341093584  
2024-07-08 12:28:45.341094615  Current thread 0x00007f68ea7fc6c0 (most recent call first):
2024-07-08 12:28:45.341095552    File "/opt/frigate/frigate/video.py", line 119 in capture_frames
2024-07-08 12:28:45.341096567    File "/opt/frigate/frigate/video.py", line 346 in run
2024-07-08 12:28:45.341097518    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
2024-07-08 12:28:45.341098434    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
2024-07-08 12:28:45.341107659  
2024-07-08 12:28:45.341108643  Thread 0x00007f68e9ffb6c0 (most recent call first):
2024-07-08 12:28:45.341109635    File "/usr/lib/python3.9/multiprocessing/synchronize.py", line 261 in wait
2024-07-08 12:28:45.341110587    File "/usr/lib/python3.9/multiprocessing/synchronize.py", line 349 in wait
2024-07-08 12:28:45.341111636    File "/opt/frigate/frigate/video.py", line 195 in run
2024-07-08 12:28:45.341112572    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
2024-07-08 12:28:45.341113479    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
2024-07-08 12:28:45.341125657  
2024-07-08 12:28:45.341126605  Thread 0x00007f68e97fa6c0 (most recent call first):
2024-07-08 12:28:45.341127479    File "/opt/frigate/frigate/log.py", line 95 in run
2024-07-08 12:28:45.341128456    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
2024-07-08 12:28:45.341129421    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
2024-07-08 12:28:45.341130295  
2024-07-08 12:28:45.341131216  Thread 0x00007f695d26e740 (most recent call first):
2024-07-08 12:28:45.341132183    File "/usr/lib/python3.9/threading.py", line 1049 in _wait_for_tstate_lock
2024-07-08 12:28:45.341133087    File "/usr/lib/python3.9/threading.py", line 1033 in join
2024-07-08 12:28:45.341133999    File "/opt/frigate/frigate/video.py", line 383 in capture_camera
2024-07-08 12:28:45.341134987    File "/usr/lib/python3.9/multiprocessing/process.py", line 108 in run
2024-07-08 12:28:45.341135962    File "/usr/lib/python3.9/multiprocessing/process.py", line 315 in _bootstrap
2024-07-08 12:28:45.341136939    File "/usr/lib/python3.9/multiprocessing/popen_fork.py", line 71 in _launch
2024-07-08 12:28:45.341137932    File "/usr/lib/python3.9/multiprocessing/popen_fork.py", line 19 in __init__
2024-07-08 12:28:45.341151151    File "/usr/lib/python3.9/multiprocessing/context.py", line 277 in _Popen
2024-07-08 12:28:45.341152254    File "/usr/lib/python3.9/multiprocessing/context.py", line 224 in _Popen
2024-07-08 12:28:45.341153224    File "/usr/lib/python3.9/multiprocessing/process.py", line 121 in start
2024-07-08 12:28:45.341154214    File "/opt/frigate/frigate/app.py", line 520 in start_camera_capture_processes
2024-07-08 12:28:45.341155109    File "/opt/frigate/frigate/app.py", line 696 in start
2024-07-08 12:28:45.341156154    File "/opt/frigate/frigate/__main__.py", line 17 in <module>
2024-07-08 12:28:45.341157078    File "/usr/lib/python3.9/runpy.py", line 87 in _run_code
2024-07-08 12:28:45.341157991    File "/usr/lib/python3.9/runpy.py", line 197 in _run_module_as_main
2024-07-08 12:28:45.341158834  Fatal Python error: Bus error
2024-07-08 12:28:45.341159572  
2024-07-08 12:28:45.341160483  Thread 0x00007f68eb7fe6c0 (most recent call first):
2024-07-08 12:28:45.341161446    File "/usr/lib/python3.9/threading.py", line 312 in wait
2024-07-08 12:28:45.341164104    File "/usr/lib/python3.9/multiprocessing/queues.py", line 233 in _feed
2024-07-08 12:28:45.341165033    File "/usr/lib/python3.9/threading.py", line 892 in run
2024-07-08 12:28:45.341165988    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
2024-07-08 12:28:45.341166904    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
2024-07-08 12:28:45.341167678  
2024-07-08 12:28:45.341168582  Thread 0x00007f68eaffd6c0 (most recent call first):
2024-07-08 12:28:45.341169474    File "/opt/frigate/frigate/log.py", line 95 in run
2024-07-08 12:28:45.341170445    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
2024-07-08 12:28:45.341186257    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
2024-07-08 12:28:45.341187094  
2024-07-08 12:28:45.341187992  Current thread 0x00007f68ea7fc6c0 (most recent call first):
2024-07-08 12:28:45.341188925    File "/opt/frigate/frigate/video.py", line 119 in capture_frames
2024-07-08 12:28:45.341189793    File "/opt/frigate/frigate/video.py", line 346 in run
2024-07-08 12:28:45.341190713    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
2024-07-08 12:28:45.341191682    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
2024-07-08 12:28:45.341192434  
2024-07-08 12:28:45.341193349  Thread 0x00007f68e9ffb6c0 (most recent call first):
2024-07-08 12:28:45.341194436    File "/usr/lib/python3.9/multiprocessing/synchronize.py", line 261 in wait
2024-07-08 12:28:45.341195381    File "/usr/lib/python3.9/multiprocessing/synchronize.py", line 349 in wait
2024-07-08 12:28:45.341196261    File "/opt/frigate/frigate/video.py", line 195 in run
2024-07-08 12:28:45.341197195    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
2024-07-08 12:28:45.341198095    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
2024-07-08 12:28:45.341198798  
2024-07-08 12:28:45.341199670  Thread 0x00007f68e97fa6c0 (most recent call first):
2024-07-08 12:28:45.341200549    File "/opt/frigate/frigate/log.py", line 95 in run
2024-07-08 12:28:45.341201474    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
2024-07-08 12:28:45.341202439    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
2024-07-08 12:28:45.341203161  
2024-07-08 12:28:45.341204032  Thread 0x00007f695d26e740 (most recent call first):
2024-07-08 12:28:45.341205018    File "/usr/lib/python3.9/threading.py", line 1049 in _wait_for_tstate_lock
2024-07-08 12:28:45.341205924    File "/usr/lib/python3.9/threading.py", line 1033 in join
2024-07-08 12:28:45.341206854    File "/opt/frigate/frigate/video.py", line 383 in capture_camera
2024-07-08 12:28:45.341207775    File "/usr/lib/python3.9/multiprocessing/process.py", line 108 in run
2024-07-08 12:28:45.341208735    File "/usr/lib/python3.9/multiprocessing/process.py", line 315 in _bootstrap
2024-07-08 12:28:45.341209700    File "/usr/lib/python3.9/multiprocessing/popen_fork.py", line 71 in _launch
2024-07-08 12:28:45.341210649    File "/usr/lib/python3.9/multiprocessing/popen_fork.py", line 19 in __init__
2024-07-08 12:28:45.341228083    File "/usr/lib/python3.9/multiprocessing/context.py", line 277 in _Popen
2024-07-08 12:28:45.341229107    File "/usr/lib/python3.9/multiprocessing/context.py", line 224 in _Popen
2024-07-08 12:28:45.341230054    File "/usr/lib/python3.9/multiprocessing/process.py", line 121 in start
2024-07-08 12:28:45.341231018    File "/opt/frigate/frigate/app.py", line 520 in start_camera_capture_processes
2024-07-08 12:28:45.341231899    File "/opt/frigate/frigate/app.py", line 696 in start
2024-07-08 12:28:45.341232820    File "/opt/frigate/frigate/__main__.py", line 17 in <module>
2024-07-08 12:28:45.341233706    File "/usr/lib/python3.9/runpy.py", line 87 in _run_code
2024-07-08 12:28:45.341234623    File "/usr/lib/python3.9/runpy.py", line 197 in _run_module_as_main
2024-07-08 12:28:45.341235426  Fatal Python error: Bus error
2024-07-08 12:28:45.341236156  
2024-07-08 12:28:45.341237038  Thread 0x00007f68eb7fe6c0 (most recent call first):
2024-07-08 12:28:45.341237924    File "/usr/lib/python3.9/threading.py", line 312 in wait
2024-07-08 12:28:45.341254266    File "/usr/lib/python3.9/multiprocessing/queues.py", line 233 in _feed
2024-07-08 12:28:45.341255227    File "/usr/lib/python3.9/threading.py", line 892 in run
2024-07-08 12:28:45.341256182    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
2024-07-08 12:28:45.341257083    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
2024-07-08 12:28:45.341257813  
2024-07-08 12:28:45.341258674  Thread 0x00007f68eaffd6c0 (most recent call first):
2024-07-08 12:28:45.341259568    File "/opt/frigate/frigate/log.py", line 95 in run
2024-07-08 12:28:45.341260493    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
2024-07-08 12:28:45.341261384    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
2024-07-08 12:28:45.341262099  
2024-07-08 12:28:45.341262989  Current thread 0x00007f68ea7fc6c0 (most recent call first):
2024-07-08 12:28:45.341263903    File "/opt/frigate/frigate/video.py", line 119 in capture_frames
2024-07-08 12:28:45.341264766    File "/opt/frigate/frigate/video.py", line 346 in run
2024-07-08 12:28:45.341265703    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
2024-07-08 12:28:45.341282977    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
2024-07-08 12:28:45.341283754  
2024-07-08 12:28:45.341284662  Thread 0x00007f68e9ffb6c0 (most recent call first):
2024-07-08 12:28:45.341285629    File "/usr/lib/python3.9/multiprocessing/synchronize.py", line 261 in wait
2024-07-08 12:28:45.341286604    File "/usr/lib/python3.9/multiprocessing/synchronize.py", line 349 in wait
2024-07-08 12:28:45.341287478    File "/opt/frigate/frigate/video.py", line 195 in run
2024-07-08 12:28:45.341288421    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
2024-07-08 12:28:45.341289341    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
2024-07-08 12:28:45.341290033  
2024-07-08 12:28:45.341290884  Thread 0x00007f68e97fa6c0 (most recent call first):
2024-07-08 12:28:45.341291708    File "/opt/frigate/frigate/log.py", line 95 in run
2024-07-08 12:28:45.341292634    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
2024-07-08 12:28:45.341293546    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
2024-07-08 12:28:45.341294285  
2024-07-08 12:28:45.341295160  Thread 0x00007f695d26e740 (most recent call first):
2024-07-08 12:28:45.341296088    File "/usr/lib/python3.9/threading.py", line 1049 in _wait_for_tstate_lock
2024-07-08 12:28:45.341319145    File "/usr/lib/python3.9/threading.py", line 1033 in join
2024-07-08 12:28:45.341320163    File "/opt/frigate/frigate/video.py", line 383 in capture_camera
2024-07-08 12:28:45.341321083    File "/usr/lib/python3.9/multiprocessing/process.py", line 108 in run
2024-07-08 12:28:45.341322066    File "/usr/lib/python3.9/multiprocessing/process.py", line 315 in _bootstrap
2024-07-08 12:28:45.341323018    File "/usr/lib/python3.9/multiprocessing/popen_fork.py", line 71 in _launch
2024-07-08 12:28:45.341323968    File "/usr/lib/python3.9/multiprocessing/popen_fork.py", line 19 in __init__
2024-07-08 12:28:45.341324892    File "/usr/lib/python3.9/multiprocessing/context.py", line 277 in _Popen
2024-07-08 12:28:45.341325821    File "/usr/lib/python3.9/multiprocessing/context.py", line 224 in _Popen
2024-07-08 12:28:45.341326758    File "/usr/lib/python3.9/multiprocessing/process.py", line 121 in start
2024-07-08 12:28:45.341327731    File "/opt/frigate/frigate/app.py", line 520 in start_camera_capture_processes
2024-07-08 12:28:45.341328625    File "/opt/frigate/frigate/app.py", line 696 in start
2024-07-08 12:28:45.341343128    File "/opt/frigate/frigate/__main__.py", line 17 in <module>
2024-07-08 12:28:45.341344152    File "/usr/lib/python3.9/runpy.py", line 87 in _run_code
2024-07-08 12:28:45.341345102    File "/usr/lib/python3.9/runpy.py", line 197 in _run_module_as_main
2024-07-08 12:28:45.341345910  Fatal Python error: Bus error
2024-07-08 12:28:45.341346637  
2024-07-08 12:28:45.341347505  Thread 0x00007f68eb7fe6c0 (most recent call first):
2024-07-08 12:28:45.341348405    File "/usr/lib/python3.9/threading.py", line 312 in wait
2024-07-08 12:28:45.341349338    File "/usr/lib/python3.9/multiprocessing/queues.py", line 233 in _feed
2024-07-08 12:28:45.341350246    File "/usr/lib/python3.9/threading.py", line 892 in run
2024-07-08 12:28:45.341351203    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
2024-07-08 12:28:45.341352123    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
2024-07-08 12:28:45.341352866  
2024-07-08 12:28:45.341353728  Thread 0x00007f68eaffd6c0 (most recent call first):
2024-07-08 12:28:45.341354592    File "/opt/frigate/frigate/log.py", line 95 in run
2024-07-08 12:28:45.341355517    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
2024-07-08 12:28:45.341356417    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
2024-07-08 12:28:45.341357166  
2024-07-08 12:28:45.341358066  Current thread 0x00007f68ea7fc6c0 (most recent call first):
2024-07-08 12:28:45.341359005    File "/opt/frigate/frigate/video.py", line 119 in capture_frames
2024-07-08 12:28:45.341359879    File "/opt/frigate/frigate/video.py", line 346 in run
2024-07-08 12:28:45.341360820    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
2024-07-08 12:28:45.341361716    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
2024-07-08 12:28:45.341362441  
2024-07-08 12:28:45.341363313  Thread 0x00007f68e9ffb6c0 (most recent call first):
2024-07-08 12:28:45.341364257    File "/usr/lib/python3.9/multiprocessing/synchronize.py", line 261 in wait
2024-07-08 12:28:45.341365188    File "/usr/lib/python3.9/multiprocessing/synchronize.py", line 349 in wait
2024-07-08 12:28:45.341366090    File "/opt/frigate/frigate/video.py", line 195 in run
2024-07-08 12:28:45.341367020    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
2024-07-08 12:28:45.341367909    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
2024-07-08 12:28:45.341368632  
2024-07-08 12:28:45.341369508  Thread 0x00007f68e97fa6c0 (most recent call first):
2024-07-08 12:28:45.341370367    File "/opt/frigate/frigate/log.py", line 95 in run
2024-07-08 12:28:45.341371269    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
2024-07-08 12:28:45.341372163    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
2024-07-08 12:28:45.341372895  
2024-07-08 12:28:45.341373759  Thread 0x00007f695d26e740 (most recent call first):
2024-07-08 12:28:45.341374717    File "/usr/lib/python3.9/threading.py", line 1049 in _wait_for_tstate_lock
2024-07-08 12:28:45.341375593    File "/usr/lib/python3.9/threading.py", line 1033 in join
2024-07-08 12:28:45.341376489    File "/opt/frigate/frigate/video.py", line 383 in capture_camera
2024-07-08 12:28:45.341377427    File "/usr/lib/python3.9/multiprocessing/process.py", line 108 in run
2024-07-08 12:28:45.341378386    File "/usr/lib/python3.9/multiprocessing/process.py", line 315 in _bootstrap
2024-07-08 12:28:45.341379330    File "/usr/lib/python3.9/multiprocessing/popen_fork.py", line 71 in _launch
2024-07-08 12:28:45.341380299    File "/usr/lib/python3.9/multiprocessing/popen_fork.py", line 19 in __init__
2024-07-08 12:28:45.341381221    File "/usr/lib/python3.9/multiprocessing/context.py", line 277 in _Popen
2024-07-08 12:28:45.341382222    File "/usr/lib/python3.9/multiprocessing/context.py", line 224 in _Popen
2024-07-08 12:28:45.341383169    File "/usr/lib/python3.9/multiprocessing/process.py", line 121 in start
2024-07-08 12:28:45.341384124    File "/opt/frigate/frigate/app.py", line 520 in start_camera_capture_processes
2024-07-08 12:28:45.341385020    File "/opt/frigate/frigate/app.py", line 696 in start
2024-07-08 12:28:45.341385928    File "/opt/frigate/frigate/__main__.py", line 17 in <module>
2024-07-08 12:28:45.341386810    File "/usr/lib/python3.9/runpy.py", line 87 in _run_code
2024-07-08 12:28:45.341387724    File "/usr/lib/python3.9/runpy.py", line 197 in _run_module_as_main
2024-07-08 12:29:48.335039759  [2024-07-08 08:29:48] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for reolink-bush. Keeping the 6 most recent segments out of 7 and discarding the rest...
2024-07-08 12:29:48.336151618  [2024-07-08 08:29:48] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for reolink-frontdoor. Keeping the 6 most recent segments out of 7 and discarding the rest...
2024-07-08 12:29:48.337311511  [2024-07-08 08:29:48] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for amcrest-frontyard. Keeping the 6 most recent segments out of 7 and discarding the rest...

Operating system

Other Linux

Install method

Docker CLI

Network connection

Wired

Camera make and model

various (amcrest, reolink)

Any other information that may be helpful

Running in a K8S Cluster
Was observing with 4GB for /dev/shm, increased to 6GB, but still observing the problem
I don't observe a general increase in /dev/shm, just suddenly it's full
I just happened to catch the UI when /dev/shm was full before the UI went down

Answered by a-lunatic

Jul 8, 2024

I had this problem and I read something about the camera names with a dash(-) in the name as I got 4 cameras and all had the - and after renaming them and adding the ui order setting to sort them the way I had it fixed it for me.

View full answer

NickM-27 · 2024-07-08T12:58:09Z

NickM-27
Jul 8, 2024
Collaborator Sponsor

would be good to see more logs leading up to the error, hard to know why this might happen

15 replies

NickM-27 Jul 30, 2024
Collaborator Sponsor

Logs are already limited in size and automatically rotated when that limit is hit. Logs won't be the thing that is ballooning in size causing the space to be taken up .

cdmichaelb Jul 30, 2024

I did a bunch more digging again and turns out I just came back to this again: #12338 (reply in thread)

Except now, it's roughly 6060 files sized 442k, totaling approx my total /dev/shm storage allowance. The only difference now is that I moved logs to the SSD rather than using /dev/shm, so more were able to fit before filling up.

docker exec frigate find /dev/shm -maxdepth 1 -type f -exec du -h {} + 2>/dev/null | sort -rh | head -n 10
452K /dev/shm/garage1722367823.138389
452K /dev/shm/garage1722367822.938462
452K /dev/shm/garage1722367822.598442
452K /dev/shm/garage1722367822.338316
452K /dev/shm/garage1722367822.138773
452K /dev/shm/garage1722367821.938267
452K /dev/shm/garage1722367821.598273
452K /dev/shm/garage1722367821.338567
452K /dev/shm/garage1722367821.138488
452K /dev/shm/garage1722367820.938562
docker exec frigate find /dev/shm -maxdepth 1 -type f | wc -l
6060

I can't really extrapolate what the issue might be since I don't know what these files are for, why they aren't being removed, why they exist so long. Best I've been able to do for my self is just delete them all periodically, but that's not really a great solution since it requires somewhat frequent manual management for something that should be automated.

My first thought is oh I'll just move these to SSD, but eventually that'll fill up too.

NickM-27 Jul 30, 2024
Collaborator Sponsor

When this happens something has failed. So far we've seen limited cases of this happening. We have a plan to prevent it from causing the system to crash but that will require substantial changes to parts of the code

cdmichaelb Jul 30, 2024

Awesome, in the short term I should continue to just delete the files whenever frigate bricks? Is it safe to, temporarily, set up a script that just deletes them every time I restart frigate?

NickM-27 Jul 30, 2024
Collaborator Sponsor

I mean like I said when this happens it is usually indicative that some other part of frigate has failed catastrophically so a restart is required

rwlove · 2024-07-08T13:09:19Z

rwlove
Jul 8, 2024
Author

frigate_log.txt

0 replies

a-lunatic · 2024-07-08T13:28:56Z

a-lunatic
Jul 8, 2024

I had this problem and I read something about the camera names with a dash(-) in the name as I got 4 cameras and all had the - and after renaming them and adding the ui order setting to sort them the way I had it fixed it for me.

5 replies

rwlove Jul 8, 2024
Author

I'll give it a shot. Can you help me understand the UI Order Setting better? Are you saying that I MUST specify an order?

a-lunatic Jul 8, 2024

Just add this to each camera and number them the order you want

    ui:
      # Optional: Adjust sort order of cameras in the UI. Larger numbers come later (default: shown below)
      # By default the cameras are sorted alphabetically.
      order: 0
      # Optional: Whether or not to show the camera in the Frigate UI (default: shown below)
      dashboard: True

rwlove Jul 8, 2024
Author

OK, I'm running now with your two suggestions. I suppose if I go a week without the /dev/shm error then I'll close this ticket. It seems to reproduce every few days.

cdmichaelb Jul 14, 2024

I'm using underscores _ not dashes -. Have you had any issues with those?

rwlove Jul 17, 2024
Author

After converting dashes to underscores and setting a camera order, I have not reproduced this crash in over a week.

SamKirsch10 · 2024-07-15T13:41:53Z

SamKirsch10
Jul 15, 2024

Looks like this is the same issue as me #12140.

6 replies

NickM-27 Jul 15, 2024
Collaborator Sponsor

I think almost every user has dashes or underscores, it does not seem likely to me that that causes any issues, I have underscores on my 8 cameras and have never seen this.

SamKirsch10 Jul 15, 2024

Yeah. I figured it was a stretch

SamKirsch10 Jul 15, 2024

I see only failures for the three Wifi (others are POE)... Gunna remove those and see if we have the same crashes

NickM-27 Jul 15, 2024
Collaborator Sponsor

on the hour is when the previews are encoded into video which is a very efficient process. I tried doing some testing inserting artificial delays in that but the SHM did not increase at all. As far as I can tell, the only explanation I have is that some setups are using more CPU resources and when the previews are generated this pushes CPU usage higher which causes a delay in some processes leading to this BUS error.

This Bus Error can happen for a number of different reasons so it is difficult to assign a specific cause as there are many. The issue as a whole being that frigate fails when one part that deals with SHM fails will require substantial changes to fix so that most likely won't come until a release after 0.14.0

SamKirsch10 Jul 15, 2024

That makes sense with my Grafana charts. For now I have a daily crontab that restarts the container, so it seems to keep it at bay

SamKirsch10 · 2024-07-15T13:58:20Z

SamKirsch10
Jul 15, 2024

Will run another test. I overrode and updated the go2rtc binary with latest https://github.com/AlexxIT/go2rtc/releases/tag/v1.9.4

1 reply

NickM-27 Jul 15, 2024
Collaborator Sponsor

there are multiple fatal bugs in 1.9.3 and 1.9.4, which is why frigate has not updated yet #12117

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug]: /dev/shm fills up and freezes Frigate #12338

{{title}}

Replies: 5 comments 27 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

[Bug]: /dev/shm fills up and freezes Frigate #12338

Describe the problem you are having

Steps to reproduce

Version

Frigate config file

Relevant log output

Operating system

Install method

Network connection

Camera make and model

Any other information that may be helpful

Replies: 5 comments · 27 replies

NickM-27 Jul 8, 2024 Collaborator Sponsor

NickM-27 Jul 30, 2024 Collaborator Sponsor

NickM-27 Jul 30, 2024 Collaborator Sponsor

NickM-27 Jul 30, 2024 Collaborator Sponsor

rwlove Jul 8, 2024 Author

rwlove Jul 8, 2024 Author

rwlove Jul 8, 2024 Author

rwlove Jul 17, 2024 Author

NickM-27 Jul 15, 2024 Collaborator Sponsor

NickM-27 Jul 15, 2024 Collaborator Sponsor

NickM-27 Jul 15, 2024 Collaborator Sponsor

Replies: 5 comments 27 replies

NickM-27
Jul 8, 2024
Collaborator Sponsor

NickM-27 Jul 30, 2024
Collaborator Sponsor

NickM-27 Jul 30, 2024
Collaborator Sponsor

NickM-27 Jul 30, 2024
Collaborator Sponsor

rwlove
Jul 8, 2024
Author

rwlove Jul 8, 2024
Author

rwlove Jul 8, 2024
Author

rwlove Jul 17, 2024
Author

NickM-27 Jul 15, 2024
Collaborator Sponsor

NickM-27 Jul 15, 2024
Collaborator Sponsor

NickM-27 Jul 15, 2024
Collaborator Sponsor