[Support]: Frigate Out of Memory crash during export, export stuck

[bubba925]

Describe the problem you are having

I proposed yesterday and realized my Frigate camera caught it all. Used the export feature and let it do its thing. Looks like after it successfully exported I got an out of memory error and it killed the process, Frigate rebooted and now the export is stuck in process. I was able to grab the export from the folder and it is completed.

Version

0.13.1-34FB1C2

Frigate config file

# yaml-language-server: $schema=http://192.168.50.53:5000/api/config/schema.json
mqtt:
  host: 192.168.50.148
  port: 1883
  topic_prefix: frigate
  user: 
  password: 
  stats_interval: 60
detectors:
  coral_pci:
    type: edgetpu
    device: pci
go2rtc:
  streams:
    reolink:
    - ffmpeg:http://192.168.33.50/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=&password=#video=copy#audio=copy#audio=aac
    reolink_sub:
    - ffmpeg:http://192.168.33.50/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user=&password=#video=copy#audio=copy#audio=aac
    kitchen:
    - rtsp://:@192.168.33.51:554/cam/realmonitor?channel=1&subtype=0
    - ffmpeg:kitchen#audio=aac
    kitchen_sub:
    - rtsp://:@192.168.33.51:554/cam/realmonitor?channel=1&subtype=2
    garage:
    - rtsp://:@192.168.33.53:554/cam/realmonitor?channel=1&subtype=0
    - ffmpeg:garage#audio=aac
    garage_sub:
    - rtsp://:@192.168.33.53:554/cam/realmonitor?channel=1&subtype=2
    front_south:
    - rtsp://:@192.168.33.52:554/cam/realmonitor?channel=1&subtype=0
    - ffmpeg:front_south#audio=aac
    front_south_sub:
    - rtsp://:@192.168.33.52:554/cam/realmonitor?channel=1&subtype=2
    front_north:
    - rtsp://:@192.168.33.54:554/cam/realmonitor?channel=1&subtype=0
    - ffmpeg:front_north#audio=aac
    front_north_sub:
    - rtsp://:@192.168.33.54:554/cam/realmonitor?channel=1&subtype=2
    backyard:
    - rtsp://:@192.168.33.55:554/cam/realmonitor?channel=1&subtype=0
    - ffmpeg:backyard#audio=aac
    backyard_sub:
    - rtsp://:@192.168.33.55:554/cam/realmonitor?channel=1&subtype=2
  webrtc:
    candidates:
    - 192.168.50.53:8555
    - stun:8555
snapshots:
  enabled: true
  clean_copy: true
  timestamp: false
  bounding_box: true
  crop: false
  required_zones:
  - front_door
  - street_south
  - driveway_south
  - front_yard_south
  - street_north
  - driveway_north
  - front_yard_north
  - back_deck
  - back_yard
  - garage_area
  retain:
    default: 30
objects:
  filters:
    person:
      min_score: 0.6
      threshold: 0.8
      min_area: 300
    bird:
      min_score: 0.65
      threshold: 0.85
ffmpeg:
  hwaccel_args: -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format
    yuv420p
  input_args: preset-rtsp-restream
  output_args:
    record: preset-record-generic-audio-copy
cameras:
  kitchen:
    ffmpeg:
      inputs:
      - path: rtsp://192.168.50.53:8554/kitchen?video=copy&audio=aac
        roles:
        - record
      - path: rtsp://192.168.50.53:8554/kitchen_sub?video=copy
        roles:
        - detect
    detect:
      enabled: false
    record:
      enabled: true
      retain:
        days: 5
        mode: all
    motion:
      mask:
      - 1237,0,1235,66,1180,66,1177,26
      - 798,0,720,0,717,23,793,30
  garage:
    ffmpeg:
      inputs:
      - path: rtsp://192.168.50.53:8554/garage?video=copy&audio=aac
        roles:
        - record
      - path: rtsp://192.168.50.53:8554/garage_sub?video=copy
        roles:
        - detect
    snapshots:
      enabled: true
    detect:
      enabled: true
      width: 1280
      height: 720
      fps: 5
    objects:
      track:
      - person
      - dog
      - cat
    zones:
      garage_area:
        coordinates: 1280,720,0,720,0,147,0,0,194,0,1280,0,1280,166
        objects:
        - person
        - cat
        - dog
    record:
      enabled: true
      retain:
        days: 7
        mode: all
      events:
        retain:
          default: 15
          mode: active_objects
        pre_capture: 5
        post_capture: 8
        objects:
        - person
        - dog
        - cat
    motion:
      mask:
      - 1229,21,1226,62,1184,57,1182,26
  doorbell:
    ffmpeg:
      inputs:
      - path: rtsp://192.168.50.53:8554/reolink?video=copy&audio=aac
        roles:
        - record
      - path: rtsp://192.168.50.53:8554/reolink_sub?video=copy
        roles:
        - detect
      output_args:
        record: preset-record-generic-audio-aac
    detect:
      enabled: true
      width: 640
      height: 480
      fps: 4
    objects:
      track:
      - person
      - dog
      - cat
      - car
      filters:
        person:
          min_area: 500
    live:
      stream_name: reolink
    motion:
      mask:
      - 370,38,640,31,640,0,374,0
    record:
      enabled: true
      retain:
        days: 7
        mode: all
      events:
        required_zones:
        - front_door
        retain:
          default: 15
          mode: active_objects
        pre_capture: 5
        post_capture: 8
        objects:
        - person
        - dog
        - cat
        - car
    zones:
      front_door:
        coordinates: 
          640,274,573,295,518,278,479,290,471,180,538,234,640,146,640,0,302,0,0,0,0,480,640,480
        objects:
        - person
        - dog
        - cat
    mqtt:
      crop: false
      required_zones:
      - front_door
  front_south:
    ffmpeg:
      inputs:
      - path: rtsp://192.168.50.53:8554/front_south?video=copy&audio=aac
        roles:
        - record
      - path: rtsp://192.168.50.53:8554/front_south_sub?video=copy
        roles:
        - detect
    detect:
      enabled: true
      width: 1280
      height: 720
      fps: 5
    objects:
      track:
      - person
      - dog
      - cat
      - car
      - bird
      filters:
        person:
          min_area: 350
    motion:
      mask:
      - 426,677,906,720,1182,720,1154,647,1003,606,940,496,865,463,795,498,650,477,591,403,336,406
      - 1280,0,829,0,866,114,979,202,1149,329,1280,356
    zones:
      street_south:
        coordinates: 
          938,166,960,127,839,89,657,56,568,38,467,35,309,31,204,37,0,71,0,126,382,96,653,110
        objects:
        - person
        - dog
        - cat
        - bird
      driveway_south:
        coordinates: 419,720,0,720,0,174,399,154
        objects:
        - car
        - person
        - dog
        - cat
        - bird
      front_yard_south:
        coordinates: 407,144,391,345,398,428,1280,720,1256,457,1070,249,754,177
        objects:
        - car
        - person
         # - dog
        - cat
        - bird
    record:
      enabled: true
      retain:
        days: 7
        mode: all
      events:
        required_zones:
        - street_south
        - driveway_south
        - front_yard_south
        retain:
          default: 15
          mode: active_objects
        pre_capture: 5
        post_capture: 8
        objects:
        - person
        - dog
        - cat
        - car
    mqtt:
      crop: false
      required_zones:
      - street_south
      - driveway_south
      - front_yard_south
  front_north:
    ffmpeg:
      inputs:
      - path: rtsp://192.168.50.53:8554/front_north?video=copy&audio=aac
        roles:
        - record
      - path: rtsp://192.168.50.53:8554/front_north_sub?video=copy
        roles:
        - detect
    detect:
      enabled: true
      width: 1280
      height: 720
      fps: 5
    objects:
      track:
      - person
      - dog
       # - cat
      - car
      - bird
      filters:
        person:
          min_area: 350
    zones:
      street_north:
        coordinates: 
          535,80,918,98,1144,147,1135,129,1031,92,913,69,708,32,500,21,311,39,0,102,0,147,341,82
        objects:
        - person
        - dog
          #- cat
        - bird
      driveway_north:
        coordinates: 658,720,1169,674,1132,569,1280,355,891,130,426,120
        objects:
        - car
        - person
        - dog
         # - cat
        - bird
      front_yard_north:
        coordinates: 641,720,0,720,0,194,425,121
        objects:
        - car
        - person
        - dog
         # - cat
        - bird
    record:
      enabled: true
      retain:
        days: 7
        mode: all
      events:
        required_zones:
        - street_north
        - driveway_north
        - front_yard_north
        retain:
          default: 15
          mode: active_objects
        pre_capture: 5
        post_capture: 8
        objects:
        - person
        - dog
        #  - cat
        - car
    mqtt:
      crop: false
      required_zones:
      - street_north
      - driveway_north
      - front_yard_north
    motion:
      mask:
      - 1233,27,1233,70,1175,65,1175,28
  backyard:
    ffmpeg:
      inputs:
      - path: rtsp://192.168.50.53:8554/backyard?video=copy&audio=aac
        roles:
        - record
      - path: rtsp://192.168.50.53:8554/backyard_sub?video=copy
        roles:
        - detect
    snapshots:
      enabled: true
    detect:
      enabled: true
      width: 1280
      height: 720
      fps: 5
    objects:
      track:
      - person
      - dog
      - cat
      - car
      - bird
    motion:
      mask:
      - 1250,189,1202,56,1156,0,1030,0,760,0,765,132,986,182,988,243,1022,260,1100,293,1227,292
      - 932,250,742,225,742,253,899,274
    zones:
      back_yard:
        coordinates: 1280,720,233,720,197,191,760,197,1280,340
        objects:
        - car
        - dog
        - person
        - cat
        - bird
      back_deck:
        coordinates: 189,0,658,0,961,0,1039,31,1019,183,658,121,196,101
        objects:
        - person
        - dog
        - cat
        - bird
    record:
      enabled: true
      retain:
        days: 7
        mode: all
      events:
        retain:
          default: 15
          mode: active_objects
        pre_capture: 5
        post_capture: 8
        objects:
        - person
        - dog
        - cat
        - car
birdseye:
  enabled: true
  mode: objects

Relevant log output

2024-03-07 19:12:16.362628812  [INFO] Preparing Frigate...
2024-03-07 19:12:16.404494037  [INFO] Starting Frigate...
2024-03-07 19:12:18.697670531  [2024-03-07 19:12:18] frigate.app                    INFO    : Starting Frigate (0.13.1-34fb1c2)
2024-03-07 19:12:24.801908696  [2024-03-07 19:12:24] peewee_migrate.logs            INFO    : Starting migrations
2024-03-07 19:12:24.817984822  [2024-03-07 19:12:24] peewee_migrate.logs            INFO    : There is nothing to migrate
2024-03-07 19:12:24.818514744  [2024-03-07 19:12:24] frigate.app                    INFO    : Running database vacuum
2024-03-07 19:12:30.836107972  [2024-03-07 19:12:30] frigate.app                    INFO    : Recording process started: 303
2024-03-07 19:12:30.840039820  [2024-03-07 19:12:30] frigate.app                    INFO    : go2rtc process pid: 97
2024-03-07 19:12:30.870531365  [2024-03-07 19:12:30] frigate.app                    INFO    : Output process started: 314
2024-03-07 19:12:30.986864993  [2024-03-07 19:12:30] detector.coral_pci             INFO    : Starting detection process: 313
2024-03-07 19:12:30.986869057  [2024-03-07 19:12:30] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as pci
2024-03-07 19:12:30.997283561  [2024-03-07 19:12:30] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found
2024-03-07 19:12:31.057157906  [2024-03-07 19:12:30] frigate.app                    INFO    : Camera processor started for kitchen: 344
2024-03-07 19:12:31.061889891  [2024-03-07 19:12:31] frigate.app                    INFO    : Camera processor started for garage: 355
2024-03-07 19:12:31.062288093  [2024-03-07 19:12:31] frigate.app                    INFO    : Camera processor started for doorbell: 358
2024-03-07 19:12:31.062562015  [2024-03-07 19:12:31] frigate.app                    INFO    : Camera processor started for front_south: 360
2024-03-07 19:12:31.062852696  [2024-03-07 19:12:31] frigate.app                    INFO    : Camera processor started for front_north: 361
2024-03-07 19:12:31.063107246  [2024-03-07 19:12:31] frigate.app                    INFO    : Camera processor started for backyard: 362
2024-03-07 19:12:31.063358835  [2024-03-07 19:12:31] frigate.app                    INFO    : Capture process started for kitchen: 364
2024-03-07 19:12:31.072455908  [2024-03-07 19:12:31] frigate.app                    INFO    : Capture process started for garage: 367
2024-03-07 19:12:31.082146216  [2024-03-07 19:12:31] frigate.app                    INFO    : Capture process started for doorbell: 374
2024-03-07 19:12:31.100516258  [2024-03-07 19:12:31] frigate.app                    INFO    : Capture process started for front_south: 382
2024-03-07 19:12:31.100520642  [2024-03-07 19:12:31] frigate.app                    INFO    : Capture process started for front_north: 387
2024-03-07 19:12:31.104669483  [2024-03-07 19:12:31] frigate.app                    INFO    : Capture process started for backyard: 389
2024-03-08 00:45:19.958011776  [2024-03-08 00:45:19] frigate.video                  ERROR   : kitchen: Unable to read frames from ffmpeg process.
2024-03-08 00:45:19.958894850  [2024-03-08 00:45:19] frigate.video                  ERROR   : kitchen: Unable to read frames from ffmpeg process.
2024-03-08 00:45:19.958898707  [2024-03-08 00:45:19] frigate.video                  ERROR   : kitchen: Unable to read frames from ffmpeg process.
2024-03-08 00:45:19.958900884  [2024-03-08 00:45:19] frigate.video                  ERROR   : kitchen: Unable to read frames from ffmpeg process.
2024-03-08 00:45:19.959201434  [2024-03-08 00:45:19] frigate.video                  ERROR   : kitchen: Unable to read frames from ffmpeg process.
2024-03-08 00:45:19.959577722  [2024-03-08 00:45:19] frigate.video                  ERROR   : kitchen: ffmpeg process is not running. exiting capture thread...
2024-03-08 00:45:22.520517840  [2024-03-08 00:45:22] watchdog.kitchen               ERROR   : Ffmpeg process crashed unexpectedly for kitchen.
2024-03-08 00:45:22.520522651  [2024-03-08 00:45:22] watchdog.kitchen               ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2024-03-08 00:45:22.520971411  [2024-03-08 00:45:22] ffmpeg.kitchen.detect          ERROR   : [AVHWFramesContext @ 0x7f2704053440] Failed to sync surface 0x15: 23 (internal decoding error).
2024-03-08 00:45:22.520974997  [2024-03-08 00:45:22] ffmpeg.kitchen.detect          ERROR   : [h264 @ 0x55e7def52140] Failed to transfer data to output frame: -5.
2024-03-08 00:45:22.520977077  [2024-03-08 00:45:22] ffmpeg.kitchen.detect          ERROR   : Error while processing the decoded data for stream #0:0

FFprobe output from your camera

[{"return_code":0,"stderr":"","stdout":{"programs":[],"streams":[{"avg_frame_rate":"15/1","codec_long_name":"H.264/AVC/MPEG-4AVC/MPEG-4part10","height":2160,"width":3840},{"avg_frame_rate":"0/0","codec_long_name":"AAC(AdvancedAudioCoding)"}]}},{"return_code":0,"stderr":"","stdout":{"programs":[],"streams":[{"avg_frame_rate":"5/1","codec_long_name":"H.264/AVC/MPEG-4AVC/MPEG-4part10","height":720,"width":1280}]}}]

Frigate stats

No response

Operating system

Proxmox

Install method

Docker Compose

Coral version

M.2

Network connection

Wired

Camera make and model

IPC-T58IR-SE 3.6mm

Any other information that may be helpful

No response

[NickM-27]

are these logs from now or when it failed?

[NickM-27]

No, debug won't capture it. The problem is the logs are getting flushed on container restart.

[bubba925]

If i did something like docker logs --follow frigate > frigatelogs.txt would that work to catch it if I started it up again? Kind of shooting in the dark with my limited knowledge.

[NickM-27]

I don't know about the command syntax but yes that would work

[bubba925]

Did it, crashed again. Same outcome of logs even when following and outputting to another file. Not sure what other logs to find since my ubuntu logs just show the out of memory and killing the python process.

Export settings I used are real time, kitchen camera, 1620 to 1700. Finished file is 1.2GB in size. My /dev/shm and /tmp/cache are 1.00GiB and 953.7MiB respectively.

[bubba925]

I renamed the export to remove the in-progress and the exports looks fine now on the exports page and is no longer "loading" unsure of what caused it to crash repeatedly.