0.15 beta / Shift between snapshots, object lifecycle and recorded video

[arpel]

Describe the problem you are having

Hi,
I'm a (new) user of Frigate+ with a custom trained model (mobiledet), I'm paying extra attention to the new detections and there is almost always a shift in my alerts :

• the "snapshot", as in the thumbnails of the UI or in the "explore" tab is always "good", showing the detected object
• the associated recorded video (from the same explore window for this detection or from the associated alerts (all detections are alerts) is almost late by more than 10 sec (my cameras have timestamps : the snapshot show 19:06:28 and the video start at 19:06:40)
• the "object lifecyle" tab for the same tracked object shows "no image" for IN (even if the actual snapshot ... exists) and has an "OUT" image showing a bounding box with no object but nowhere near where the object actually left the scene).

I have good inference times (7,5ms) and my system is far from overloaded (n100 with long time load <0,7).

Do you have any clue of what could be happening ?

Version

0.15.0-9ae839a

What browser(s) are you using?

No response

Frigate config file

mqtt:
  enabled: true
  host: 192.168.0.61

ffmpeg:
  hwaccel_args: preset-vaapi

semantic_search:
  enabled: true
  model_size: small
  reindex: false

genai:
  enabled: false
  provider: gemini
  api_key: [geminiAPI]
  model: gemini-1.5-flash

birdseye:
  enabled: false

detectors:
  coral:
    type: edgetpu
    device: usb

model:
  path: plus://[plusmodelID]
  
snapshots:
  enabled: true
  retain:
    default: 20
  quality: 90 #default 70
  # Removing 2mB PNGs
  clean_copy: true

record:
  enabled: true
  retain:
    days: 0
    mode: motion
  sync_recordings: true
  alerts:
    retain:
      days: 9
  detections:
    retain:
      days: 9

objects:
  track:
    - person
    - bicycle
    - car
    - cat
    - dog
    - bird
    - face
    - package
    - licence_plate
  filters:
    car:
      min_score: 0.65
      threshold: 0.85
    cat:
      min_score: 0.65
      threshold: 0.85
    dog:
      min_score: 0.6
      threshold: 0.8
    person:
      min_score: 0.6
      threshold: 0.85

cameras:
  hik_patio:
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/hik_patio
          input_args: preset-rtsp-restream
          hwaccel_args: preset-intel-qsv-h265
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/hik_patio_sub
          input_args: preset-rtsp-restream
          hwaccel_args: preset-intel-qsv-h265
          roles:
            - detect
    detect:
      enabled: true
      width: 1280
      height: 720
      fps: 5
      # Default = 5 times fps
      max_disappeared: 80
    zones:
      porte_garage:
        coordinates: 0,720,391,720,337,199,0,202
      portail:
        coordinates: 469,85,584,84,579,21,470,22
    motion:
      mask:
        - 737,0,988,0,815,720,680,720
        - 0,59,365,61,362,0,0,0
        - 1148,720,1096,572,849,569,816,720
    # Try this
        - 1280,278,1109,194,1117,0,1280,0
      threshold: 30
      contour_area: 20
      improve_contrast: true
    mqtt:
      timestamp: false
      bounding_box: false
      crop: true
      quality: 100
      height: 720

  hik_allee:
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/hik_allee
          input_args: preset-rtsp-restream
          hwaccel_args: preset-intel-qsv-h265
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/hik_allee_sub
          input_args: preset-rtsp-restream
          hwaccel_args: preset-intel-qsv-h265
          roles:
            - detect
    detect:
      enabled: true
      width: 1280
      height: 720
      #fps: 8
      fps: 5
      max_disappeared: 80
    objects:
      filters:
        car:
          mask: 0.097,0.57,0.528,0.373,0.661,0.331,0.893,0.507,0.668,0.993,0.202,1
    mqtt:
      timestamp: false
      bounding_box: false
      crop: true
      quality: 100
      height: 720

    motion:
      mask:
        - 0.268,0.067,0.268,0,0,0,0,0.065
        - 0.272,0,1,0,1,0.256,0.265,0.106
      threshold: 30
      contour_area: 20
      improve_contrast: true
    zones:
      Portail:
        coordinates: 0.242,0.168,0.245,0.274,0.125,0.316,0.115,0.199
        loitering_time: 0
      Chemin:
        coordinates: 1,0.336,0.873,0.331,0.69,1,1,1
        loitering_time: 0
        objects: person
      Allée:
        coordinates: 0.249,0.182,0.251,0.279,0.127,0.321,0.14,0.638,0.51,0.435,0.518,0.309
        loitering_time: 0

  hik_portail:
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/hik_portail
          input_args: preset-rtsp-restream
          hwaccel_args: preset-intel-qsv-h264
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/hik_portail_sub
          input_args: preset-rtsp-restream
          hwaccel_args: preset-intel-qsv-h264
          roles:
            - detect
    detect:
      enabled: true
      width: 640
      #width: 854
      height: 480
      fps: 4
      max_disappeared: 80
    objects:
      filters:
        car:
          mask: 0.163,0.409,0.851,0.423,0.877,0.653,0.165,0.575
    zones:
      Portail:
        coordinates: 0,0.189,0.174,0.189,0.18,0.593,1,0.691,1,1,0,1
        inertia: 9
        loitering_time: 0
    motion:
      mask:
        - 0.013,0,0.012,0.041,0.268,0.041,0.269,0
        - 0.789,0,0.879,0.735,1,0.733,1,0
        - 0.1,0.238,0.066,0.327,0,0.327,0,0,0.286,0
        - 0,0.665,0.016,0.666,0.016,0.693,0,0.693
      threshold: 40
      contour_area: 10
      improve_contrast: 'true'

go2rtc:
  streams:
    hik_portail:
      - rtsp://[ad/pass]@192.168.0.202:554/Streaming/Channels/101
      - isapi://[ad/pass]@192.168.0.202:80/
    hik_portail_sub:
      - ffmpeg:rtsp://[ad/pass]@192.168.0.202:554/Streaming/Channels/102
    hik_allee:
      - rtsp://[ad/pass]@192.168.0.201/Streaming/Channels/101
    hik_allee_sub:
      - rtsp://[ad/pass]@192.168.0.201/Streaming/Channels/102
    hik_patio:
      - rtsp://[ad/pass]@192.168.0.200/Streaming/Channels/101
    hik_patio_sub:
      - rtsp://[ad/pass]@192.168.0.200/Streaming/Channels/102
  ffmpeg:
    volume: -af "volume=2"

logger:
  # Optional: default log level (default: shown below)
  #default: info
  default: warning
version: 0.15-0

Relevant Frigate log output

N/A no error in the log

Relevant go2rtc log output

N/A no error in the log

FFprobe output from your camera

N/A

Frigate stats

No response

Install method

Docker Compose

docker-compose file or Docker CLI command

version: "3.9"
services:
  frigate:
    container_name: frigate
    privileged: true # this may not be necessary for all setups
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:9ae839a-amd64
    shm_size: "160mb"
    devices:
      - /dev/dri/renderD128:/dev/dri/renderD128 # For intel hwaccel, needs to be updated for your hardware
      - /dev/bus/usb:/dev/bus/usb

    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /home/arpel/.frigate:/config
      - /media/truenas-shares/Frigate:/media/frigate
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          #size: 1000000000
          #size: 500000000
          size: 256000000
    ports:
      - "5000:5000"
      - "8554:8554" # RTSP feeds
      - "8555:8555/tcp" # WebRTC over tcp
      - "8555:8555/udp" # WebRTC over udp
      - "1984:1984"
    network_mode: "host"
    environment:
      FRIGATE_RTSP_PASSWORD: "password"
      PLUS_API_KEY: "KEY"

Object Detector

Coral

Network connection

Wired

Camera make and model

Hikvision DS-2143

Screenshots of the Frigate UI's System metrics pages

No response

Any other information that may be helpful

No response

[NickM-27]

this is usually due to an issue of timing between the detect and record stream. You might be better off using qsv instead of preset-vaapi

[NickM-27]

would be good to see screenshots of the system and camera metrics

[arpel]

It's better with vaapi as a preset for all streams but I still have recordings not complying with "pre_capture" : the video basically starts between -6sec and 0 sec from the snapshot timestamp.

[NickM-27]

Are you referring to the review page?

[arpel]

I'm referring to the the "Object" page.
Your question is just making my doubt about the actual meaning of "pre_capture" which is related to "movement" and associated global recording but not specifically about an object (but then what is the real link between a movement - related to a tracked object type - and this object ?).

[NickM-27]

Pre capture has been fixed for the next beta

[hawkeye217]

For the object lifecycle, you should adjust your annotation offset either from the UI (the Object Lifecycle pane has a gear icon for settings) for via your config (annotation_offset).

[NickM-27]

To be clear, If there is a large disparity in general recordings vs detect streams then it is probably best to get that sorted out first

[arpel]

I've changed my global settings to "hwaccel_args: preset-intel-qsv-h265" and I"m still getting ~10sec delay, here's an exemple from today.
Snapshot :

Video :

Stats - System

Stats - Camera

[NickM-27]

you might try connecting directly to the camera instead of the restream and see how that helps things

[ivitjuk]

Hi, I would like to report exactly the same issue. Time on the "Snapshot" image matches the event time, while time on the "Object Lifecycle" image is approximately 10s behind the event. Camera time is well synchronized.

See the screenshots bellow:

Object Lifecycle:

Snapshot:

[ivitjuk]

This is on frigate:0.15.0-beta2

[ivitjuk]

Still seeing this on frigate:0.15.0-beta4. No delay on the "Snapshot" view, and ~10s delay on the "Object lifecycle" view.

[ivitjuk]

FYI, I've been digging into this issue for a few days now, and I believe it is the same issue as described in [1]. Reliably, after a few hours of running frigate, camera timestamp in video segments stored in /tmp/cache will lag behind timestamp in the segment file name by approximately 10s. This 10s boundary is probably related to the fact frigate uses that interval for storing segments.

So I added -use_wallclock_as_timestamps 1 to the preset-rtsp-restream-low-latency set of ffmpeg input args. Frigate has been running for ~24h with drift no longer than 1s. So it seems like solution from that email thread does help. Note that I did not need to apply other fixes from the email thread.

Complete set of input args I use is as follows:

input_args: "-rtsp_transport tcp -timeout 5000000 -fflags nobuffer -flags low_delay -use_wallclock_as_timestamps 1"

[1] https://lists.ffmpeg.org/pipermail/ffmpeg-user/2022-July/055200.html