Consider use of live_flv ffmpeg flag with cameras that have timestamp issues

[charlesmunger]

I'm the owner of some Reolink RLC-410 cameras, and I used the config linked in the docs to get a stream that decodes successfully. However, I had an issue where clips would have delayed frames followed by sped up sections. I suspect that this is caused by an interaction of the following options:

-use_wallclock_as_timestamps in frigate, a CPU (rather than tpu) detector, and

<clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='yes'/>
</clock>

in my VM configuration. When an object is detected, the CPU detector puts the CPU under heavy load. This in turn delays processing of frames from the RTMP input, which means that the real time clock drifts from the actual correct inter-frame timing. When played back, the clip appears to freeze and speed up intermittently depending on the load when it was recorded. While this is certainly exacerbated by the VM, it's possible for it to happen on any system that's under load or bandwidth constrained.

I researched and tested out the live_flv option for ffmpeg. It appears to be designed for exactly this use case - timestamp discontinuities in livestreamed FLV. I specified this configuration:

      input_args:
        - -fflags
        - nobuffer
        - -flags
        - low_delay
        - -strict
        - experimental
        - -rw_timeout
        - '5000000'
        - -f
        - live_flv

Which then evaluates to this:

      "ffmpeg_cmds": [
        {
          "cmd": "ffmpeg -hide_banner -loglevel warning -fflags nobuffer -flags low_delay -strict experimental -rw_timeout 5000000 -f live_flv -i rtmp://10.0.0.253/bcs/channel0_main.bcs?channel=0&stream=0&user=frigate&password=********* -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an /tmp/cache/tree_driveway-%Y%m%d%H%M%S.mp4 -c copy -f flv rtmp://127.0.0.1/live/tree_driveway",
          "roles": [
            "clips",
            "rtmp"
          ]
        },
        {
          "cmd": "ffmpeg -hide_banner -loglevel warning -fflags nobuffer -flags low_delay -strict experimental -rw_timeout 5000000 -f live_flv -i rtmp://10.0.0.253/bcs/channel0_sub.bcs?channel=0&stream=0&user=frigate&password=********* -r 7 -f rawvideo -pix_fmt yuv420p pipe:",
          "roles": [
            "detect"
          ]
        }

This completely fixed my issues with clips - they're rock solid now, perfectly smooth motion. I've been running this configuration for a few months. I also tried introducing artificial CPU load and starving the ffmpeg process of resources, and it handled those just fine. I'd suggest updating the documentation to use it for reolink cameras, or even changing it to the default frigate behavior for rtmp streams if some users test it on other cameras and it works well.

[skulumani]

I have two RLC-520 cameras. The following options are what works for me. Using that posted above @charlesmunger caused only a green image to display

input_args:
  - -avoid_negative_ts
  - make_zero
  - -flags
  - nobuffer
  - -flags
  - low_delay
  - -strict
  - experimental
  - --flags
  - +genpts+discardcorrupt
  - -rw_timeout
  - '5000000'
  - -use_wallclock_as_timestamps
  - '1'
  - -f
  - live_flv

[charlesmunger]

@skulumani
Thanks for testing the config out! Can you confirm that you're using the RTMP stream, not RTSP? Can you also share the ffmpeg_cmds from the debug page (redacted of password in URL)

[skulumani]

Hello @charlesmunger

Here are the ffmpeg commands

I checked a couple of clips from this morning and still seem to have the stuttering so perhaps it's not working as well on my system. I should check it by minimizing CPU usage as well. However this set of options at least shows the video.

Also my cameras are connected via RTMP as shown below.

      "ffmpeg_cmds": [
        {
          "cmd": "ffmpeg -hide_banner -loglevel warning -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -rw_timeout 5000000 -use_wallclock_as_timestamps 1 -f live_flv -i rtmp://stream:[email protected]/bcs/channel0_sub.bcs?token=sdasdasd&channel=0&stream=1&user=stream&password=dgdiPUdFwqkqBwRFSq -c copy -f flv rtmp://127.0.0.1/live/back -r 15 -f rawvideo -pix_fmt yuv420p pipe:",
          "roles": [
            "detect",
            "rtmp"
          ]
        },
        {
          "cmd": "ffmpeg -hide_banner -loglevel warning -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -rw_timeout 5000000 -use_wallclock_as_timestamps 1 -f live_flv -i rtmp://stream:[email protected]/bcs/channel0_main.bcs?token=sdasdasd&channel=0&stream=0&user=stream&password=dgdiPUdFwqkqBwRFSq -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -vcodec copy -acodec copy /tmp/cache/back-%Y%m%d%H%M%S.mp4",
          "roles": [
            "clips"
          ]
        }
      ]

[MarkGodwin]

Your mileage may vary, but I find the raw http stream that the Reolink Web interfaces use seems to have a lower overhead and is more reliable than RTMP & RTSP, perhaps because it doesn't have to wrap the video data in RTMP packets:

http:/192.168.1.193/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=admin&password=****

It appears that what comes over the wire is the raw .FLV data.

This is what I have been using with Frigate on top of the Reolink NVR for the last few months, but I'm looking forward to trying the live_flv flag too.

[charlesmunger]

@skulumani by any chance have your cameras been on for more than 49.7 days? That produces an integer overflow in the FLV timestamp that ffmpeg seems to handle poorly.

[skulumani]

@charlesmunger Most likely. That's a good find. Let me try a restart. I also think I recall an ability to set an auto reboot as well

…

On Sat, Aug 14, 2021, 16:59 Charles Munger ***@***.***> wrote: @skulumani <https://github.com/skulumani> by any chance have your cameras been on for more than 49.7 days? That produces an integer overflow in the FLV timestamp that ffmpeg seems to handle poorly. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1467 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABIFUZVXAALTAY3KFUQZUH3T43KMNANCNFSM5BVJ37VQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email> .

[jasonpstokes]

You can schedule a daily/weekly restart on the camera through its web interface in System / Maintenance.

Thanks for the above, some more things to try! Although my RLC-520A cameras on firmware 1.0.274 have been much more stable. (Note this Firmware is likely still in testing and just for the 520A, so won't help the OP but many help others who find this thread.)

[charlesmunger]

Your mileage may vary, but I find the raw http stream that the Reolink Web interfaces use seems to have a lower overhead and is more reliable than RTMP & RTSP, perhaps because it doesn't have to wrap the video data in RTMP packets:

http:/192.168.1.193/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=admin&password=****

It appears that what comes over the wire is the raw .FLV data.

This is what I have been using with Frigate on top of the Reolink NVR for the last few months, but I'm looking forward to trying the live_flv flag too.

@MarkGodwin Can you share the yaml configuration you're using and the generated ffmpeg URL (redacted of your password)? Also, can you confirm if you were originally encountering the stuttering clips issue?

[skulumani]

@charlesmunger Tried a restart of my cameras and have them set to autoreboot each week now.

Using the original ffmpeg commands listed below:

      input_args:
        - -fflags
        - nobuffer
        - -flags
        - low_delay
        - -strict
        - experimental
        - -rw_timeout
        - '5000000'
        - -f
        - live_flv

And still see green image for the camera feed. The following works for my cameras but I still experience a slight stutter.

input_args:
  - -avoid_negative_ts
  - make_zero
  - -flags
  - nobuffer
  - -flags
  - low_delay
  - -strict
  - experimental
  - --flags
  - +genpts+discardcorrupt
  - -rw_timeout
  - '5000000'
  - -use_wallclock_as_timestamps
  - '1'
  - -f
  - live_flv

It is noticeably reduced by using the live_flv option. Also reducing CPU load seems to have helped as well. Will need get a Coral device whenever they are back in stock. Thank you for the suggestions

[skulumani]

As an update for others with similar experiences I've switch to a Coral device. As explained in the documentation, this allows for much faster response rates (8ms vs. over 150ms in my case of 2 x 2560x1920 cameras). Previously I would have to do detection on 640x480 stream but now can handle larger resolutions which allows for detection of much smaller objects.

CPU usage is reduced and the clips are stuttering at a much less noticeable level. At this resolution one has to do some experimenting to find a FPS that allows for ease of detection (5 FPS ) vs. having smooth clips (higher FPS desired). However in the case of the RLC-520 there is only a single stream at high resolution so at around 15 FPS seems like the maximum FPS that my setup can handle.

So a combination of the Coral and the live_flv option seems to have helped my situation.

[blakeblackshear]

Note that the fps in your config does not have to match your camera. You can use a 15fps camera stream and specify 5fps in the config. If you are using the same stream for both clips/recordings/detect, ffmpeg will limit the detect to 5fps, but will still direct copy the 15fps stream for clips/recordings.

[skulumani]

Note that the fps in your config does not have to match your camera. You can use a 15fps camera stream and specify 5fps in the config. If you are using the same stream for both clips/recordings/detect, ffmpeg will limit the detect to 5fps, but will still direct copy the 15fps stream for clips/recordings.

Hi @blakeblackshear. I just tried this on my setup. Cameras set at 30FPS and Frigate config set to 5FPS. A clip is recorded at about 1/6 speed as a result. Are there any special ffmpeg commands required?

https://youtu.be/8RLCrjnJ68c

[blakeblackshear]

Post your ffmpeg cmd from the debug page

[skulumani]

Thanks for taking a look

      "ffmpeg_cmds": [
        {
          "cmd": "ffmpeg -hide_banner -loglevel warning -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -rw_timeout 5000000 -use_wallclock_as_timestamps 1 -f live_flv -i rtmp://stream:<password>@192.168.200.16/bcs/channel0_sub.bcs?token=sdasdasd&channel=0&stream=1&user=stream&password=<password>q -c copy -f flv rtmp://127.0.0.1/live/back",
          "roles": [
            "rtmp"
          ]
        },
        {
          "cmd": "ffmpeg -hide_banner -loglevel warning -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -rw_timeout 5000000 -use_wallclock_as_timestamps 1 -f live_flv -i rtmp://stream:<password>@192.168.200.16/bcs/channel0_main.bcs?token=sdasdasd&channel=0&stream=0&user=stream&password=<password> -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -vcodec copy -acodec copy /tmp/cache/back-%Y%m%d%H%M%S.mp4 -r 5 -f rawvideo -pix_fmt yuv420p pipe:",
          "roles": [

      "ffmpeg_cmds": [
        {
          "cmd": "ffmpeg -hide_banner -loglevel warning -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -rw_timeout 5000000 -use_wallclock_as_timestamps 1 -f live_flv -i rtmp://stream:<password>@192.168.200.15/bcs/channel0_sub.bcs?token=sdasdasd&channel=0&stream=0&user=stream&password=<password> -c copy -f flv rtmp://127.0.0.1/live/front",
          "roles": [
            "rtmp"
          ]
        },
        {
          "cmd": "ffmpeg -hide_banner -loglevel warning -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -rw_timeout 5000000 -use_wallclock_as_timestamps 1 -f live_flv -i rtmp://stream:<password>@192.168.200.15/bcs/channel0_main.bcs?token=sdasdasd&channel=0&stream=0&user=stream&password=<password> -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -vcodec copy -acodec copy /tmp/cache/front-%Y%m%d%H%M%S.mp4 -r 5 -f rawvideo -pix_fmt yuv420p pipe:",
          "roles": [
            "clips",
            "detect"
          ]
        }
      ],
   
}

[blakeblackshear]

I'm not sure why that would be happening. The clips are assembled from the segments that are using the copy codec, so nothing should be changing. Maybe remove the wallclock as timestamps parameter in your input args.

[jasonpstokes]

@skulumani I use a slightly different path for my RLC-520A camera main stream (channel=1) so maybe try that too? Here is my Frigate config for one camera FYI, which mostly (#1567) works well!

ffmpeg:
  hwaccel_args:
    - -hwaccel
    - vaapi
    - -hwaccel_device
    - /dev/dri/renderD128

cameras:
  Driveway:
    ffmpeg:
      inputs:
        - path: rtmp://192.168.1.71/bcs/channel0_sub.bcs?channel=0&stream=0&user=admin&password={FRIGATE_RTSP_PASSWORD}
          roles:
            - detect
        - path: rtmp://192.168.1.71/bcs/channel0_main.bcs?channel=1&stream=0&user=admin&password={FRIGATE_RTSP_PASSWORD}
          roles:
            - record
            - rtmp
      input_args:
        - -fflags
        - nobuffer
        - -flags
        - low_delay
        - -strict
        - experimental
        - -rw_timeout
        - '5000000'
        - -f
        - live_flv
    detect:     # line needed for v9 beta 2
      width: 640
      height: 480
      fps: 7

    motion:
      mask:
        - 222,469,222,447,417,447,417,469  # masks date and timestamp (set to bottom centre)

    objects:
      track:
        - person
        - dog

    record:
      enabled: true
      retain_days: 14
      events:
        enabled: true
        pre_capture: 3
        retain:
          default: 14

    snapshots:
      retain:
        default: 14

[charlesmunger]

It sounds like nobody in this thread who tried live_flv saw issues - the only negative is that for some cameras, live_flv was not enough to make the stream work without the additional timestamp resetting flags. @blakeblackshear, what would you want to see before:

1. Updating the docs to recommend use of live_flv instead of use_wallclock_as_timestamps
2. Changing RTMP urls to default to live_flv

[flaektrem]

Updating the docs to recommend use of live_flv instead of use_wallclock_as_timestamps

@charlesmunger I use both right now with no issues (see below).
If I remove args marked with "#", clips are stuttering when viewing. But no stuttering if I download and view in VLC.
With the following config there´s no stuttering at all. No matter where I view the clips.

ffmpeg:
  input_args:
    - -avoid_negative_ts #
    - make_zero #
    - -fflags
    - nobuffer
    - -flags
    - low_delay
    - -strict
    - experimental
    - -fflags #
    - +genpts+discardcorrupt #
    - -rw_timeout
    - '5000000'
    - -use_wallclock_as_timestamps #
    - "1" #
    - -f
    - live_flv

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[charlesmunger]

I have investigated further and found the issue - for me, at least, the nobuffer and low_delay options were the source of stuttering clips. I believe the reason your clips don't stutter, @flaektrem , is because your input_args override the first -fflags with the second one to specify +genpts+discardcorrupt. That's the general behavior of ffmpeg flags, although I can't prove that's how -fflags works. I now have stutter-free clips with:

  roles:
    - detect
  input_args:
    - -fflags
    - nobuffer
    - -flags
    - low_delay
    - -strict
    - experimental
    - -rw_timeout
    - '5000000'
    - -f
    - live_flv

and

roles:
  - record
input_args: &record_reolink_args
  - -strict
  - experimental
  - -rw_timeout
  - '5000000'
  - -f
  - live_flv

So try this configuration, and you might see lower delay on your detect stream, and a simpler setup for your record stream.

[flaektrem]

Thanks! I will try that.
I didn't know you can have stream specific args. So you set all args for every camera separately?
I have all args globaly set.

What does &record_reolink_args mean?
Maybe you can show me your full config?