Create profile presets for re-encoding stream to VP9 or AV1 for any platform that can support it

[Xev]

I want to record cameras that provide only H264 and H265 streams, in VP9 now and AV1 soon.

Main reason is space-saving and that most desktops and portable devices have hardware acceleration support for VP9, and will have for AV1 soon.
That means that if I setup in config for detection role to use regular h264 stream, and set up recording role with the same stream but with preset for ffmpeg for re-encoding, then I should have recordings on hard drive in format that I can play in browsers on any desktop or phone.

I don't want to wait eons for camera manufacturers to start selling cameras that support VP9 or AV1 codec, and especially I don't want to wait even more years on top of that for those cameras to get reasonably priced.

Main fault of h265 which modern cameras readily provide is that most browsers don't have license to decode it, which means even if I correctly feed frigate h265 stream, I won't be able to browse recordings in frigate's web interface, rendering those recordings useless.

Currently I am forced to store h264 recordings on hard drive, which take a lot of space if they are in good quality.
With how good Frigate got with support for graphic cards, I assume based on my rudimentary knowledge that a lot of models (of intel cpus, amd cpus, nvidia gpus, amd gpus, intel arc) already support hardware decoding and encoding of VP9, and linux kernels and ffmpegs version had support for it for ages.
Which means for anyone that wants to save space, it should be as easy to purchase for example Nvidia GPU starting from 900 series or intel cpu starting from 10th gen, install it and add preset to config.

Drivers in linux kernels and ffmpeg versions that can support gpus and cpus that support hardware encoding of AV1 should start appearing soon, even if they are not already there.
Then it will be as simple as switching profile preset in config from VP9 to AV1, and everything should still work flawlessly?

Why wait for industry to play catch up, when we can go ahead of curve by ourselves?

[nglessner]

My wish list:

camera streams to frigate: H264
frigate live streaming, object detection, etc: H264
frigate 24/7 recording: transcoding to H265, AV1 or VP9 using intel quick sync

[moandcompany]

Your wishlist for re-encoding is probably best handled through go2rtc's features Docs

The goal for re-encoding of recordings can effectively be done through transcoding of the actual camera stream to your desired codec (i.e. H265) via ffmpeg as used by go2rtc.

[nglessner]

Thanks @moandcompany . I tried this configuration (and many variations of it) with little luck. Oddly, the h264_2mbps custom profile with the added #hardware tag on the end of the stream definition did show gpu usage (10-15% range). The h264 and h265 software profiles worked, but as expected with cpu only. Neither of the custom hardware acceleration profiles worked. The log would say "invalid data found when processing input".

At this point I'm expected the issue is either related to the N305's capabilities or that the camera is a Reolink doorbell camera (I have tried both rtsp and rtmp streams). It's also possible that I have something configured incorrectly as well. Does the igpu driver on the proxmox host need to match the version listed in vapi on the docker container?

[NickM-27]

no, the versions don't need to match.

[nglessner]

Oh and is there a better source for ffmpeg settings than their site? Something that cross references hardware and platforms? I've tried looking, but came up empty.

[NickM-27]

I learned off of their site along with searching different things

[moandcompany]

Since you're trying to use QuickSync , perhaps the following doc page would be useful:
https://trac.ffmpeg.org/wiki/Hardware/QuickSync#Transcode

Here is one of their examples using qsv for a hardware accelerated hevc/h265 -> hardware accelerated h264 transcode.

ffmpeg -hwaccel qsv -c:v hevc_qsv -i input.mp4 -vf 'vpp_qsv=framerate=60,scale_qsv=w=1920:h=1080' -c:v h264_qsv output.mp4

[mike2307]

Did anyone had success with configuring go2rtc for transcoding to AV1 or VP9 and can provide a reference config?
Thanks in advance :)

[Arol450]

ffmpeg:
hwaccel_args: preset-intel-qsv-h264

go2rtc:
streams:
Cars_live:
- ffmpeg:rtsp://:@192.168.2.44/Streaming/Channels/1

cameras:
Cars:
ffmpeg:
output_args:
record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v av1_qsv -global_quality:v 31 -an
inputs:
- path: rtsp://localhost:8554/Cars_live
input_args: -loglevel error -hwaccel qsv -qsv_device /dev/dri/renderD128 -hwaccel_output_format qsv -c:v h264_qsv
roles:
- record
- path: rtsp://:@192.168.2.44/Streaming/Channels/102
roles:
- detect
detect:
enabled: true
width: 640
height: 480
motion:
threshold: 30
contour_area: 10
improve_contrast: true
live:
stream_name: Cars_live

[coledeck]

I wanted to report success of using AV1 encoding and OpenVino on an intel arc a310. I am running inside a docker container, inside LXC, on proxmox. Thanks @Arol450 for a good starting point.

Some of my shittier cameras do not work with AV1 re-encode, the encoder crashes. They are exceptionally shitty cameras from 2014 and can only run at 5fps @ 960p so I don't really care anyway. My other cameras work fine.

File sizes went massively down. From 100MB for a 30 second clip (the cameras have genuinely awful encoders) to a whopping 2MB with AV1 and no discernible quality loss. Super critical for me as my ISP upload speed is only 25mbps, so now we can actually look at the cameras remotely! Is there a way to re=encode live streams? My Arc a3130 is at 2% usage with 7 cameras... I can spare the resouces to re-encode but I didn't see anything relating to live in the config reference? Might have to dig in with go2rtc or something.

My cameras are h.264 AXIS cameras, 1280x960 at 15 fps.

Configs:

Proxmox:

Latest proxmox 8.3.2 with latest kernel for good intel GPU driver support

In LXC config: (all of the cgroup / apparmor are almost certainly not be necessary, it is leftover from google coral setup, main bit is the devices)

...
lxc.apparmor.profile: unconfined
lxc.cgroup2.devices.allow: a
lxc.cap.drop:
lxc.cgroup2.devices.allow: c 120:* rwm
dev0: /dev/dri/card0,gid=44,uid=0
dev1: /dev/dri/renderD128,gid=105,uid=0

Running a debian unprivileged container.

docker-compose.yml

...
devices:
  - /dev/dri
cap_add:
  - CAP_PERFMON

Frigate global config:

ffmpeg:
  hwaccel_args: preset-intel-qsv-h264

Per-camera, for the cameras where it worked:

    ffmpeg:
      output_args:
        record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v av1_qsv -global_quality:v 31 -an

I did not need the input_args, it actually caused issues setting it.

Setting the output_args globally would have worked if not for my shitbox cameras that do something funky enough that ffmpeg and qsv lose their minds and frigate shuts down. With only the slightly-less-shit cameras, it works fine.

[Arol450]

The performance gain of AV1 video storage is amazing, isn't it? Even on a slow remote VPN connection, video playback of the stored files can be accelerated easily to x8 or even x16. And the files are so astonishing small, with absolutely no loss in video quality.

I initially tried to use go2rtc to transcode the video stream(s). Sadly ffmpeg which is being invoked by go2rtc to do that, does NOT support av1 video format for the rtsp transport protocol (yet). I tried to work around this trying out all kinds of other formats, even tried to use pipes to do that.

Hello, FFmpeg doesn't support publishing AV1 with RTSP yet. Open an issue in the FFmpeg bug tracker.

Originally posted by @aler9 in #2261

People have been working around this limitation using VLC or gstreamer, to stream AV1 rtsp from a ffmeg AV1 transcoded pipe.

---

Ideally I would like to be able to grab the stream with ffmpeg and decode with qsv(h264/5). Then use a ffmpeg filter_complex to invoke qsv hardware filters, two copies, one hardware-resized for further detection and processing, raw data output into a pipe for frigate; the other one would be av1 encoded and being used for recording and live view purposes.
This way only one ffmpeg instance is running per each camera, not two. And the hardware video handling is so impressive, one task per ffmpeg instance would be enough, not two as it is the default in frigate these days. (4 tasks per camera total atm)
Maybe throw in a hardware overlay filter for timestamps.
And then - finally - the need for dual streams coming from the cameras would be gone. This would save approximately 0.8W electrical power per camera, at least with my HikVisions. That adds up quickly, over time, and number of cameras.

I played around a bit with the scale_qsv hardware filter. The scaled down version is quite a bit better than what the camera does. I wonder whether this would improve the object detector quality as well?

Just my $0.02.
But I am glad you got it working, too.

[NickM-27]

I played around a bit with the scale_qsv hardware filter. The scaled down version is quite a bit better than what the camera does. I wonder whether this would improve the object detector quality as well?

Frigate already uses qsv_scale

Also, ffmpeg 7 supports qsv av1 I believe, frigate frigate 0.15 uses by default