Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CDVDVideoCodecDRMPRIME: Fix use of v4l2request hwdevice #25467

Merged
merged 1 commit into from
Dec 21, 2024

Conversation

Kwiboo
Copy link
Member

@Kwiboo Kwiboo commented Jul 9, 2024

Description

This PR extend CDVDVideoCodecDRMPRIME to support using any AVCodecHWConfig that support drm_prime output.

Created as a draft PR because work-in-progress FFmpeg patches are still being worked on.

The reworked v2 patches was posted in early August, a v3 series with minimal changes should reach ffmpeg-devel next few days.

Motivation and context

Current out-of-tree FFmpeg V4L2 Request API hwaccel patches abuse the drm hwdevice type.

Revised v2 patches posted for upstream FFmpeg changed this to use a new dedicated v4l2request hwdevice type.

Use of device ? device : "(null)" to work around string pointer is null exception from libfmt when a nullptr device was being logged.

How has this been tested?

Using work-in-progress FFmpeg patches on a LibreELEC based image.

What is the effect on users?

None, users can use both old out-of-tree FFmpeg V4L2 Request API hwaccel patches or newer reworked patches for upstream FFmpeg.

Screenshots (if appropriate):

Types of change

  • Bug fix (non-breaking change which fixes an issue)
  • Clean up (non-breaking change which removes non-working, unmaintained functionality)
  • Improvement (non-breaking change which improves existing functionality)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that will cause existing functionality to change)
  • Cosmetic change (non-breaking change that doesn't touch code)
  • Student submission (PR was done for educational purposes and will be treated as such)
  • None of the above (please explain below)

Checklist:

  • My code follows the Code Guidelines of this project
  • My change requires a change to the documentation, either Doxygen or wiki
  • I have updated the documentation accordingly
  • I have read the Contributing document
  • I have added tests to cover my change
  • All new and existing tests passed

@Kwiboo Kwiboo force-pushed the v4l2request-hwdevice branch from d612fc9 to 873591e Compare July 9, 2024 15:48
Kwiboo added a commit to Kwiboo/FFmpeg that referenced this pull request Aug 6, 2024
This is a follow up to a very old series from April 2019 [1] and
December 2020 [2], adding V4L2 Request API hwaccels for stateless
decoding of MPEG2, H.264 and HEVC.

These hwaccels has in one form or another been used in LibreELEC
community, nightly and release images since Dec 20th 2018.

HISTORY

The initial v4l2-request hwaccel code was mainly created as a proof of
concept at the end of 2018 by me and Jernej Skrabec. Back when Bootlin's
crowdfunding campaign for upstream Linux kernel driver for Allwinner VPU
had started to bear fruit.

At the time we had very little knowledge on how to properly interact
with any of the V4L2 APIs, and the existing V4L2 M2M code in FFmpeg
seemed very complex to start working with, so we started from scratch.

(That is the main reason why these hwaccels still does not use any of
the existing V4L2 M2M code in FFmpeg.)

The hwaccels had one major limitation, it waited on the kernel to
complete decoding before continuing with next request. Due to hwaccels
still was able to decode up to 4k 50-60fps on Allwinner and Rockchip
boards, time was never spent trying to improve this limitation.

The initial version of these hwaccels was merged as patches into
LibreELEC in April 2019 [3], with my FFmpeg tree as main source for any
update of FFmpeg v4l2-request patches included in LibreELEC.

A RFC was sent in April 2019 [1], with very little feedback and the
required kernel headers not being merged until a year later there was
never any new RFC revision submitted.

After the release of the Raspberry Pi 4 in June 2019 there was hope that
RPi folks would rework and/or improve our proof-of-concept to something
better that could be upstreamed, as history has show this did not happen.
And instead we have just ended up with a second FFmpeg v4l2-request
implementation that has only really been used for HEVC decoding on RPi.

Due to personal reasons I took a long break from doing open-source
contributions during ~2020-2023. During this time Jernej's FFmpeg tree
became the main source for FFmpeg v4l2-request patches in LibreELEC [4].

In December 2020 a v1 of this series was sent [2], at that time kernel
headers for H.264 had been merged into Linux kernel, and I cannot
remember why it never went any further.

Last update to the v4l2-request patches in LibreELEC was done by me in
November 2023 [5].

Along the way there has been noteworthy contributions including from
Boris Brezillon, Ezequiel Garcia, Alex Bee and Benjamin Gaignard.

PRESENT DAY

The version submitted in this series have seen major refactoring to the
common code, to make it more ready for upstreaming and also a rework of
how buffers and requests are handled.

The limitation of waiting for decoding to complete has been removed and
now multiple pending requests can be in-flight at the same time. This
was a requirement to better support HEVC decoding on RPi.

The older versions also incorrectly required use of a DRM hwdevice,
however V4L2 decoding has nothing to do with the DRM subsystem in Linux.

Instead in this version a new V4L2 Request API hwdevice has been added.
Thanks to this it is now possible to specify what media device to use
for decoding, in case multiple decoders exists on a system. E.g. using a
-init_hw_device v4l2request:/dev/media1 parameter.

This version only add support for MPEG2, H264 and HEVC. Support for VP8,
VP9 and AV1 is planned and be added in next revision or in a follow up
series.

HOW TO USE

To use the V4L2 Request API hwaccels you must build FFmpeg on a system
with recent Linux kernel headers, v6.0+. It also requires libdrm and
libudev to successfully build.

This can then be runtime tested on multiple Allwinner and Rockchip
devices. To runtime test this on a RPi 4 or 5 you should use latest
rpi-6.6.y kernel.

  ffmpeg -hwaccel v4l2request -hwaccel_output_format drm_prime \
         -i <input-path> -map 0:v -f null -

This series has been tested with cedrus driver on Allwinner H6, hantro
and rkvdec driver on Rockchip RK3399, and rpivid driver on RPi 4/5.

PRs have also been opened for Kodi and mpv to assist with the transition
from DRM to a new V4L2REQUEST hwdevice type.

- Kodi: xbmc/xbmc#25467
- mpv: mpv-player/mpv#14511

With those PRs applied it should be possible to playback video using
kodi-gbm or mpv, see the PRs above for more details.

It should also be possible to run fluster test suites with following PR:

- fluster: fluendo/fluster#179

I am expecting that there will be a new revision adding VP8, VP9 and AV1
support in a week or two.

Please get back with any type of feedback!

A copy of this series can also be found at [6].

[1] https://lists.ffmpeg.org/pipermail/ffmpeg-devel/2019-April/242316.html
[2] https://lists.ffmpeg.org/pipermail/ffmpeg-devel/2020-December/273579.html
[3] LibreELEC/LibreELEC.tv#3405
[4] https://github.com/LibreELEC/LibreELEC.tv/commits/master/packages/multimedia/ffmpeg/patches/v4l2-request
[5] LibreELEC/LibreELEC.tv#8356
[6] https://github.com/Kwiboo/FFmpeg/commits/v4l2request-2024-v2/

Signed-off-by: Jonas Karlman <[email protected]>
@jernejsk
Copy link
Contributor

This looks good to me and imo it could be merged regardless of ffmpeg patches state. People can invent their own ffmpeg extensions...

@Kwiboo Kwiboo marked this pull request as ready for review September 22, 2024 10:15
@Kwiboo Kwiboo added Type: Improvement non-breaking change which improves existing functionality Platform: Linux labels Sep 22, 2024
@Kwiboo Kwiboo requested a review from lrusak September 22, 2024 17:45
@Kwiboo
Copy link
Member Author

Kwiboo commented Sep 22, 2024

This looks good to me and imo it could be merged regardless of ffmpeg patches state. People can invent their own ffmpeg extensions...

Agree, changes in this PR should be fine regardless of patches for FFmpeg, have removed the draft status of this PR.

Copy link
Contributor

@jernejsk jernejsk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Kwiboo Kwiboo added this to the Piers 22.0 Alpha 1 milestone Sep 25, 2024
@popcornmix
Copy link
Member

I added this to my local build over the weekend on a Pi5. Seemed fine.

Current out-of-tree FFmpeg V4L2 Request API hwaccel use the drm hwdevice
type, however, patches being prepared for upstream FFmpeg changes this
to use a new v4l2request hwdevice type.

Extend support for using any hwconfig that can support drm_prime output.

Signed-off-by: Jonas Karlman <[email protected]>
@Kwiboo Kwiboo force-pushed the v4l2request-hwdevice branch from 873591e to ed32425 Compare October 1, 2024 20:40
@neo1973 neo1973 merged commit b2edd8c into xbmc:master Dec 21, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Platform: Linux Type: Improvement non-breaking change which improves existing functionality v22 Piers
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants