AVC LP Multi-Slice Encode corrupted #389

uartie · 2018-10-29T21:37:30Z

Encoding h264 low-power with multi-slice produces corrupted output.

Reproduce Steps:

In gst-msdk:

GST_VAAPI_ALL_DRIVERS=1 LIBVA_DRIVER_NAME=iHD gst-launch-1.0 videotestsrc num-buffers=100 \
  ! video/x-raw,format=NV12 ! msdkh264enc rate-control=cqp qpp=28 target-usage=4 \
  gop-size=30 num-slices=4 b-frames=0 hardware=true low-power=1 \
  ! video/x-h264,profile=high ! h264parse ! filesink location=test.h264

In gst-vaapi:

GST_VAAPI_ALL_DRIVERS=1 LIBVA_DRIVER_NAME=iHD gst-launch-1.0 videotestsrc num-buffers=100 \
  ! video/x-raw,format=NV12 ! vaapih264enc rate-control=cqp init-qp=28 quality-level=4 \
  keyframe-period=30 num-slices=4 max-bframes=0 tune=3 \
  ! video/x-h264,profile=high ! h264parse ! filesink location=test.h264

In ffmpeg-vaapi:

LIBVA_DRIVER_NAME=iHD ffmpeg -hwaccel vaapi -vaapi_device /dev/dri/renderD128 -v verbose \
  -f rawvideo -pix_fmt yuv420p -s:v 1280x720 -i input.yuv -vf 'format=nv12,hwupload' \
  -c:v h264_vaapi -low_power 1 -profile:v high -g 30 -qp 28 -bf 0 -quality 4 -slices 4 \
  -vframes 100 -y test.h264

Playback test.h264:

LIBVA_DRIVER_NAME=iHD GST_VAAPI_ALL_DRIVERS=1 gst-launch-1.0 filesrc location=test.h264 \
  ! h264parse ! vaapih264dec ! vaapisink

See more details in https://trac.ffmpeg.org/ticket/7523

Software Stack:

libva (master) heads/master-0-g285267586a3d  https://github.com/01org/libva
gmmlib (master) heads/master-0-gaedd82e2da01  https://github.com/intel/gmmlib
intel-media-driver (master) heads/master-0-g9d258e13eba1  https://github.com/intel/media-driver
gstreamer (master) heads/master-0-g3eb334a7f9a5  https://anongit.freedesktop.org/git/gstreamer/gstreamer
gst-plugins-base (master) heads/master-0-ga15baf797603  https://anongit.freedesktop.org/git/gstreamer/gst-plugins-base
gst-plugins-good (master) heads/master-0-g560a6155bd5b  https://anongit.freedesktop.org/git/gstreamer/gst-plugins-good
gst-plugins-bad (master) heads/master-0-gafbbc3a97ec3  https://anongit.freedesktop.org/git/gstreamer/gst-plugins-bad
gstreamer-vaapi (master) heads/master-0-gd8442e479ea8  https://anongit.freedesktop.org/git/gstreamer/gstreamer-vaapi
ffmpeg (master) heads/master-0-g4a6c286acc45  https://git.ffmpeg.org/ffmpeg

The text was updated successfully, but these errors were encountered:

wangyan42164 · 2018-10-31T08:35:44Z

@uartie Could you please clarify it. GST is OK and FFMPEG fails? Or both fails. And what platform do you use? KBL or SKL?
Thanks

uartie · 2018-10-31T15:59:48Z

It fails with both FFMPEG and GST (i.e. all commands above under "Reproduce Steps"). I am using KBL locally, but our CI shows same issue exists on KBL, CFL and WHL.

uartie · 2018-10-31T16:04:11Z

I have not tested it on SKL since the intel-media-driver docs don't include AVC E2 (LP) support.

wangyan42164 · 2018-10-31T23:33:36Z

@uartie Thanks for your clarifying. I will check it.

mypopydev · 2018-11-03T03:11:41Z

@wangyan42164 @uartie SKL failed too

wangyan42164 · 2018-11-06T01:24:45Z

@uartie @mypopydev I am debugging #361. It will cause GPU hang when encoding the first frame. I am not sure whether this issue is similar with it. I will check it.

XinfengZhang · 2018-11-12T12:59:16Z

@wangyan42164 please help to check the slice structure , for Low power mode, iHD just support equal_rows slice structure.

wangyan42164 · 2018-11-15T03:04:17Z

@XinfengZhang @uartie @mypopydev
In fact, equal_row slice may be too strict. It should be enough for row aligned. And current media driver will force one slice for one row if gstreamer/ffmpeg send non row-aligned slice parameter.
The root cause is m_isVdencSuperSliceEnabled is enabled in media driver when using low power encoding. For super slice, first_mb_in_slice of slice header should be zero always.
I have submitted one patch for gst-vaapi: https://gitlab.freedesktop.org/gstreamer/gstreamer-vaapi/merge_requests/15
ffmpeg should be similar. I will also submit one for this.
msdk may also need one? I will check it.
I can also submit one PR/patch for media driver for ensuring row aligned slice. After this, it will cause gstreamer/ffmpeg encoding error report.
Thanks.

xhaihao · 2018-11-15T04:09:36Z

How is the middle-ware aware of the driver's limitation? Your patch (https://gitlab.freedesktop.org/gstreamer/gstreamer-vaapi/merge_requests/15) doesn't make sense to me

xhaihao · 2018-11-15T04:30:39Z

it makes sense first_mb_in_slice is 0 for the first slice, but for other slices, first_mb_in_slice shouldn't be 0.

wangyan42164 · 2018-11-15T05:03:47Z

@xhaihao @XinfengZhang Could we add one query API into libva for super slice usage? We may need more discussion.

uartie · 2018-11-30T19:29:10Z

Is there an easy, temporary fix/wa at the driver level that we can apply without needing new VA API?

uartie · 2018-12-19T17:30:51Z

any update?

uartie · 2019-04-30T16:04:12Z

This seems to be fixed in gst-msdk and ffmpeg-qsv, but gst-vaapi and ffmpeg-vaapi still has this issue. I'm not sure when or what fixed this in gst-msdk/ffmpeg-qsv. My current stack is:

libva (master) heads/master-0-g4d9d819d4859 
gmmlib (master) heads/master-0-g0e5e49651e7d 
intel-media-driver (master) heads/master-0-g8018ad53fc9e 
msdk (master) heads/master-0-g2f0a8af81330 
ffmpeg (master) heads/master-0-gd0e4d0429ed2
gstreamer (master) heads/master-0-gab60a330db71 
gst-plugins-base (master) heads/master-0-g123c2f8b5dd4 
gst-plugins-good (master) heads/master-0-g037d70c01bbe 
gst-plugins-bad (master) heads/master-0-g3a0a82a783b2 
gstreamer-vaapi (master) heads/master-0-ga3ce4446c168

Can someone confirm and/or point to commit that fixed it in above 2 cases?

uartie · 2019-04-30T16:32:08Z

@dvrogozh are you aware of any specific fix/commit in the msdk component that would have fixed/workaround this issue in gst-msdk and ffmpeg-qsv (see my last comment).

XinfengZhang · 2019-07-29T02:22:41Z

how about driver handle it, such as decode the slice parameter from the packed header then pack it again inside driver？

wangyan-intel · 2019-07-29T03:13:17Z

@XinfengZhang Yes. I also think so. I am working on this.

AVC VDENC when multi-slice encoding need force first_mb_in_slice to 0. Fixes intel#389. Signed-off-by: Yan Wang <[email protected]>

wangyan42164 · 2019-07-30T08:52:32Z

@uartie I have submitted #697 to fix it on media driver side. Now ffmpeg/gst-vaapi can work fine for the case. BTW, I find that MSDK also set first_mb_in_slice to 0 in his implementation. So ffmpeg/gst use MSDK has not this issue.

Now I tried your decoding command to decode the output H264 stream. It displays rightly.
But I still get failure on my local KBL platform by your https://github.com/intel/vaapi-fits like the following:
./vaapi-fits run test/gst-vaapi/encode/avc.py:cqp:lp.test -k 'slices=4'
It shows PSNR difference:
148727 [2019-07-30 07:34:38.438865] NOTICE: slash: DETAIL: time(gst:2) = 11.8865s
148728 [2019-07-30 07:34:38.506857] NOTICE: slash: DETAIL: time(psnr:1) = 0.0678s
148729 [2019-07-30 07:34:38.507039] NOTICE: slash: DETAIL: drv.iHD.psnr:expect = None
148730 [2019-07-30 07:34:38.507113] NOTICE: slash: DETAIL: drv.iHD.psnr:actual = [40.9678, 60.4905, 58.777, 56.6767, 61.0828, 59.622]
148731 [2019-07-30 07:34:38.528668] TRACE: slash.core.error: Error added: AssertionError: Invalid reference value
I am not sure the path of generating the golden data? We may need compare it and AVC VDENC.

Currently AVC VDENC will adjust slice layout dynamically if ffmpeg/gst-vaapi set multi-slice but cannot meted.
E.g. When set slices = 4, resolution=320x240, AVC VDENC will changes to row slice layout automatically.
If need 4 slices, you have to set the resolution to 320x256. MSDK will do this padding.
So I think ffmpeg/gst-vaapi need some additional logic. I will try to fix it in ffmpeg/gst-vaapi.
Thanks

Yan Wang

uartie · 2019-08-16T16:49:39Z

@wangyan42164 I can update vaapi-fits once your patch is merged.

zhuqingliang · 2019-09-05T02:48:45Z

some avc cbr vdenc still failed on my ICL:
Cmd: ffmpeg -hwaccel vaapi -vaapi_device /dev/dri/renderD128 -v verbose -f rawvideo -pix_fmt yuv420p -s:v 720x480 -r:v 30 -i ./yuv/720x480p.yuv -vf 'format=nv12,hwupload' -c:v h264_vaapi -profile:v high -rc_mode CBR -g 60 -slices 3 -b:v 2000k -maxrate 2000k -low_power 1 -vframes 150 -y ./480p-cbr-high-30-60-3-2000k-2000k-1.h264

message:
[h264_vaapi @ 0x118ab40] Failed to end picture encode issue: 24 (internal encoding error).
[h264_vaapi @ 0x118ab40] Encode failed: -5.
Video encoding failed

AVC VDENC when multi-slice encoding need force first_mb_in_slice to 0. Fixes #389. Signed-off-by: Yan Wang <[email protected]>

AVC VDENC when multi-slice encoding need force first_mb_in_slice to 0. Fixes intel#389. Signed-off-by: Yan Wang <[email protected]>

AVC VDENC when multi-slice encoding need force first_mb_in_slice to 0. Fixes #389. Signed-off-by: Yan Wang <[email protected]>

uartie mentioned this issue Dec 5, 2018

[gstreamer-vaapi][avc vdenc encode]the encoded file has garbage #441

Closed

uartie mentioned this issue Apr 24, 2019

[avc vdenc] multislice encode file have garbage and decode error #612

Closed

wangyan42164 mentioned this issue Jul 30, 2019

Adjust packed slice header for AVC VDENC when multi-slice encoding. #697

Merged

XinfengZhang assigned wangyan-intel Aug 29, 2019

XinfengZhang mentioned this issue Oct 12, 2019

release scope discussion for 19.4 #748

Closed

intel-mediadev closed this as completed in #697 Oct 12, 2019

intel-mediadev pushed a commit that referenced this issue Oct 12, 2019

Adjust packed slice header for AVC VDENC when multi-slice encoding.

8c83e39

AVC VDENC when multi-slice encoding need force first_mb_in_slice to 0. Fixes #389. Signed-off-by: Yan Wang <[email protected]>

wangyan42164 mentioned this issue Nov 21, 2019

Adjust packed slice header for AVC VDENC when multi-slice encoding. #785

Merged

intel-mediadev pushed a commit that referenced this issue Nov 26, 2019

Adjust packed slice header for AVC VDENC when multi-slice encoding.

4073674

AVC VDENC when multi-slice encoding need force first_mb_in_slice to 0. Fixes #389. Signed-off-by: Yan Wang <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

AVC LP Multi-Slice Encode corrupted #389

AVC LP Multi-Slice Encode corrupted #389

uartie commented Oct 29, 2018

wangyan42164 commented Oct 31, 2018

uartie commented Oct 31, 2018

uartie commented Oct 31, 2018

wangyan42164 commented Oct 31, 2018

mypopydev commented Nov 3, 2018

wangyan42164 commented Nov 6, 2018 •

edited

Loading

XinfengZhang commented Nov 12, 2018

wangyan42164 commented Nov 15, 2018 •

edited

Loading

xhaihao commented Nov 15, 2018

xhaihao commented Nov 15, 2018

wangyan42164 commented Nov 15, 2018

uartie commented Nov 30, 2018

uartie commented Dec 19, 2018

uartie commented Apr 30, 2019

uartie commented Apr 30, 2019 •

edited

Loading

XinfengZhang commented Jul 29, 2019

wangyan-intel commented Jul 29, 2019

wangyan42164 commented Jul 30, 2019

uartie commented Aug 16, 2019

zhuqingliang commented Sep 5, 2019

AVC LP Multi-Slice Encode corrupted #389

AVC LP Multi-Slice Encode corrupted #389

Comments

uartie commented Oct 29, 2018

wangyan42164 commented Oct 31, 2018

uartie commented Oct 31, 2018

uartie commented Oct 31, 2018

wangyan42164 commented Oct 31, 2018

mypopydev commented Nov 3, 2018

wangyan42164 commented Nov 6, 2018 • edited Loading

XinfengZhang commented Nov 12, 2018

wangyan42164 commented Nov 15, 2018 • edited Loading

xhaihao commented Nov 15, 2018

xhaihao commented Nov 15, 2018

wangyan42164 commented Nov 15, 2018

uartie commented Nov 30, 2018

uartie commented Dec 19, 2018

uartie commented Apr 30, 2019

uartie commented Apr 30, 2019 • edited Loading

XinfengZhang commented Jul 29, 2019

wangyan-intel commented Jul 29, 2019

wangyan42164 commented Jul 30, 2019

uartie commented Aug 16, 2019

zhuqingliang commented Sep 5, 2019

wangyan42164 commented Nov 6, 2018 •

edited

Loading

wangyan42164 commented Nov 15, 2018 •

edited

Loading

uartie commented Apr 30, 2019 •

edited

Loading