Fail to initialise on ubuntu 20.04 on M3-8100

[Changliu52]

Hi @bmegli
I was testing the on the 20.04 on M3-8100Y.
I have set LIBVA_DRIVER_NAME=i965

the encoder seems to still have problem to initialise. The console output is:

AVHWDeviceContext @ 0x55af43cac0c0] Trying to use DRM render node for device 0.
[AVHWDeviceContext @ 0x55af43cac0c0] libva: VA-API version 1.7.0
[AVHWDeviceContext @ 0x55af43cac0c0] libva: User environment variable requested driver 'i965'
[AVHWDeviceContext @ 0x55af43cac0c0] libva: Trying to open /usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so
[ INFO] [1599692843.153209344]: Device with serial number 938422076491 was found.

[ INFO] [1599692843.153343191]: Device with physical ID 2-2-2 was found.
[ INFO] [1599692843.153456225]: Device with name Intel RealSense D435 was found.
[ INFO] [1599692843.153897987]: Device with port number 2-2 was found.
[ INFO] [1599692843.156157938]: getParameters...
[AVHWDeviceContext @ 0x55af43cac0c0] libva: Found init function __vaDriverInit_1_6
[AVHWDeviceContext @ 0x55af43cac0c0] libva: va_openDriver() returns 0
[AVHWDeviceContext @ 0x55af43cac0c0] Initialised VAAPI connection: version 1.7
[AVHWDeviceContext @ 0x55af43cac0c0] VAAPI driver: Intel i965 driver for Intel(R) Kaby Lake - 2.4.0.
[AVHWDeviceContext @ 0x55af43cac0c0] Driver not found in known nonstandard list, using standard behaviour.
[h264_vaapi @ 0x55af43cf9680] Input surface format is nv12.
[h264_vaapi @ 0x55af43cf9680] Using VAAPI profile VAProfileH264ConstrainedBaseline (13).
[h264_vaapi @ 0x55af43cf9680] Using VAAPI entrypoint VAEntrypointEncSlice (6).
[h264_vaapi @ 0x55af43cf9680] Using VAAPI render target format YUV420 (0x1).
[h264_vaapi @ 0x55af43cf9680] RC mode: VBR.
[h264_vaapi @ 0x55af43cf9680] RC target: 50% of 2000000 bps over 500 ms.
[h264_vaapi @ 0x55af43cf9680] RC buffer: 1000000 bits, initial fullness 750000 bits.
[h264_vaapi @ 0x55af43cf9680] RC framerate: 30/1 (30.00 fps).
[h264_vaapi @ 0x55af43cf9680] Using intra and P-frames (supported references: 4 / 1).
[h264_vaapi @ 0x55af43cf9680] All wanted packed headers available (wanted 0xd, found 0x1f).
[AVHWDeviceContext @ 0x55af43d1d580] Trying to use DRM render node for device 0.
[AVHWDeviceContext @ 0x55af43d1d580] libva: VA-API version 1.7.0
[AVHWDeviceContext @ 0x55af43d1d580] libva: User environment variable requested driver 'i965'
[AVHWDeviceContext @ 0x55af43d1d580] libva: Trying to open /usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so
[AVHWDeviceContext @ 0x55af43d1d580] libva: Found init function __vaDriverInit_1_6
[AVHWDeviceContext @ 0x55af43d1d580] libva: va_openDriver() returns 0
[AVHWDeviceContext @ 0x55af43d1d580] Initialised VAAPI connection: version 1.7
[AVHWDeviceContext @ 0x55af43d1d580] VAAPI driver: Intel i965 driver for Intel(R) Kaby Lake - 2.4.0.
[AVHWDeviceContext @ 0x55af43d1d580] Driver not found in known nonstandard list, using standard behaviour.
[h264_vaapi @ 0x55af43d3ee40] Input surface format is bgr0.
[h264_vaapi @ 0x55af43d3ee40] No usable encoding profile found.
hve: cannot open video encoder codec
nhve: failed to initalize hardware encoder
[ERROR] [1599692843.173446829]: nhve is unable to initialise!

[Changliu52]

The implementation works fine on M3-7Y30. For some reason, the bgr0 format doesn't seem to work on m3-8100Y

[bmegli]

Hi @Changliu52,

Responding to this issue will take some time (I am busy with some deadlines at work, I have to assemble test environment with 20.04, etc).

Which example of RNHVE are you calling? realsense-nhve-h264?

If it is H.264 (not HEVC Main10) it should work with both i965 and the default driver (newer).

[Changliu52]

Thanks for the prompt reply.
I initialised the NHVE library as follows:

        net_config_ = {0};
	net_config_.ip = ip_char;
	net_config_.port = (uint16_t)port_depth_;

     	hw_config_.pixel_format = "bgr0";
	hw_config_.profile = FF_PROFILE_H264_CONSTRAINED_BASELINE; //FF_PROFILE_H264_MAIN, FF_PROFILE_H264_CONSTRAINED_BASELINE, FF_PROFILE_H264_BASELINE
	hw_config_.encoder = NULL;
	hw_config_.width = image_width_;
	hw_config_.height= image_height_;
	hw_config_.framerate = 30;
	hw_config_.max_b_frames = 0;
	hw_config_.device = NULL; //NULL as last argv argument, or device path
	hw_config_.bit_rate = 1000000;
	if( (streamer_color_ = nhve_init(&net_config_, &hw_config_)) == NULL )
		return 1; // failure

I have a separate depth streamer NHVE initiated in nv12 format, which seem to have worked according to the console output.
The only hw_config_ difference is that pixel_format. Not sure why the bgr0 can cause No usable encoding profile found. error.

[bmegli]

@Changliu52,

I confirm that in 20.04 (or rather with 20.04 FFmpeg version) there is a problem with non nv12/yuv420p/p010le pixel formats.

Looking at FFmpeg code this seems to be the commit where behavior changed, some notes are also here.

My shallow understanding after peeking into code is that smart profile auto-detection code tries to match software pixel format with compatible VAAPI profile. I am guessing here that it fails because there is no profile for bgr0/rgb0 (those were hardware pixel format conversions, under the hood it was loosing some information when converting 4:4:4 chroma of bgr0 to 4:2:0 of nv12).

Possible ways of solving this:

1. Just use nv12 or yuv420p if applicable
2. Try to hwupload the software pixel format instead of directly passing it to encoder, possibly (hardware?) convert it, maybe the avfilter hwupload path will not have this problem.
3. Try to fix avcodec so that it again accepts other pixel formats

Since I have only looked at it, I may be wrong. It would be nice to check if it works with bgr0 on other generations than AmberLakeY (which M3-8100 is).

The funny thing is that avcodec, when queried for acceptable formats for VAAPI still reports bgr0.

[Changliu52]

Hi @bmegli,
Really appreciate your detailed feedback. It looks exactly as the problem I experianced.
Seems using bgr0 format is not supported by the VAAPI anymore.

Do you recommend any method to convert bgr0 data to yuv420p? The opencv Im using only offers the conversion for the other way round.

Im also looking at testing the new Lattepanda delta https://www.dfrobot.com/product-1908.html?tracking=5f61ea809f1a0. maybe this one can work?

[bmegli]

Hi @Changliu52,

I have just checked it on KabyLake in 20.04 (FFmpeg 4.2.4) with the same result for bgr0. This confirms that it is not really AmberLake Y(m3-8100) problem but rather software layer problem.

Seems using bgr0 format is not supported by the VAAPI anymore.

Precisely speaking the change is probably on the FFmpeg level, not the VAAPI level.

Do you recommend any method to convert bgr0 data to yuv420p? The opencv Im using only offers the conversion for the other way round.

Probably it should be just CV_BGR2YUV_YV12 or less likely CV_BGR2YUV_I420. Are you certain your OpenCV doesn't support it?

Short term - any method is ok.

Long term:

• understanding what and why really happened (is it FFmpeg regression?)
• fixing or having fixed FFmpeg if it is regression
• if not possible long term workaround performing hardware conversion

With optimized pixel format conversion (I expect OpenCV has such) you will probably not see any difference from hardware conversion (apart from a bit higher CPU usage).

Im also looking at testing the new Lattepanda delta https://www.dfrobot.com/product-1908.html?tracking=5f61ea809f1a0. maybe this one can work?

As above, this doesn't seem to be caused by hardware, rather by software (FFmpeg version).

Delta possibly may not support HEVC Main10 encoding (used for depth encoding).

---

Edits:

• m3-7y30 -> m3-8100 (mistake)

[bmegli]

By using grep on FFmpeg releases we see that this commit was introduced between 4.0 and 4.1.

git status
On branch release/4.1
Your branch is up to date with 'origin/release/4.1'.

git log | grep "vaapi_encode: Choose profiles dynamically"
    vaapi_encode: Choose profiles dynamically
# nothing like that in previous releases

By browsing 2018 August ffmpeg-devel mailing list archive we see that the commit was introduced as one of 41 smaller patches reflecting larger effort in FFmpeg VAAPI support.

The fact that it is 4.1 release means that there is no trivial way to check it (e.g. Ubuntu 18.04 is FFmpeg 3.4, Ubuntu 19.04 is already at 4.2).

[bmegli]

The implementation works fine on M3-7Y30. For some reason, the bgr0 format doesn't seem to work on m3-8100Y

Probably it doesn't work with bgr0 starting from FFmpeg 4.1 (so both in Ubuntu 19.04 and 20.04)

[bmegli]

@Changliu52

I have filed a bug on FFmpeg bug tracker for this.

Upvote it if you are interested in having this fixed/investigated.

[Changliu52]

@bmegli Appreciate the feedback and details. Really keen to see they could bring back the rgb0 support.

[Changliu52]

not sure how to up vote it? do I have to register an account?

[bmegli]

do I have to register an account?

Yes, exactly.

Also try the CV convertion:

Do you recommend any method to convert bgr0 data to yuv420p? The opencv Im using only offers the conversion for the other way round.

Probably it should be just CV_BGR2YUV_YV12

[bmegli]

From discussion in FFmpeg trac it looks like the long long-term solution will be to invoke vaapi_scale filter for hardware color conversion + some logic to decide when such conversion is needed.

HVE master already uses vaapi_scale filter (not for color conversion, for scaling if requested by user).
NHVE and RNHVE both use older version of HVE without scaling (thorugh vaapi_scale) support.

So the roadmap would be:

• test accelerated conversion in HVE
• add logic when to invoke conversion in HVE
• bump HVE verstion in NHVE, and NHVE version in RNHVE