YUV444 and lossless mode with NVENC4

[totaam]

Issue migrated from trac ticket # 796

component: encodings | priority: major | resolution: fixed | keywords: nvenc

2015-01-27 06:11:41: totaam created the issue

---

Split from #466 and #653.

Like it says on the tin.

[totaam]

2015-01-27 10:49:07: antoine uploaded file nvenc-yuv444p-v2.patch (30.7 KiB)

work in progress

[totaam]

2015-01-27 12:15:32: antoine changed status from new to assigned

[totaam]

2015-01-27 12:15:32: antoine commented

---

I just don't see what we're doing wrong.. But only the "Y" plane gets encoded.

The ffmpeg nvenc code does pretty much the same thing: the input buffer is YUV444 with a dstPitch*dst_h offset.

[totaam]

2015-01-28 11:29:59: antoine commented

---

Took a while but the fix was simple (forgot to set the buffer format when registering it - wasn't set in previous NVENC SDK versions).

YUV444 works correctly as of r8571.

Still TODO:

• some client decoding errors on the first frame, ie:

avcodec error decoding 48538 bytes of data with \
    options={'quality': 74, 'encoding': 'h264', 'speed': 39, 'frame': 0, \
             'csc': 'YUV444P', 'pts': 0} (frame 0, step 1 of 1)

• lossless mode
• compare performance with NV12 mode
• we cannot switch from NV12 to YUV444 without doing an encoder re-init..

[totaam]

2015-02-03 07:09:47: totaam commented

---

lossless mode support added in r8604.

BTW, there's a new codec supported (maybe h265 which is meant to be supported in SDK v5 - but which is completely undocumented?):

[0] H264 : 6BC82762-4E63-4CA4-AA85-1E50F321F6BF
[1] None : 790CDC88-4522-4D7B-9425-BDA9975F7603

[totaam]

2015-02-03 09:34:26: antoine uploaded file quality-speed-refactoring.patch (16.6 KiB)

in order to be able to re-initialize the encoder, we need both speed and quality updated at the same time

[totaam]

2015-02-03 10:14:51: antoine uploaded file quality-speed-refactoring-v2.patch (27.6 KiB)

now split init functions so we can re-use them

[totaam]

2015-02-03 13:32:15: antoine commented

---

As of r8606 we can do an encoder re-init on the fly, which only takes a few milliseconds. r8608 allows us to use lossless mode properly, and also now from the system tray menu for h264.

From my limited testing: YUV444 performance is very good, only 10 to 15% slower than NV12, lossless mode is also impressive: fast and we just use more space.

Note: you need the "new" set of license keys to be able to use more than 2 encoding sessions on the same card... and we should probably detect that this is the case and tune the load balancing accordingly.

[totaam]

2015-02-04 11:50:56: antoine changed status from assigned to new

[totaam]

2015-02-04 11:50:56: antoine changed owner from antoine to smo

[totaam]

2015-02-04 11:50:56: antoine commented

---

The decoding problems are fixed in r8620 (was caused by B frames - we will deal with this better eventually, see #800).

Ready for testing. Some defaults to be aware of:

• we switch to YUV444 over 85% quality (but with edge resistance of 10 points both ways) - is this the right value?

XPRA_NVENC_YUV444_THRESHOLD=85

• we use lossless at 100% quality (which means you have to change this value or use the command line or tray to be able to use lossless mode):

XPRA_NVENC_LOSSLESS_THRESHOLD=100

Lossless mode isn't as useful as before, since auto-refresh is now meant to do a decent job of eventually fixing lossy paints. But since it is reasonably cheap to use, maybe we should use it more?

The current pixel format and lossless flag can be seen with (including thresholds as of r8621):

xpra info | grep encoder

@smo: please re-assign to afarr once you've got it setup?

[totaam]

2015-03-24 18:43:30: smo changed owner from smo to afarr

[totaam]

2015-03-24 18:43:30: smo commented

---

I have a reasonable setup now that afarr can test this on.

Re-assigning to him.

[totaam]

2015-03-24 20:12:31: afarr commented

---

Smo's setup against a windows client 0.15.0 r8002 works as expected.

When stretching a firefox window as some video was initiating, there was a second with a yellow and red misrendering, but it quickly solved itself and I couldn't reproduce.

Setting XPRA_NVENC_LOSSLESS_THRESHOLD=100 server side, then setting quality to lossless I was able to toggle lossless to on (window[2].encoder.lossless=1) ... it took several seconds for the heuristics to adjust, but once it had done so, the lossless (and the x264 encoder when I changed the quality to merely Best) rendered exceptionally, even when I stretched the window size to gargantuan proportions on a 4K monitor (window[2].size=(2446, 1883)).

Testing with an osx client, 0.15.0 r8006, it was a little harder to set the quality to 100/lossless (had to use the control channel, since osx menus don't have any hooks for adjusting quality). Again, it took several seconds to adjust, with some odd green and black misrendering and a few seconds more in which only a subregion was responsive to scrolling and such... but once it adjusted itself it also behaved very well.

[totaam]

2015-04-03 02:44:04: afarr changed status from new to closed

[totaam]

2015-04-03 02:44:04: afarr set resolution to fixed

[totaam]

2015-04-03 02:44:04: afarr commented

---

Ok, this seems to be good enough for current purposes. Closing.