x264 10-bit support

[totaam]

As per this test report.
x264 performs very well with 10 bit data.

The big difficulty is that this is a compile time switch so we would need a different build of the x264 libraries with 10-bit support...
And the symbols would likely clash with the 8-bit ones.
So we would need to choose which one to load early.

[totaam]

This is no longer a compile time switch: x264 supports both 8-bit and 10-bit outputs, and you don't have to do anything special: Previously you had to compile x264 with --bit-depth=10, and then link your ffmpeg to either an 8-bit or 10-bit libx264, but that is now unnecessary

Now found in x264_param_t: i_bitdepth.

The only colorspace type that explicitly mentions 10-bit is X264_CSP_V210: X264_CSP_V210.
So maybe we'll have to upsample and use the X264_CSP_HIGH_DEPTH bit: the csp has a depth of 16 bits per pixel component. BGR(A) and RGB should be trivial to convert to 48 / 64-bit, though it's not clear what sort of bit layout it will be expecting for I420 (aka YUV420P) or I444.

[totaam]

Some preliminary / related changes and minor tweaks in r26894, r26895, r26896, r26897, r26898.

The patch attached seems to do something: we convert the r210 image to 48-bit per pixel (via r210_to_rgb48) and x264 accepts it.

Still TODO:

• the CSC is inlined into enc_x264 (slow: just cython argb code), needs to be moved to csc_swscale
• client side will need CSC no matter what: non-opengl case will need downsampling to 24-bit, opengl can handle higher bit depths but the data from dec_avcodec may need a CSC step (what is the bit layout of YUV422P10LE?) and also changes to the texture upload code (can't use BYTE..) - hopefully no changes to the shaders?

[totaam]

Related changes:

• more bit depths with opengl #2826 more bit depths with opengl
• x264 and ffmpeg RPM conflicts #2827 x264 and ffmpeg RPM conflicts
• r26900 better decoder assert message

Updated patch will follow.

[totaam]

2020-07-06 17:27:01: antoine uploaded file x264-10bit.patch (14.8 KiB)

work in progress

[totaam]

Done in r26908.
Verified that the 10-bit h264 stream we generate is correct using:

XPRA_SAVE_TO_FILE=test10bit xpra start --start=xterm --pixel-depth=30

And then attaching a client with:

xpra attach --no-mmap --pixel-depth=30 --encodings=h264 --opengl=force

---

TODO:

• decode and render in opengl: render GBRP10 in opengl client #2828
• replace r210_to_bgr48 hack call with a proper csc step (swscale?)

[totaam]

2020-07-09 11:38:55: antoine uploaded file csc-cython-r210.patch (14.5 KiB)

restore csc_cython so we have a r210 csc module

[totaam]

r26940 + r26941 + r26943: do the csc step via csc_cython

This will do for this ticket.

[totaam]

Fixed for csc in r26960.