nvenc v6 support

[totaam]

https://developer.nvidia.com/nvidia-video-codec-sdk new features that might be of interest to us:

• Unified SDK for video encoding and decoding - so we can decode, even on win32, without needing another library
• Support for input surfaces in RGB format - saves doing the CSC step using CUDA
• "Various quality and performance improvements in encoding"
• GPUs supported for H.265 (HEVC) encoding:

• GeForce GTX 960, GTX 980. GTX Titan X
• Quadro M4000, M5000, M6000
• Tesla M4, M6, M60

HEVC hardware encoding would be worth testing.

[totaam]

Initial support added in r11471.

Summary of the most noteworthy changes (excluding various indentation changes and typos we can ignore, new constants, etc):

• NVENCAPI_STRUCT_VERSION is packed differently now - genius, so version 6.0 looks like 0.6 (wtf!)
• function version packing has also changed, causes lots of noise in the diff
• added NV_ENC_H264_PROFILE_PROGRESSIVE_HIGH_GUID
• removed all the tiled picture formats, which never worked anyway (always looked like a bug to me)
• added ARGB, ARGB10 and AYUV - so it looks like we can do transparency?
• _NV_ENC_CONFIG_HEVC has changed, a lot
• talks about MV buffers and ME only mode

It looks like a GTX 980 or newer is required for testing HEVC..

[totaam]

Improvements and fixes in r11815, r11816, r11817 (most of which should be backported)

Support code added in r11818 so we can at least try to probe HEVC... (still failing).
Will need to compare API parameters between the C client and the Cython version.
This works:

./NvEncoder -codec 1 -i /opt/nvenc4/Samples/YUV/1080p/HeavyHandIdiot.3sec.yuv -size 1920 1080 -o test.hevc

But our selftests do not.

[totaam]

2016-02-09 00:25:10: antoine commented

---

Instrumenting the C and Cython versions to dump the contents of the NV_ENC_INITIALIZE_PARAMS structure, I can see that most of the structure is empty (as expected), the differences are:

• for codec 0 / h264:
• with the C sample code:

060005f06227c86b634ea44caa851e50f321f6bf05b7dfb2bd4e494c9b5f24a7
77d3e587800700003804000080070000380400001e0000000100000000000000
0100000000000000000000000000000000000000000000007854e70100000000
8007000038040000000000000000000000000000000000000000000000000000

• with our cython code:

060005f06227c86b634ea44caa851e50f321f6bf05b7dfb2bd4e494c9b5f24a7
77d3e58780070000400400008007000040040000010000001e00000000000000
010000000000000000000000000000000000000000000000905ba6d30f560000
8007000040040000000000000000000000000000000000000000000000000000

• for codec 1 / hevc:
• with the C sample code:

060005f088dc0c7922457b4d9425bda9975f760305b7dfb2bd4e494c9b5f24a7
77d3e587800700003804000080070000380400001e0000000100000000000000
01000000000000000000000000000000000000000000000078a4400100000000
8007000038040000000000000000000000000000000000000000000000000000

• with our cython code:

060005f088dc0c7922457b4d9425bda9975f7603c521df49fa6deb4f97876acc
9effb72620000000200000002000000020000000010000001e00000000000000
01000000000000000000000000000000000000000000000070b17cd342560000
2000000020000000000000000000000000000000000000000000000000000000

[totaam]

2016-02-09 22:08:10: antoine uploaded file nvenc6-debug-forcedefault.patch (1.9 KiB)

debug init structure and force default preset

[totaam]

With the patch above applied, the structure is closer to the C version:

060005f088dc0c7922457b4d9425bda9975f760305b7dfb2bd4e494c9b5f24a7
77d3e58720000000200000002000000020000000010000001e00000000000000
01000000000000000000000000000000000000000000000070054a3d67550000
2000000020000000000000000000000000000000000000000000000000000000

[totaam]

HEVC works as of r11896. The trick was to discover that HEVC has an undocumented minimum size of 72x72..
The tests need improving so we can feed better test input into the encoder, but with the (crappy) existing tests it looks like HEVC is marginally slower than H264 (~15%) at the same resolution and settings, but compresses at lot better. (TBC)

The big disappointment is that it does not perform any better at higher resolutions (even with the low-latency profiles), and it has the same limits as h264: 4096x4096 maximum size. (this is currently hardcoded in our codec, we should be probing it instead: FIXME added in r11897)
Let's hope that other implementations deliver better performance. (#451?).
This will do for now.

Test related fix in r11895. (could be backported - may apply to libvpx testing)