nvenc support for shadowing on win32

[totaam]

Issue migrated from trac ticket # 558

component: platforms | priority: critical | resolution: fixed | keywords: win32 nvenc

2014-04-10 07:51:24: totaam created the issue

---

Shouldn't be too hard to do. As of r6082, the code compiles.
What's left to do:

• build pycuda from source, against a recent cuda sdk
• ensure we package what we need
• look for the nvencodeapi.dll at runtime? (somewhere with the drivers?)

Once that's done, we can have superfast session shadowing, which would be nice. The biggest remaining bottleneck is likely to be how we capture the screen's pixels.

[totaam]

2014-08-19 05:11:31: totaam commented

---

Scheduling for 0.16

[totaam]

2015-07-22 13:01:36: totaam changed status from new to assigned

[totaam]

2015-07-22 13:01:36: totaam changed owner from antoine to totaam

[totaam]

2015-07-22 13:01:36: totaam commented

---

Mostly done in r9999, and found some bugs along the way, it seems to work, but:

• the colours are a little bit off, as if the YUV input planes were not aligned properly in memory
• the screen capture part was dreadfully slow, now improved see ms windows shadow server improvements #389#comment:5

[totaam]

2015-07-26 11:18:40: antoine uploaded file nvenc-vadjust.patch (2.4 KiB)

patch to make it easier to tweak the cuda vertical output size at runtime using an env var

[totaam]

2015-07-26 11:20:19: antoine commented

---

With the patch above, I can make it encode properly by using the magic value:

set XPRA_NVENC_VADJUST=-68

Which makes it convert to YUV444P using a vertical size of 1020 instead of 1088.
No idea why this magic value is "right", yet.

[totaam]

2015-07-27 12:38:28: antoine uploaded file nvenc-hacked-win32.patch (7.5 KiB)

more hacks to tweak input values to the kernel and encoder

[totaam]

2015-07-27 12:51:49: antoine commented

---

With the patch above, we get the correct size for the output picture, though the VADJUST value does make the input buffers overlap a bit which corrupts the picture near the top.
It seems that the video encoder vertical size used is not the one we specify (1080 rounded up to 1088) but it uses 1020 instead.. which is just a really weird number to choose: 255*4.
Now, we could just find the next size up that rounds down to what we need, but the exact formulae used is unlikely to be obvious. (255?) And I see no way of knowing when this hack should be applied. Only on win32? Only with some driver versions?
It is worth mentioning that these latest drivers seem to have fixed the long standing bug which exposed the wrong values for the input formats. (forcing us to use the value as a mask..) Could this be related?
We have always rounded up the width of the input buffer from 1920 to 2048, does this matter?

[totaam]

2015-07-30 03:57:45: antoine commented

---

Simple maths:

• the (assumed but wrong) horizontal padding: 2048-1920=128
• 2048/128=16 (we shrink by 1/16)
• 1080/16=67.5 (which can be rounded to 68)

So based on these (wrong) assumptions, I could explain the 68 lines adjustment.
It is wrong because we round up to 2560, not 2048... But maybe something else rounds like that.

[totaam]

2015-08-06 14:05:02: antoine uploaded file glxspheres-nvenc-voffset.png (732.8 KiB)

shows the visual corruption with a Linux server and client

[totaam]

2015-08-06 14:05:58: antoine changed priority from minor to critical

[totaam]

2015-08-06 14:05:58: antoine commented

---

I've updated my Linux drivers to the beta version 355.06 and I am now seeing the same defects here. I should have known that expecting a stable ABI was too much to ask...

So now we need to build a blacklist, or version driven workaround.
Not sure if the driver version number is even available on win32 since we use the kmod version on Linux!

[totaam]

2015-08-08 08:23:01: antoine commented

---

Sure enough, downgrading the drivers on my test win32 box fixes nvenc.

Some options for figuring out the version number on win32 (which may also be able to replace the ugly proc module parsing on Linux in some cases):

• NVAPI
• GetDeviceCaps for DRIVERVERSION.
• nvidia-ml-py

Immediate problem that I can see: we're running a 32-bit python interpreter, and the 64 bit "NVML.DLL" and won't load...

[totaam]

2015-08-08 10:01:14: antoine changed status from assigned to closed

[totaam]

2015-08-08 10:01:14: antoine set resolution to fixed

[totaam]

2015-08-08 10:01:14: antoine commented

---

Implemented a simple cython wrapper for NVAPI in r10240, we just statically link against nvapi.lib (nvapi64.lib on 64-bit - untested).
This gives us a version number we can check against.

For the time being we just print a big warning, maybe this should fail the whole nvenc module?
This will do for now, we need #389 for this to be truly useful anyway (but the bugs found along the way were worth it!).