ms windows shadow server improvements

[totaam]

The current shadow server for win32 sort of works but we need to improve:

• keyboard mapping is not done: we must generate a lookup map for keycodes
• we may be able to export windows individually: Image.frombuffer with 16-bit image data (grabs from a specific hwnd)
• if we are going to screenscrape, then we should do it faster: Fastest method of screen capturing
• capture the cursor: How to make a bitmap of the current mouse pointer?
• fixed multi-monitor code? win32ui vs wxPy screen capture multi monitor
• Various methods for capturing the screen
• there is no way to connect to the server without tcp mode, maybe we should add a simple named pipe listener: createNamedPipe in python (also similar code in winswitch)

[totaam]

Screenscraping may be the only feasible way to get 3D apps to work:
Since the hardware-accelerated 3D rendering commands typically bypass the GDI layer, most screen scrapers (WinVNC, LiveMeeting, GoToMeeting, etc.) cannot pick up the 3D rendering area of Windows applications, or they will not detect when the 3D area has been updated.

(from DRC @virtualgl-users)

[totaam]

Some fixes:

• bug which caused 100% cpu on win32 shadow servers win32 shadow server consumes 100% cpu even when inactive #691
• r7794 keyboard is almost usable (but completely hackish: missing modifiers, no key repeat, etc)
• • some minor tweaks: r7793, etc

[totaam]

• generic shadow improvements are tracked in generic shadow improvements #899
• clipboard: makes sense to re-do the using native code as per re-implement clipboard without nested main #812.
• keyboard: better modifiers handling?
• nvenc: nvenc support for shadowing on win32 #558
• use window list to split into distinct window sources? (see Support SHAPE extension #501)
• service? tray menu to enable sharing?
• named pipes support so we don't have to require TCP: win32pipe

Generic Window API links:

• EnumWindows (used by seamless mode)
• GetWindowInfo to get WINDOWINFO (size, border, type, status..)
• GetWindowRect
• GetWindowText
• IsWindowUnicode

Window Event Hooks:

• Hooks: A global hook monitors messages for all threads in the same desktop as the calling thread. A thread-specific hook monitors messages for only an individual thread. A global hook procedure can be called in the context of any application in the same desktop as the calling thread, so the procedure must be in a separate DLL module.
• RegisterShellHookWindow: This function is not intended for general use. It may be altered or unavailable in subsequent versions of Windows.
• SetWindowsHookEx or SetWindowsHook: (needs the DLL to be injected? for all but mouse events)

• with WH_SHELL (The function receives notifications of Shell events from the system.): ShellProc callback function, in particular:
  • HSHELL_REDRAW
  • maybe also: HSHELL_WINDOWACTIVATED, HSHELL_WINDOWCREATED, HSHELL_WINDOWDESTROYED and HSHELL_WINDOWREPLACED, which we could tie with the list of rectangles if we keep the hwnd at hand.
• with WH_CBT (The system calls this function before activating, creating, destroying, minimizing, maximizing, moving, or sizing a window):

• SetWinEventHook: Sets an event hook function for a range of events

Example code:

• Log Windows focus changes
• How to use winapi SetWinEventHook in python
• PyHook
• pipeTestService.py

[totaam]

r10002 uses Pillow's ImageGrab module to capture the screen on win32, which is already much faster.

It is still quite slow: timeit clocks it at about 100ms for 1080p on a mid-range system, which is a lot better than GTK, but has room for improvement:

• we call tobytes - which makes a copy, we could avoid this by going pure native via pywin32 and managing the target buffer's lifetime ourselves (and ideally, straight into nvenc's input buffer - tall order!)
• we use the BGRX format so that we can use nvenc... (we could use whatever RGB format is native to win32 and just add the necessary kernel to nvenc instead, see nvenc support for shadowing on win32 #558)

Some links:

• Python windows 7 screenshot without PIL
• GetDIBits
• BitBlt

[totaam]

The code in r10002 only captured one screen, this is fixed in r10026, see #637#comment:11 for details. Will backport.

r10028 should help speed things up a bit by re-using the same bitmap and context objects until the display settings change.
But this is still not good enough: it takes about 100ms for copying 4 megapixels (2 x 1080p screens).
About 65% of this time is spent in BitBlt, and the rest is in GetBitmapBits.
The BitBlt performance with Aero enabled is just poor. Disabling aero is a temporary workaround.

We want to stay on the GPU if we can (#365), especially if we're going to use the GPU for encoding (ie: nvenc) - the difference is in the order of a factor 100!

More info:

• Ways to capture the screen, in particular with direct3d9: To do this with Direct3D, you could use GetRenderTargetData to copy the rendering surface (retrieved with GetRenderTarget) to an off-screen surface (which can be created with CreateOffscreenPlainSurface).
• Desktop Duplication API (Windows 8 onwards), looks like a proper solution to shadowing. We could also handle the rectangle "move" messages it sends, which is a lot better than plain "damage" events.

This will have to do for now. (nvenc needs fixing first anyway... see #389)

[totaam]

r11482 added a basic systray for the shadow server which only shows: "about", "exit" and "close menu".

[totaam]

2015-12-27 08:11:04: antoine uploaded file namedpipes.patch (12.7 KiB)

work in progress named pipes support

[totaam]

Moving the named pipes to #1150.

[totaam]

Keyboard issues in #1099.

Stumbled upon this info python-win32: Global Window Messages: What you probably want is a WH_CBT hook so that you can catch WM_ACTIVATE and WM_DEACTIVATE.
There is some code that does something similar in pyAA apparently.

[totaam]

See my original question to python-win32 mailing list: https://mail.python.org/pipermail/python-win32/2009-September/009584.html
And Win32 WH_CBT Hook - close window before it is created, WH_CBT hook, using SetWindowsHookEx to monitor windows events, KeyboardHookProc in DLL doesn't do anything when called from python

This is going to take some effort and will require Windows 8 or later and probably building things with MSVC 2013 (not the "free" express version..) for the newer API: IDXGIOutputDuplication: The IDXGIOutputDuplication interface accesses and manipulates the duplicated desktop image.

Printing may be possible using redmon: The RedMon port monitor redirects a special printer port to a program

[totaam]

For sound see: wasapisink

r12394 will use directsoundsrc by default for win32 shadow servers

[totaam]

2016-05-22 12:20:35: antoine commented

---

• r12647 should fix sound shadowing on win32 (untested - virtualbox is having a fit)
• r12707 (skip warnings)

[totaam]

2016-06-17 10:16:22: antoine commented

---

• we can disable compositing by calling: DwmEnableComposition. This will fix screen capture on Vista and later.
• maybe we should switch to using GetDIBits instead of GetBitmapBits, as per: Desktopmagic screengrab_win32 (ctypes), GetDIBits to PIL image example
• good directx example: Doing readback from Direct3D textures and surfaces, this is probably the right way of hooking up NVENC nvenc support for shadowing on win32 #558
• good IDXGIOutput1::DuplicateOutput example: RPI-GPU-rdpClient wddm.h
• ffmpeg option: gdigrab, directshow

[totaam]

2016-06-17 14:28:41: antoine commented

---

r12850 disables composition on vista and later, r12852 + r12853 + r12854 improve error handling - which fires a lot more with vista onwards when the UAC prompt shows up.

[totaam]

A good solution for solving shadow performance issues is #1317 (requires an nvidia card...)

[totaam]

If we go down the directx route, with mingw #678 these may be helpful / needed:

• w32api-directx-standalone: Alternative DirectX headers for MinGW w32api headers
• setting up mingw for directx, ie: g++ dxtest.cpp -o dxtest.exe -I/dxsdk/include -L/dxsdk/lib -DUNICODE -ld3d9
• Universal DirectX SDK: The goal of this open source project is build an universal DirectX SDK. It can work with multiple versions of Visual Studio, MinGW, and other compilers
• How to capture the screen in DirectX 9 to a raw bitmap in memory without using D3DXSaveSurfaceToFile
• Hooking DirectX EndScene from an injected DLL and in particular: Case study: Fraps and how it uses SetWindowsHookEx.
• Capture screen using DirectX - but this uses the slow GetFrontBufferData...
• Fastest method of screen capturing
• Various methods for capturing the screen
• Resource Management Best Practices
• D3D hooking
• C# – SCREEN CAPTURE AND OVERLAYS FOR DIRECT3D 9, 10 AND 11 USING API HOOKS
• Direct3DHook
• msdn: ways to capture the screen
• Execution time: GetFrontBufferData, GetBackBuffer, GetRenderTargetData

So, by injecting the DLL into every process, we could get individual windows as surfaces.

If we are willing to require windows 8 or later, the solution is likely to be the Desktop Duplication API:

• Desktop Duplication Sample
• DXGICaptureSample
• Very Fast Screen Capture With Windows 8, C# and DirectX11

Some links on the logon stuff, which we need for running as a service:

• LogonUser function
• Service functions
• Service Logon Accounts
• pywinrm: Python library for Windows Remote Management (WinRM)
• Impersonating Principals on Windows: You need to authenticate a thread temporarily as another principal on a Windows machine—for example, to make something run with the appropriate administrative rights.

For audio:

• How do I capture the audio that is being played?
• Loopback Recording

[totaam]

Added basic NVIDIA Capture SDK support using basic "copy to system memory" API.
Screen capturing + downloading to system RAM at 4K takes less than 30ms, so we should be able to do 30fps at 4K (given enough bandwidth / fast encoder).

Much more can be done with this API:

• TODO: error recovery, re-instantiate context on recoverable failures, bubble up if unrecoverable
• download to RGB since we don't use the alpha channel (will save memory bandwidth and CPU time)
• integrate with NVENC - via CUDA?
• port to Linux - the examples refuse to run on Fedora 26 (could be glvnd related?)
• meh: support 32-bit mode?
• ARGB10 10-bit colour modes
• download as YUV420 when quality is set lower? (NVENC is probably better in all cases)

[totaam]

Minor updates in r15727 + r15730. Helped uncover a core imagewrapper bug: r15728.

The NVFBC progress should now be tracked here: #1317.

[totaam]

Some scripts that could be useful - and not just for shadow servers:

• List Print Job Information - if we could somehow get a jobid from the print command, ADDJOB_INFO_1 has one, but we're not calling AddJob ourselves..
• List All Possible Video Controller Resolutions - for randr-like support
• List Current Display Configuration Values: could also be useful in the client, so we know the real display depth?
  Here's some ctypes code to access the data:

from comtypes.client import CreateObject
o=CreateObject('WbemScripting.SWbemLocator')
s=o.ConnectServer('.', 'root\\cimv2')
res=s.ExecQuery('SELECT * FROM Win32_DisplayConfiguration')
print([int(x.Properties_['BitsPerPel'].Value) for x in res])

Could also be useful for detecting virtualbox:

print([(x.Properties_['DeviceName'].Value) for x in res])
[u'VirtualBox Graphics Adapter']

• List Sound Card Properties - so we can select the right one (but how to reconcile the deviceid with gstreamer's device number?)
• List Printer Capabilities / List Printer Information: export more printer information (orientation, etc)

Now for making a service:

• cx_freeze example
• LogonUser in ctypes example
• Windows Logon as a user without a password with Python
• createprocessasuser and getting the output

[totaam]

Moving the system service to #1527.

Python3 #1568 is making progress, this may help.

[totaam]

For capturing the screen with directx, a good example is found in the gstreamer source: winscreencap.

See also multi-monitor handling as windows: #1805

[totaam]

NVFBC is now deprecated on windows-10: #1317#comment:11.

DDA example: nvEncDXGIOutputDuplicationSample

[totaam]

Links:

• OBS has Windows Graphics Capture support: Windows Graphics Capture, which is new to Windows 10
• gstreamer: dxgiscreencapsrc: Add screen capture plug-in using Desktop Duplication API

See also:

• Desktop Duplication API returns empty frame
• Linking Cuda (cudart.lib) makes DXGI DuplicateOutput1() fail
• Window Capture with DXGI

[totaam]

Windows 10 Screen capture example code looks simple enough.
Using Windows.Graphics.Capture from a Cython C / C++ context will be harder.