Home

This is the start of a brain dump to help give collaborators some guidance and direction for implementation. At the moment, it will mainly consist of consolidating information from some conversations in Caster issues and the Gitter channel into one place for easier reading.

Problem/Solution Statement

Caster and other Natlink/Dragonfly applications run as non-elevated processes. As Caster is primarily designed for use by software developers, many of the programs we use need to run with elevated permissions ("Run As Administrator"), which means that Caster can't interact with them.

The goal of this project is to create a helper application that can run in the background as an elevated process to act as an intermediary. It would accept commands from Caster or other Dragonfly apps, through a messaging API, and translate those commands into mouse and keyboard inputs which are then sent to the target application.

Running the POC

Currently, this helper runs as a system tray application that has an HTTP server running on port 2083 (this should be made configurable before we leave the proof of concept stage). There is also a Windows service that can run as System user when the machine first boots up, and can launch the system tray app without causing the UAC prompt to appear, but it needs some work before it will work reliably. Right now, this is just a proof of concept, and only the Key action from dragonfly has been fully implemented. To install and test, do the following:

• Make sure you have the Microsoft .NET framework 4.5 or later installed.
• Download the binary zip file under releases. (Or, alternately, you can build the project yourself using Visual Studio 2017 Community Edition.)
• Extract the application files and double-click CasterUIAutomation.SystemTray.exe.
• If the UAC prompt appears, click Yes.

You should see a new icon appear in your system tray. You can start, stop, and exit the helper app by clicking on the icon. You can test that it's working by opening your browser and entering the following in the address bar:

http://localhost:2083/key?spec=c-t/50,g,o,o,g,l,e,dot,c,o,m

If everything is working, this should send a Key command to the HTTP server listening on port 2083, which will simulate a Control+T keypress to open a new tab, wait a half second, and type google.com in the address bar of the new tab.

I was able to successfully test it in Caster by altering the letters2 in caster\lib\alphanumeric.py like so:

import urllib2, urllib
def letters2(big, letter):
    url = 'http://localhost:2083/key'
    data = urllib.urlencode({
        'spec' : letter if str(big) == "" else "shift:down, " + letter + ", shift:up"
        })
    request = urllib2.Request(url, data)
    response = urllib2.urlopen(request)
    #if str(big) != "":
    #    Key("shift:down").execute()
    #Key(letter).execute()
    #if str(big) != "":
    #    Key("shift:up").execute()

To make this work for real, code should be integrated into the Dragonfly actions. It seems like the most appropriate place (at least for the Key action) would be the DynStrActionBase class in the action_base.py file. When Dragonfly is configured to use the helper, rather than parsing the spec in the initialize or _execute methods, the spec would be sent to localhost:2083, taking care to evaluate spec = self._spec % data in the case where the spec is a dynamic string and data is passed in to _execute.