[Question] Is it possible to be notified when a command is executed

[stoffeastrom]

I would like to be able to know when a command has been executed. Consider doing

ptyProcess.write(`git status\r`);

Is it possible to be notified when the above command has been executed? I have an Electron app where commands are executed through the UI and need to take actions when the command is finished e.g routing

[jerch]

@stoffeastrom

Short answer - no.

Long answer:
The pty/tty system itself can not do this for the slave part. It allows this only for the master with the SIGHUP signal, for historical reasons. In the old days that was useful in case the output device hung up (a modem, a teletype printer or a real terminal).

When you run a program on the slave side from a terminal emulator, it is usually run "within" a shell. The shell is therefore the first program at the slave side and becomes the session leader in its own process group. When you start your git status the shell will spawn a new process as child and attach it to the controlling terminal (the slave end of the pty) as foreground job. As the foreground job it has the right to output something to the terminal. When it finishes the shell will be informed by SIGCHLD. A shell would normally use this to revert the foreground back to itself, e.g. the shell prompt is showing up again. As you can see there is an "event" for the end of a subprocess, the problem is - it is not propagated back through the pty to the master side. It bubbles only at the slave side up to the session leader. Also those signals are not accessible from the outer world unless you write a debugger. (Note the session leader gets informed about it, so writing your own shell would do ;))

Still it is possible to get a hold of the program at the slave side by OS dependent process control tools. The module does this in pty_getproc by querying procfs for linux and some job/session control magic for OSX. Downside of this - to get informed in a timely fashion it will work only by busy polling those process interfaces and will be cumbersome for several reasons, mainly cpu time and time resolution.

[stoffeastrom]

@jerch Thanks for the detailed answer :)

I kind of threw it out here knowing it would be hard. I decided to work around it since it's just an internal getting started with git/gerrit app.

[Tyriar]

I'm going to look at doing this in the future at a higher level: xtermjs/xterm.js#576 microsoft/vscode#20676

[jerch]

With https://github.com/mheily/libkqueue it would be possible to use BSD style kqueue proc events on linux and windows to achieve this. See http://doc.geoffgarside.co.uk/kqueue/proc.html for an example.

[jerch]

@Tyriar Inspired by python's psutil package I started something similar here https://github.com/jerch/node-psutil. It gonna have support for a process wait call as well.

Problem is - only way to check reliably for a 'terminate' event of a process under unix is to poll the process regularly (done by a kill(pid, 0) call, see https://github.com/jerch/node-psutil/blob/master/_posix.js#L32). Also this can be done more efficiently within C++ and libuv I fear it is the only possible way under linux atm. For BSDs with the full kqueue implementation it can be switched to a real kernel event based paradigm, for Windows it will have to use WaitForMultipleObjects within a subthread (both not yet done).

[jrop]

This is by no means bullet proof, but I wrote a module called autoterm that handles this by waiting for a specific prompt string to show up in the stream. For simple cases it works beautifully.

[Tyriar]

@jerch I think $PS1 injection does a great job solving this issue on Unix-like systems. The main issue imo is how to reliably do it on Windows, I was thinking a user configurable regex (with pre-defined ones for default prompts on cmd, WSL, PS) that extracted the cwd. That approach isn't super reliable though.

@jrop thanks for chiming in 😃. It looks like you're only listening for a single data event which wouldn't work for long running processes? Also you're using a custom prompt which would be a big regression imo, users love the fact that they get their shell already configured the way they like it.

[jrop]

@Tyriar no, it listens for data events until the prompt is matched. Also the shell and the prompt are configurable, so it works for customized prompts as well.

[Tyriar]

@jrop ah ok, that's pretty much shell integration as described in xtermjs/xterm.js#576 then isn't it? Just workout picking up and modifying the user's $PS1?

[jrop]

@Tyriar I have a customized prompt and this method still works because I can pass in a customized regex.

[jrop]

@Tyriar I will have to look into shell Integration, not sure to be honest

[Tyriar]

@jrop my 2 ways to support this were:

• For Windows: Search for a regex matching the prompt for the current shell (with default cmd, PowerShell and WSL bash prompts hard coded).
• For Mac/Linux: Inject custom escape sequences into $PS1 so that xterm.js can pick them up and know what's the cwd, prompt/prompt input and output.

[jrop]

@Tyriar Alright, was typing from a phone previously, so hopefully I can be more clear now.

I just looked into shell integration and my solution is more general because it does not rely on custom escape sequences. However, this does not stop injecting sequences into $PS1 that can easily picked up. The way my solution worked was as follows: 1) initially when constructing an instance of my lib, spawn a shell (which is configurable, though it defaults to /bin/sh). 2) When .run(...) is called, my lib starts listening to (potentially multiple) data events, and once it matches the prompt (via either a default provided RegExp or a custom RegExp), it considers the shell to be prompting and the previous task is resolved as complete. Of course, in the unfortunate case that the prompt string appears in the output of a program, this will yield false positives, of course. However, for the simple automation I was targeting, this case is extremely unlikely.

For the record, the case I was targeting was that we need to automate some deployment scripts, which need a TTY allocated over SSH. Of course, ssh supports the -tt flag (which worked when we tested), but then nesting commands can get kind of messy with multiple levels of string escapes, etc.

[Tyriar]

@jrop thanks for the clarification 😃

[jerch]

@Tyriar Yeah the prompt injection can help to get close, would this be possible under windows with the prompt command too? The only thing is - the prompt does not mark the end of the process itself, it marks the return of the shell as foreground process. I think it won't work with fg/bg tricks, therefore I am trying to get a hold of the real process behind. Maybe a best effort solution will do in the end.

[Tyriar]

@jerch whatever the solution I'd want it to work on cmd, powershell, wsl bash, cygwin, etc. Let me know if you come up with anything more reliable/performant than regex matching every line.

[jerch]

@Tyriar I made some progress with the psutil module. Under linux I am able to spot sub process creation and termination if the process runs long enough (at least 50 ms) with this poller:

var psutil = require('./index');

var p = psutil.Process(14299); // shell pid to spot
var children = [];

var poller = setInterval(function () {
    if (!p.isRunning())
        clearInterval(poller);
    var _children = p.children();
    for (var i=0; i<_children.length; ++i) {
        if (children.indexOf(_children[i]) == -1) {
            var sub = psutil.Process(_children[i]);
            console.log('### process start: ', sub.pid);
            sub.on('terminate', function(process) {
                console.log('### process end: ', process.pid);
            });
        }
    }
    children = _children;
}, 50);

It can be moved to libuv/C++ to to reduce the CPU load and narrow the timeframe, still its another poller and cant spot short running subprocesses, bummer. From my investigations it is not possible to leave the polling field for process interaction in a platform independent way since most platforms just dont give APIs for pure event based process handling (only shining exception is kqueue under BSDs with the ability to register callbacks for process creation and termination). Neither linux nor solaris can do this. For windows it is partly doable for process termination at least by a blocking subthread. Well I am not sure yet about this approach. Any other conceptual ideas?

[Tyriar]

Yeah I'd want to stay away from a polling solution, this is what I do not in VS Code's terminal to track when the process title changes, it falls apart on short lived processes though https://github.com/Microsoft/vscode/blob/e4bccec8ee656480e2d6fced77fbff683fed079e/src/vs/workbench/parts/terminal/electron-browser/terminalProcess.js#L132

[jerch]

@Tyriar Now it gets hacky - I found a way under linux to intercept process creation by overloading the fork call with LD_PRELOAD. Process termination can also be catched by this with a fifo in the custom fork and select in some monitoring code.
Pros - it will make process creation and termination kinda event based, nomore busy pollers.
Cons - overloading libc functions is hacky, not clue yet about security or stability implications.

A similar approach by hooking into CreateProcess should also be possible for windows (done there by some antivirus software and game hackers).

[jerch]

@Tyriar Please have a look at https://github.com/jerch/subprocess-watcher for a proof of concept under linux. It is nearly 100% event based (just a single cleanup poller due to messed up event order, ehem).

It uses LD_PRELOAD to hook into process creation and sends appropriate events over a fifo. Termination is spotted by inotify IN_CLOSE_WRITE events (the creation hook creates one additional fifo for every process, that gets closed by the system when the process dies).

[Tyriar]

@jerch pretty cool, would this work under macOS too?

[jerch]

Not tested yet, but I think the fork /exec* overwrites will work there too. The inotify stuff should be doable with kqueue. But first I have to see what is possible with kqueue itself, maybe the overwrites are not needed there at all. kqueue can catch process creation for sure, still it might not work reliably due to the sandbox idea under macOS. Also I dont know if I can get the command line and arguments with kqueue alone.

If I manage to get all the infos it should be possible to do something like this:

var monitor = new Monitor(<parent_pid>);  // pid of the shell

monitor.on('child', function(child){
    if (child.cmdline == 'git status') {
        console.log('git status started...');
        child.on('terminate', function() {
            console.log('git status finished...');
        });
    }
});

[jerch]

Ok tested on OSX - the fork/exec* thing works, for process termination I found a simpler way by using a library destructor. The latter works also under Linux (prolly all POSIX systems) which makes the fifos obsolete and simplifies the logic a lot. Basically now I can route everything over just one fifo and create events in node as soon as something was written to it.
One thing bugs me - the exitcode seems to be out of reach. I was only able to catch it under Linux, under OSX I tried to grab struct kproc_info in an atexit-handler. Although main should be left by that time already, the exitcode in the process structure is not yet set to the right value (contains always 0).

BTW kqueue is not sufficient under OSX, it lacks the NOTE_TRACK flag to spot child child processes and can't return the cmdline. Therefore I went with the overwrite approach there, too.

[jerch]

The library destructor way has its own problems - while it can spot all graceful process exits, a process abort triggered by the kernel cant be spotted this way. As long as the process aborts by faulty condition from within I can catch it by overwriting all exit calls, as soon as the faulty state is more hazardous and the kernel itself decides to cleanup the process (e.g. any memory violation or a SIGKILL) I cant get the process termination this way. Note that this works with the local fifo as long as the process doesnt close any unkown fds early (some security aware programs do).
Not sure yet how to solve this, I think it will end up with a best effort thingy by using both techniques to narrow the edge cases. An ultimate solution is not possible unless we use some sort of a kernel module ;)

You can find the test code for linux under this branch https://github.com/jerch/subprocess-watcher/tree/desctructor_exit (works similar under OSX, code not yet pushed)

[jerch]

Slowly getting there - after some refactoring there is now an example with pty: https://github.com/jerch/subprocess-watcher/blob/master/examples/pty.js. It is still linux only, next is to port it to OSX (see osx branch).

[jerch]

Basic OSX port is done. Exits are spotted with kqueue (by a small c helper), the single fifo per process idea does not work at all under OSX (maybe due to some security constraint).
I will have a look into Windows next. (Although ports to other BSDs should be straight forward they are less important imho).

[jerch]

Under Windows the trickest part is spotting the creation of a new subprocess. I tried it first with WMI, but that is not reliable enough (basically just a poller itself and misses very short running processes). Therefore I went with hooking into CreateProcess with the mhook library (see win32 branch for a first version).

[jerch]

Basic Windows ports is done. Tested with Win10 64bit, works also with 32 bit apps there (wow64). Does not work with WSL apps yet, I think this needs a linux drop in (no clue yet about the subsystem mechanics).

Btw under Windows I can always grab the exitcode, a pity POSIX limits this to the parent process.

[jerch]

Basic porting to the main platforms is done. Atm the module supports linux, osx and windows. Gonna do a refactoring of the module API before getting lost in platform specific details...

Btw I am somewhat surprised how easy it is to "undermine" OS parts and wonder if this is used by malicious software. Exp. the process creation hooks feel kinda viral (they are viral here in the meaning of getting applied over and over to children). Well, the hooks I applied do nothing fishy but I can imagine to do very evil things from here on. Suspicious at least...

[Tyriar]

Sounds like this is the sort of thing that would get flagged by Windows Defender and other anti-virus software for being naughty.

[jerch]

Only few tests here https://www.virustotal.com/en/ flag the windows binaries. On my computer Windows Defender did not flag any of it.

[jerch]

Basic API consolidation is done, module is alpha now. It comes with two small examples (a tree view and a pty invocation). The native parts are almost settled and work quite reliable. There are still some platform quirks that need to be addressed in the JS part, esp. handling of the first process still differs. Also there is no testing code yet...

[jerch]

Geez, x86 32/64bit multiarch is quite painful with ld (linux) - with ld you can annotate this by a $LIB variable in rpaths. $LIB gets expanded to distro dependant folder paths, where the corresponding lib should end up. Since there is no reliable way to catch those paths, I had to build a test executable and get those paths from a strace output, first big bummer. A pity fatELF did not make it.

Second big bummer - GYP doc sucks big time and I was not able to retrieve command output of a previous target (delayed command evaluation). Seems this is not supported by GYP at all. Therefore I had to create another helper module 'node-libexpansion' that builds a small helper binary to catch the $LIB string.

Third bummer - since GYP doc sucks big time I have no clue how to set a custom variable from the commandline during module build to let people set multiarch support on or off.

Foresight bummer - the problems with GYP will get even more complicated as soon as other multiarch platforms should be supported (How about ARM under linux and BSDs in general?) I start to wonder why people invented/use GYP at all while there are other better more complete build systems like autotools or cmake. Damn node-gyp...

Ok enough whining - x86 32/64bit multiarch works now for all supported platforms (linux, osx, windows), though under linux x64 32bit-buildtools must be installed for now. Thats a big drawback since it is uncommon to have both buildsets installed. Not sure yet how to address this.

TL;DR: GYP fails big time to address arch specific matters. Multiarch still works with quirky workarounds and is likely to fail as soon as other ld based platforms with multiarch support come in.

[jerch]

Solved the 32/64 multiarch buildset problem under linux with a test, that tries to compile IA32 under AMD64 in the node-libexpansion module. If it fails the native 64bit binaries will be build only. There are more concurrent multiarch combinations for linux (esp. with ARM there can be several, MIPS etc.) but since they are uncommon I am not going to support them. For those systems only system native binaries will be build (to be more precise - binaries from the buildset used for nodejs/node-gyp, which can be any supported in theory).

[Tyriar]

@jerch just making sure you know, I'm hesitant in shipping something like this in VS Code as it seems quite hacky and would likely fail in many environments as a result.

[jerch]

No worries, I am with you. I figured that as soon as I realized that it is only possible by touching OS core features. Under Windows it is very intrusive and since it has to place a longjump into CreateProcess at runtime, I would not call this a good citizen anymore. Under Linux the LD_PRELOAD way is documented and legal, but I still dont like idea to undermine libc core functionality. Under security aspects I think the way the library works is very dangerous and needs some attention by OS developers. I will not go into details here to not attract the wrong audience.
Add: For security reasons I deleted the repo and the npm package.

[sberney]

I was appreciating how much effort you put into this research project, increasingly fascinating stuff. You deleted my example code though 😢

[jerch]

Simple thing: the pattern can be misused for malicious things on all major operating systems. Who needs Spectre if one can hook into systems this easy. Its good that its gone, it should never have been made.