WIP: Support for starting child processes and reading output

[caesay]

API Design

I reviewed subprocess.h, System.Diagnostics.Process, as well as child_process.spawn/spawnSync initially for the api design. To start with, just the sychronous support. See async notes below.

Also, since much of the fut API is already similar to C#, I decided to keep that pattern here also. Note that I'm using internal properties here, because they seem to act much like public properties, but actually public properties would be better.

public enum* ProcessFlags
{
    None = 0,
    NoWindow = 0x8
}

public class ProcessStartInfo
{
    internal string() Command;
    internal List<string()>() ArgumentList;
    internal ProcessFlags* Flags;

    /// Start the process synchronously, block until the process exits, and return a 
    /// ProcessSyncResult object with the exit code and output.
    public ProcessSyncResult StartSync();

    /// Start a process asynchronously in the background and return immediately.
    public void FireAndForget();
}

public class ProcessSyncResult
{
    /// Returns the exit code if the process has already exited, 
    /// otherwise blocks until the process exits.
    internal int ExitCode;

    /// Returns all the std/err output.
    internal string() Output;
}

Sync / blocking process / fire and forget

This is fairly straight forward as there are blocking api's in our target langauges.

C/C++

We will need to emit the sheredom/subprocess.h header. I use the following helper function to start a process using this library:

subprocess_s util_start_subprocess(const std::vector<std::string>* command_line, int options) {
    auto size = command_line->size();
    const char** command_line_array = new const char* [size + 1];
    for (size_t i = 0; i < size; ++i) {
        command_line_array[i] = command_line->at(i).c_str();
    }
    command_line_array[size] = NULL; // last element must be NULL

    struct subprocess_s subprocess;
    int result = subprocess_create(command_line_array, options, &subprocess);
    delete[] command_line_array; // clean up the array

    if (result != 0) {
        throw std::runtime_error("Unable to start Update process.");
    }

    return subprocess;
}

The first argument is the command/path to exe, and the following arguments are the program arguments.

The subprocess_s type is our Process type. For options, we should map ProcessFlags to subprocess_option_* but also should always add the subprocess_option_combined_stdout_stderr option.

The FireAndForget function should just call util_start_subprocess and discard the result.

For the StartBlocking function:

public ProcessSyncResult StartSync()
{
    int options = subprocess_option_combined_stdout_stderr;
    if ((Flags & ProcessFlags.NoWindow) == ProcessFlags.NoWindow)
        options |= subprocess_option_no_window;

    // combine args and command into one array
    auto command = ArgumentList;
    command.insert(command.begin(), Command);

    subprocess_s process = util_start_subprocess(command, options);
    FILE* p_stdout = subprocess_stdout(&process);

    int exitCode = 0;
    subprocess_join(process, &exitCode);

    std::filebuf buf = std::basic_filebuf<char>(p_stdout);
    std::istream is(&buf);
    std::stringstream buffer;
    buffer << is.rdbuf();
    std::string output = buffer.str();

    ProcessSyncResult result;
    result.Output = output;
    result.ExitCode = exitcode;
    return result;
}

C#

Very straight forward, because we can just map most straight across.

public ProcessSyncResult StartSync()
{
    var psi = new ProcessStartInfo() {
        CreateNoWindow = (Flags & ProcessFlags.NoWindow) == ProcessFlags.NoWindow,
        FileName = Command, // mapped from our Command
        RedirectStandardError = true, // always true
        RedirectStandardInput = true, // always true
        UseShellExecute = false, // always false
    };
    
    foreach (var arg in ArgumentList) {
        psi.ArgumentList.Add(arg);
    }

    StringBuilder output = new StringBuilder();

    // we use async events here instead of StandardOutput.ReadToEnd()
    // because it means that the buffer can be processed and it wont 
    // fill up and block the process. Also, it means stdout and stderr
    // get interleaved properly.
    var process = new Process();
    process.StartInfo = psi;
    process.ErrorDataReceived += (sender, e) => {
        if (e.Data != null) {
            output.AppendLine(e.Data);
        }
    };
    process.OutputDataReceived += (sender, e) => {
        if (e.Data != null) {
            output.AppendLine(e.Data);
        }
    };

    process.Start();
    process.BeginErrorReadLine();
    process.BeginOutputReadLine();
    process.WaitForExit();

    ProcessSyncResult result;
    result.Output = output.ToString();
    result.ExitCode = process.ExitCode;
    return result;
}

Js/Ts

public ProcessSyncResult StartSync()
{
    var process = require("child_process").spawnSync(Command, ArgumentList, { encoding: "utf8" });
    ProcessSyncResult result;
    result.Output = process.stdout + "\n" + process.stderr;
    result.ExitCode = process.status;
    return result;
}

D

std.process.pipeProcess can help us to merge the streams and read all output in one go.

import std.stdio;
import std.array;
import std.process;

public ProcessSyncResult StartSync()
{
    // combine command and argumentlist into CommandArgs
    auto pipes = pipeProcess(CommandArgs, Redirect.stdout | Redirect.stderr);
    
    // Read the process output (stdout and stderr)
    string output = cast(string) pipes.readAllUTF8();
    
    // Wait for the process to complete and get the exit code
    auto status = wait(pipes.pid);
    int exitCode = status.status;

    ProcessSyncResult result;
    result.Output = output;
    result.ExitCode = exitCode;
    return result;
}

OpenCL

I don't think we can / should support this platform. Using Process should result in a compiler error.

Java

We can use the ProcessBuilder which takes a List as a constructor argument

public ProcessSyncResult StartSync()
{
    // merge Command and ArgumentList into CommandArgs
    ProcessBuilder builder = new ProcessBuilder(CommandArgs);
    builder.redirectErrorStream(true); // both out and err go to getInputStream

    Process process = builder.start();
    StringBuilder processOutput = new StringBuilder();
    int exitCode = -1;

    try (BufferedReader processOutputReader = new BufferedReader(new InputStreamReader(process.getInputStream()));)
    {
        String readLine;
        while ((readLine = processOutputReader.readLine()) != null)
        {
            processOutput.append(readLine + System.lineSeparator());
        }
        exitCode = process.waitFor();
    }

    String output = processOutput.toString().trim();

    ProcessSyncResult result;
    result.Output = output;
    result.ExitCode = exitCode;
    return result;
}

Python

import subprocess
public ProcessSyncResult StartSync()
{
    # Combine the executable and the arguments into a single command
    command = [Command] + ArgumentList
    
    # Run the command, capture stdout, stderr, and the exit code
    process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
    stdout, _ = process.communicate()  # This will block until the process completes

    # `stdout` contains the combined output of stdout and stderr
    ProcessSyncResult result;
    result.Output = stdout;
    result.ExitCode = process.returncode;
    return result;
}

Swift

import Foundation

public ProcessSyncResult StartSync()
{
    let process = Process()
    process.executableURL = URL(fileURLWithPath: Command)
    process.arguments = ArgumentList

    let pipe = Pipe()
    process.standardOutput = pipe
    process.standardError = pipe

    process.run()
    process.waitUntilExit()

    let data = pipe.fileHandleForReading.readDataToEndOfFile()
    let output = String(data: data, encoding: .utf8) ?? ""

    ProcessSyncResult result;
    result.Output = output;
    result.ExitCode = process.terminationStatus;
    return result;
}

Async / readline

There are some tricky bits here, because as far as I know there are no support for delegates or async callbacks of any kind in fut.

Perhaps we should spin out delegates/callbacks as a separate proposal? Since fut also doesn't support exception handling, we could spin out something akin to the javascript promise api and then map that to language specific features. Many languages have native async/await support, including JS/TS, python, C#, but not every language can map to async/await. Java has CompletableFuture, C# has TaskCompletionSource, JS/TS has promise, and C/C++ has callbacks, so I think that a promise/callback approach will be more compatible overall.

C/C++

public bool HasExited() {
return subprocess_alive(process) != 0;
}

C#

Js/Ts

D

OpenCL

I don't think we can / should support this platform. Using Process should result in a compiler error.

Java

Python

Swift