Introduce 'pipes', allowing users to pipe data to and control plugins from the command line

[imsnif]

What are Zellij pipes?

Zellij pipes use the existing piping capabilities of shells to allow users to visualize and manipulate data from the command line inside plugins.

Example

In this example, I demonstrate how pipes work with an example plugin. Here we take the use-case of filtering and replacing large amounts of data in real time, piping it through multiple plugins to see the data being manipulated on-screen and finally getting it back on the command line so that we can do something useful with the results (eg. append them to a file).

https://vimeo.com/900103101

How does it work?

This PR adds two major interfaces: the zellij pipe CLI command and the new pipe lifecycle method for plugins.

The zellij pipe CLI command

The zellij pipe command (a shorter zpipe alias upcoming in a following PR) can be used to pipe data from the command line: eg. cat /tmp/my-large-file.txt | zellij pipe. If used without a plugin destination (see below), the data will be piped to all running plugins in the session. Plugins not implementing this new API or not listening to this specific message will ignore it.

If the zellij pipe command is directed at a specific plugin (eg. cat /tmp/my-large-file.txt | zellij pipe --plugin https://example.com/my-plugin.wasm) and the plugin is not running, Zellij will run this plugin (downloading it and requesting user permissions as usual) and only begin piping the data to the plugin once the plugin finished loading. The CLI pipe will be blocked until then (backpressure). A plugin can be force-loaded even if an instance is already running using the --launch-new flag. The command also introduces various additional optional flags to control how a new plugin will be launched (eg. floating, skipping plugin cache, etc.)

The pipe lifecycle method for plugins

On the plugin side, a new lifecycle method has been added (also to the ZellijTile interface) called pipe.

Here's an example:

fn pipe(&mut self, pipe_message: PipeMessage) -> bool {
    let mut should_render = false;
    match pipe_message.source {
        PipeSource::Cli(input_pipe_id) => {
            if let Some(payload) = pipe_message.payload {
                self.messages_from_cli.push(payload);
                should_render = true;
            }
            if self.paused {
                // backpressure, this will pause data from the CLI pipeline until the unblock_cli_pipe_input method will be called for this id
                // from this or another plugin
                block_cli_pipe_input(&input_pipe_id);
            }
            if self.should_print_to_cli_stdout {
                // this can happen anywhere, anytime, from multiple plugins and is not tied to data from STDIN
                // as long as the pipe is open, plugins with its ID can print arbitrary data to its STDOUT side, even if the input side is blocked
                cli_pipe_output(input_pipe_id, &payload);
            }
        }
        PipeSource::Plugin(source_plugin_id) => {
            // pipes can also arrive from other plugins
        }
    }
    should_render
}

Similar to the update method, the pipe method returns a bool, true if it would like to render itself, in which case the render function will be called as normal.
When a CLI pipe has ended, it will send a last message with None as its payload to the destination plugin(s).

Plugins can control the flow of data from the CLI with backpressure. Using the block_cli_pipe_input method, a plugin can tell a specific CLI pipe not to send more data until it is unblocked using the unblock_cli_pipe_input method. Both of these method accept the pipe_id provided by the PipeMessage struct. This is purely opt in though, as Zellij controls the backpressure by default, making sure more data is sent from the CLI only once all destination plugins have processed it and rendered (if they requested a re-render).

Plugins can also print data to the pipe's STDOUT, so that users would be able to do things like cat /tmp/my-big-file.txt | zellij pipe >> /tmp/my-destination-file. This writing is completely unbound to STDIN and can happen from any plugin who has the relevant pipe_id. This can be done with the new cli_pipe_output plugin API command.

The new pipe_message_to_plugin plugin API command

As mentioned above, plugins can also start their own pipes. These pipes behave exactly like the CLI pipes described above, being optionally directed at all plugins or at a specific plugin instance. This also gives plugins the ability to launch other plugins (or other instances of themselves) by starting a pipe to a new plugin, or including a unique configuration key as the plugin destination (similar effect to the --launch-new CLI flag described above, which does this internally).

The main differnce between plugin pipes and CLI pipes is that plugin pipes do not implement backpressure. Plugin authors who expect to pipe large amounts of data are encouraged to implement backpressure on their own (eg. only sending a message down a pipe once they got confirmation from the destination plugin that it has been processed).

Example:

pipe_message_to_plugin(
    MessageToPlugin::new("message_name")
        .with_plugin_url("https://example.com/my-plugin.wasm")
        .new_plugin_instance_should_have_pane_title("new_plugin_pane_title")
);