os_pipe.rs

A cross-platform library for opening OS pipes, like those from pipe on Linux or CreatePipe on Windows. The Rust standard library provides Stdio::piped for simple use cases involving child processes, but it doesn't support creating pipes directly. This crate fills that gap.

• Docs
• Crate
• Repo

Common deadlocks related to pipes

When you work with pipes, you often end up debugging a deadlock at some point. These can be confusing if you don't know why they happen. Here are two things you need to know:

1. Pipe reads will block waiting for input as long as there's at least one writer still open. If you forget to close a writer, reads will block forever. This includes writers that you give to child processes.
2. Pipes have an internal buffer of some fixed size. On Linux for example, pipe buffers are 64 KiB by default. When the buffer is full, writes will block waiting for space. If the buffer is full and there aren't any readers, writes will block forever.

Deadlocks caused by a forgotten writer usually show up immediately, which makes them relatively easy to fix once you know what to look for. (See "Avoid a deadlock!" in the example code below.) However, deadlocks caused by full pipe buffers are trickier. These might only show up for larger inputs, and they might be timing-dependent or platform-dependent. If you find that writing to a pipe deadlocks sometimes, think about who's supposed to be reading from that pipe, and whether that thread or process might be blocked on something else. For more on this, see the Gotchas Doc from the duct crate. (And consider whether duct might be a good fit for your use case.)

Examples

Here we write a single byte into a pipe and read it back out:

use std::io::prelude::*;

let (mut reader, mut writer) = os_pipe::pipe()?;
// XXX: If this write blocks, we'll never get to the read.
writer.write_all(b"x")?;
let mut output = [0];
reader.read_exact(&mut output)?;
assert_eq!(b"x", &output);

This is a minimal working example, but as discussed in the section above, reading and writing on the same thread like this is deadlock-prone. If we wrote 100 KB instead of just one byte, this example would block on write_all, it would never make it to read_exact, and that would be a deadlock. Doing the read and write from different threads or different processes would fix the deadlock.

For a more complex example, here we join the stdout and stderr of a child process into a single pipe. To do that we open a pipe, clone its writer, and set that pair of writers as the child's stdout and stderr. (This is possible because PipeWriter implements Into<Stdio>.) Then we can read interleaved output from the pipe reader. This example is deadlock-free, but note the comment about closing the writers.

// We're going to spawn a child process that prints "foo" to stdout
// and "bar" to stderr, and we'll combine these into a single pipe.
let mut command = std::process::Command::new("python");
command.args(&["-c", r#"
import sys
sys.stdout.write("foo")
sys.stdout.flush()
sys.stderr.write("bar")
sys.stderr.flush()
"#]);

// Here's the interesting part. Open a pipe, clone its writer, and
// set that pair of writers as the child's stdout and stderr.
let (mut reader, writer) = os_pipe::pipe()?;
let writer_clone = writer.try_clone()?;
command.stdout(writer);
command.stderr(writer_clone);

// Now start the child process running.
let mut handle = command.spawn()?;

// Avoid a deadlock! This parent process is still holding open pipe
// writers inside the Command object, and we have to close those
// before we read. Here we do this by dropping the Command object.
drop(command);

// Finally we can read all the output and clean up the child.
let mut output = String::new();
reader.read_to_string(&mut output)?;
handle.wait()?;
assert_eq!(output, "foobar");

Note that the duct crate can reproduce the example above in a single line of code, with no risk of deadlocks and no risk of leaking zombie children.