std::process, with stdin and stdout from buffers

user_1401640 0
Viewed 700

I have a command cmd and three Vec<u8>: buff1, buff2, and buff3. I want to execute cmd, using buff1 as stdin, and capturing stdout into buff2 and stderr into buff3. And I'd like to do all this without explicitly writing any temporary files.

std::process seems to allow all of those things, just not all at the same time.

If I use Command::new(cmd).output() it will return the buffers for stdout and stderr, but there's no way to give it stdin.

If I use Command::new(cmd).stdin(Stdio::piped()).spawn() then I can child.stdin.as_mut().unwrap().write_all(buff1) but I can't capture stdout and stderr.

As far as I can tell, there's no way to call Command::new(cmd).stdout(XXX) to explicitly tell it to capture stdout in a buffer, the way it does by default with .output().

It seems like something like this should be possible:

Command::new(cmd)
.stdin(buff1)
.stdout(buff2)
.stderr(buff3)
.output()

Since Rust makes a Vec<u8> look like a File, but Vec doesn't implement Into<Stdio>

Am I missing something? Is there a way to do this, or do I need to read and write with actual files?

rust
2 Answers

If you're ok with using an external library, the subprocess crate supports this use case:

let (buff2, buff3) = subprocess::Exec::cmd(cmd)
    .stdin(buff1)
    .communicate()?
    .read()?;

Doing this with std::process::Command is trickier than it seems because the OS doesn't make it easy to connect a region of memory to a subprocess's stdin. It's easy to connect a file or anything file-like, but to feed a chunk of memory to a subprocess, you basically have to write() in a loop. While Vec<u8> does implement std::io::Read, you can't use it to construct an actual File (or anything else that contains a file descriptor/handle).

Feeding data into a subprocess while at the same time reading its output is sometimes referred to as communicating in reference to the Python method introduced in 2004 with the then-new subprocess module of Python 2.4. You can implement it yourself using std::process, but you need to be careful to avoid deadlock in case the command generates output while you are trying to feed it input. (E.g. a naive loop that feeds a chunk of data to the subprocess and then reads its stdout and stderr will be prone to such deadlocks.) The documentation describes a possible approach to implement it safely using just the standard library.

user_16008980

If you want to read and write with buffers, you need to use the piped forms. The reason is that, at least on Unix, input and output to a process are done through file descriptors. Since a buffer cannot intrinsically be turned into a file descriptor, it's required to use a pipe and both read and write incrementally. The fact that Rust provides an abstraction for buffers doesn't allow you to avoid the fact that the operating system doesn't, and Rust doesn't abstract this for you.

However, since you'll be using pipes for both reading and writing, you'll need to use something like select so you don't deadlock. Otherwise, you could end up trying to write when your subprocess was not accepting new input because it needed data to be read from its standard output. Using select or poll (or similar) permits you to determine when each of those file descriptors are ready to be read or written to. In Rust, these functions are in the libc crate; I don't believe that Rust provides them natively. Windows will have some similar functionality, but I have no clue what it is.

It should be noted that unless you are certain that the subprocess's output can fit into memory, it may be better to process it in a more incremental way. Since you're going to be using select, that shouldn't be too difficult.

user_87054320
Related