How can I send the stdout of one process to multiple processes using (preferably unnamed) pipes in Unix (or Windows)?

Viewed 31201

I'd like to redirect the stdout of process proc1 to two processes proc2 and proc3:

         proc2 -> stdout
       /
 proc1
       \ 
         proc3 -> stdout

I tried

 proc1 | (proc2 & proc3)

but it doesn't seem to work, i.e.

 echo 123 | (tr 1 a & tr 1 b)

writes

 b23

to stdout instead of

 a23
 b23
6 Answers

Editor's note:
- >(…) is a process substitution that is a nonstandard shell feature of some POSIX-compatible shells: bash, ksh, zsh.
- This answer accidentally sends the output process substitution's output through the pipeline too: echo 123 | tee >(tr 1 a) | tr 1 b.
- Output from the process substitutions will be unpredictably interleaved, and, except in zsh, the pipeline may terminate before the commands inside >(…) do.

In unix (or on a mac), use the tee command:

$ echo 123 | tee >(tr 1 a) >(tr 1 b) >/dev/null
b23
a23

Usually you would use tee to redirect output to multiple files, but using >(...) you can redirect to another process. So, in general,

$ proc1 | tee >(proc2) ... >(procN-1) >(procN) >/dev/null

will do what you want.

Under windows, I don't think the built-in shell has an equivalent. Microsoft's Windows PowerShell has a tee command though.

Like dF said, bash allows to use the >(…) construct running a command in place of a filename. (There is also the <(…) construct to substitute the output of another command in place of a filename, but that is irrelevant now, I mention it just for completeness).

If you don't have bash, or running on a system with an older version of bash, you can do manually what bash does, by making use of FIFO files.

The generic way to achieve what you want, is:

  • decide how many processes should receive the output of your command, and create as many FIFOs, preferably on a global temporary folder:
    subprocesses="a b c d"
    mypid=$$
    for i in $subprocesses # this way we are compatible with all sh-derived shells  
    do
        mkfifo /tmp/pipe.$mypid.$i
    done
  • start all your subprocesses waiting input from the FIFOs:
    for i in $subprocesses
    do
        tr 1 $i </tmp/pipe.$mypid.$i & # background!
    done
  • execute your command teeing to the FIFOs:
    proc1 | tee $(for i in $subprocesses; do echo /tmp/pipe.$mypid.$i; done)
  • finally, remove the FIFOs:
    for i in $subprocesses; do rm /tmp/pipe.$mypid.$i; done

NOTE: for compatibility reasons, I would do the $(…) with backquotes, but I couldn't do it writing this answer (the backquote is used in SO). Normally, the $(…) is old enough to work even in old versions of ksh, but if it doesn't, enclose the part in backquotes.

Since @dF: mentioned that PowerShell has tee, I thought I'd show a way to do this in PowerShell.

PS > "123" | % { 
    $_.Replace( "1", "a"), 
    $_.Replace( "2", "b" ) 
}

a23
1b3

Note that each object coming out of the first command is processed before the next object is created. This can allow scaling to very large inputs.

You can also save the output in a variable and use that for the other processes:

out=$(proc1); echo "$out" | proc2; echo "$out" | proc3

However, that works only if

  1. proc1 terminates at some point :-)
  2. proc1 doesn't produce too much output (don't know what the limits are there but it's probably your RAM)

But it is easy to remember and leaves you with more options on the output you get from the processes you spawned there, e. g.:

out=$(proc1); echo $(echo "$out" | proc2) / $(echo "$out" | proc3) | bc

I had difficulties doing something like that with the | tee >(proc2) >(proc3) >/dev/null approach.

Related