I have multiple subprocess Popens piped together in Python 2.7 through their stdin/stdout as follows:
p1 = Popen(shlex.split(cmd1), stdout=sp.PIPE, stderr=sp.PIPE)
p2 = Popen(shlex.split(cmd2), stdin=p1.stdout, stdout=sp.PIPE, stderr=sp.PIPE)
...
pN = Popen(shlex.split(cmdN), stdin=pN-1.stdout, stdout=sp.PIPE, stderr=sp.PIPE)
stdout, stderr = pN.communicate()
Everything works very smoothly as expected; however I would like to capture the stderr of every different process, while at the moment I can only capture the one of the final process pN through pN.communicate(). I know that each stderr can be redirected in a different file, but I would like to know if there is any way to capture the stderr of all the processes after and/or before the final communicate within Python?
Calling communicate for different processes always results in a broken pipe.
EDIT-1
Is calling communicate on each process in the same sequence as in the pipe a correct and acceptable solution?
p1 = Popen(shlex.split(cmd1), stdout=sp.PIPE, stderr=sp.PIPE)
p2 = Popen(shlex.split(cmd2), stdin=p1.stdout, stdout=sp.PIPE, stderr=sp.PIPE)
...
pN = Popen(shlex.split(cmdN), stdin=pN-1.stdout, stdout=sp.PIPE, stderr=sp.PIPE)
stdout1, stderr1 = p1.communicate()
stdout2, stderr2 = p2.communicate()
...
stdoutN, stderrN = pN.communicate()
EDIT-2
The solution proposed in EDIT 1 is wrong as it produces errors when communicate is called on an already ended process.
The only feasible solution at this time seems to redirect stderr of all processed on files and read those after making sure that all processes terminated with:
map(lambda p : p.wait(), (p1, p2, ..., pN))