How do I avoid deadlock when using subprocess.Popen to connect multiple processes by pipes?

user_15193351 0
Viewed 201

This is a followup question to @Omry Yadan's Dec/9/2018 answer at How do subprocess.Popen pipes work in Python?. I need to create a three program pipeline and collect stderr and return codes from all three programs. My current solution (below), based on that Dec/9/2018 answer, hangs. The equivalent command pasted in the shell finishes quickly.

The amount of data being piped from stdout to stdin is in the Mbyte realm. The final stdout, as well as the three stderrs, are expected to be much smaller.

#!/usr/bin/env python3

cmd1 = ["gzip", "-dc", "some_file.gz"]
cmd2 = ["filtering_program", "some", "arguments"]
cmd3 = ["collection_program", "some", "arguments"]

p1 = Popen(cmd1, stdout=PIPE, stderr=PIPE)
p2 = Popen(cmd1, stdin=p1.stdout, stdout=PIPE, stderr=PIPE)]
p3 = Popen(cmd1, stdin=p2.stdout, stdout=PIPE, stderr=PIPE)]
(outcome_stdout, outcome_stderr) = p3.communicate()
p1.wait()
p2.wait()
subprocess python-3.x python
1 Answers

Note the following issues that are documented. Also note that the pipe buffer either grows to (MacOS) is is limited to 65536 bytes by default (modern Linux)

Popen.wait

Note This will deadlock when using stdout=PIPE or stderr=PIPE and the child process generates enough output to a pipe such that it blocks waiting for the OS pipe buffer to accept more data. Use Popen.communicate() when using pipes to avoid that.

Popen.stderr

Warning Use communicate() rather than .stdin.write, .stdout.read or .stderr.read to avoid deadlocks due to any of the other OS pipe buffers filling up and blocking the child process.

user_80402720
Related