I see a number of posts here about subprocess module and it looks like it's changed quite a bit over the years. From reading documentation, I think I understand the answer to my question, but I'm asking for either confirmation or for someone to tell me what I'm missing. I'm using python 3.6.3.
I have a simple use case for subprocess. I need to build and execute a command. I need to capture stdout and stderr.
I do not need live results.
The size of stdout and stderr will be small relative to the server, so I don't think I need to be concerned about memory issues.
While stdout and std error will be small, I will expect multiple concurrent instances of the parent process to be invoked by a job scheduler.
I've been reading about deadlocks and pipe. Basically documentation states "don't do this stdout=PIPE b/c it can cause a problem with the OS pipe buffer".
But I think I can use subprocess.run and pass in
- stdout=subprocess.PIPE
- stderr=subprocess.PIPE
I think this option is different than stdout=PIPE that I'm told not to do in the documentation, b/c subprocess.run is using Popen.communicate() under the covers, which seems to handle making sure the os pipe doesn't fill up. Thus no deadlock problems, and after .run() finishes I can post process stdout and stderr.
Is this an accurate assessment or am I missing something?
Update #1:
process = Popen(command, stdout=PIPE, stderr=PIPE)
stdout, stderr = process.communicate()
In this case, are the following two things true?
- the potential deadlock issues are eliminated by use of .communicate()
- .communicate() will allow process to fully complete
This code seems pretty simple, but I've seen an array of other code that is far more complicated and I'm not sure if that code is solving a problem that I do not have.