Alternatives to Python Popen.communicate() memory limitations?

user_19410 0
Viewed 6107

I have the following chunk of Python code (running v2.7) that results in MemoryError exceptions being thrown when I work with large (several GB) files:

myProcess = Popen(myCmd, shell=True, stdout=PIPE, stderr=PIPE)
myStdout, myStderr = myProcess.communicate()
sys.stdout.write(myStdout)
if myStderr:
    sys.stderr.write(myStderr)

In reading the documentation to Popen.communicate(), there appears to be some buffering going on:

Note The data read is buffered in memory, so do not use this method if the data size is large or unlimited.

Is there a way to disable this buffering, or force the cache to be cleared periodically while the process runs?

What alternative approach should I use in Python for running a command that streams gigabytes of data to stdout?

I should note that I need to handle output and error streams.

popen memory stream python
3 Answers

For those whose application hangs after a certain amount of time when using Popen, please look for my case below:

A Rule of thumb, if you're not going to use stderr and stdout streams then don't pass/init them in the parameters of Popen! because they will fill up and cause you a lot of problems.

If you need them for a certain amount of time and you need to keep the process running, then you can close those streams at any time.

try:
    p = Popen(COMMAND, stdout=PIPE, stderr=PIPE)
    # After using stdout and stderr
    p.stdout.close()
    p.stderr.close()
except Exception as e:
    pass
user_73668500
Related