subprocess.PIPE prevents executable from closing

Question

Why subprocess.PIPE prevents a called executable from closing.

I use the following script to call an executable file with a number of inputs:

import subprocess, time

CREATE_NO_WINDOW = 0x08000000
my_proc = subprocess.Popen("myApp.exe " + ' '.join([str(input1), str(input2), str(input3)]),
                            startupinfo=subprocess.STARTUPINFO(), stdout=subprocess.PIPE,
                            creationflags = CREATE_NO_WINDOW)

Then I monitor if the application has finished within a given time (300 seconds) and if not I just kill it. I also read the output of the application to know whether it failed in doing the required tasks.

proc_wait_time = 300
start_time = time.time()
sol_status = 'Fail'
while time.time() - start_time < proc_wait_time:
    if (my_proc.poll() is None):
        time.sleep(1)
    else:
        try:
            sol_status = my_proc.stdout.read().replace('
 
','')
            break
        except:
            sol_status = 'Fail'
            break
else:
    try: my_proc.kill()
    except: None
    sol_status = 'Frozen'

if sol_status in ['Fail', 'Frozen']:
    print ('Failed running my_proc')

As you can note from the code I need to wait for myApp.exe to finish, however, sometimes myApp.exe freezes. Since the script above is part of a loop, I need to identify such a situation (by a timer), keep track of it and kill myApp.exe so that the whole script doesn't get stuck!

Now, the issue is that if I use subprocess.PIPE (which I suppose I have to if I want read the output of the application) then myApp.exe doesn't close after finishing and consequently my_proc.poll() is None is always True.

I am using Python 2.7.

user_7216865 · Answer

There was a pipe buffer limit/bug in case of huge amounts of data written to subprocess.PIPE. The easiest way to fix it is to pipe the data directly into a file:

_stdoutHandler = open('C:/somePath/stdout.log', 'w')
_stderrHandler = open('C:/somePath/stderr.log', 'w')

my_proc = subprocess.Popen(
    "myApp.exe " + ' '.join([str(input1), str(input2), str(input3)]),  
    stdout=_stdoutHandler, 
    stderr=_stderrHandler, 
    startupinfo=subprocess.STARTUPINFO(), 
    creationflags=CREATE_NO_WINDOW
)
...
_stdoutHandler.close()
_stderrHandler.close()

subprocess.PIPE prevents executable from closing

1 Answers