What is best practice to interact with subprocesses in python

user_7391648 0
Viewed 591

I'm building an apllication which is intended to do a bulk-job processing data within another software. To control the other software automatically I'm using pyautoit, and everything works fine, except for application errors, caused from the external software, which occur from time to time.

To handle those cases, I built a watchdog:

  • It starts the script with the bulk job within a subprocess

     process = subprocess.Popen(['python', job_script, src_path], stdout=subprocess.PIPE,
                                    stderr=subprocess.PIPE, shell=True)

  • It listens to the system event using winevt.EventLog module

     EventLog.Subscribe('System', 'Event/System[Level<=2]', handle_event)

  • In case of an error occurs, it shuts down everything and re-starts the script again.

Ok, if an system error event occurs, this event should get handled in a way, that the supprocess gets notified. This notification should then lead to the following action within the subprocess:

Within the subprocess there's an object controlling everything and continuously collecting generated data. In order to not having to start the whole job from the beginnig, after re-starting the script, this object has to be dumped using pickle (which isn't the problem here!)

Listening to the system event from inside the subprocess didn't work. It results in a continuous loop, when calling subprocess.Popen().

So, my question is how I can either subscribe for system events from inside a childproces, or communicate between the parent and childprocess - means, sending a message like "hey, an errorocurred", listening within the subprocess and then creating the dump?

I'm really sorry not being allowed to post any code in this case. But I hope (and actually think), that my description should be understandable. My question is just about what module to use to accomplish this in the best way?

Would be really happy, if somebody could point me into the right direction...

Br, Mic

subprocess python-3.x
1 Answers

I believe the best answer may lie here: https://docs.python.org/3/library/subprocess.html#subprocess.Popen.stdin

These attributes should allow for proper communication between the different processes fairly easily, and without any other dependancies.

Note that Popen.communicate() may suit better if other processes may cause issues.

EDIT to add example scripts:

main.py

from subprocess import *
import sys

def check_output(p):
    out = p.stdout.readline()
    return out

def send_data(p, data):
    p.stdin.write(bytes(f'{data}\r\n', 'utf8'))  # auto newline
    p.stdin.flush()

def initiate(p):
    #p.stdin.write(bytes('init\r\n', 'utf8'))  # function to send first communication
    #p.stdin.flush()
    send_data(p, 'init')
    return check_output(p)

def test(p, data):
    send_data(p, data)
    return check_output(p)

def main()
    exe_name = 'Doc2.py'
    p = Popen([sys.executable, exe_name], stdout=PIPE, stderr=STDOUT, stdin=PIPE)
    
    print(initiate(p))
    print(test(p, 'test'))
    print(test(p, 'test2'))  # testing responses
    print(test(p, 'test3'))

if __name__ == '__main__':
    main()

Doc2.py

import sys, time, random

def recv_data():
    return sys.stdin.readline()

def send_data(data):
    print(data)

while 1:
    d = recv_data()
    #print(f'd: {d}')
    if d.strip() == 'test':
        send_data('return')
    elif d.strip() == 'init':
        send_data('Acknowledge')
    else:
        send_data('Failed')

This is the best method I could come up with for cross-process communication. Also make sure all requests and responses don't contain newlines, or the code will break.

user_110725330
Related