How do I control the number of child processes used to call external commands in Python?

xh c 0
Viewed 52

I understand that using subprocess is the preferred way to invoke external commands.

But what if I want to run multiple commands in parallel, but I want to limit the number of processes generated?

for i in os.listdir(output_lineEdit):  #用for循环遍历文件夹内所有文件
    if i.split(".")[1] == "ma":  # 用if筛选出ma文件
        mapath = i
        cmd = '"{mayaBatchPath}" -batch -file "{maPath}" -script "{melFile}" "{plugins}"'.format(
                mayaBatchPath=MAYABATCHPATH,
                melFile=melFile, #mel文件
                maPath=output_lineEdit+"\\"+mapath, #源maya文件路径
                plugins="-noAutoloadPlugins",
        )
        # print(output_lineEdit+"\\"+mapath)
        p = subprocess.Popen(cmd,
                             shell=True,
                             )

If I run it directly, I'll be running 18 mayabatch.exe programs at the same time because I have 18 files in my folder; What I want is to be able to control the number of programs I run, for example, only run two at a time, then close it and run the next one.

python
1 Answers

By using subprocess.run and a multiprocessing Pool you can easily manage the number of concurrent subprocesses.

For example:

import subprocess
from multiprocessing import Pool

def run(v):
    print(v)
    subprocess.run('echo Hello; sleep 2; echo Done', shell=True)

def main():
    with Pool(5) as pool:
        pool.map(run, range(10))
    
if __name__ == '__main__':
    main()

In this trivial example we call the subprocess 10 times but the pool size is only 5 so there will never be more than 5 concurrent subprocesses.

Unless there are reasons why multithreading might not be appropriate then:

from concurrent.futures import ThreadPoolExecutor
import subprocess

def run(v):
    print(v)
    subprocess.run('echo Hello; sleep 2; echo Done', shell=True)

with ThreadPoolExecutor(5) as tpe:
    tpe.map(run, range(10))
Vlad0
Related