Run multiple versions of the same program at the same time

user_14128026 0
Viewed 238

I have a python script that takes a day to run. I need to run the same code for different parameters. However, using a for loop to iterate over all the parameters is impossible because it would result in an even longer computational time. This is a simple example to depict the situation:

ValuesParams=[1,2,3,4,5]
for i in ValuesParams:
     do something
     output file_i.csv

I would like to run 5 different programs (in a computer cluster), with 5 different values, that output 5 different csv files with different names, but doing it at the same time. Because running that for loop in a single program would take 5 days. In reality it's not just 5 parameters, and the program is not a simple loop.

How can I do this? Any piece of advice or information on what to look for would be incredibly useful.

EDIT Thanks to all the replies, specially remram. They put me in the path to find the answer.

Solution 1: I created a script RunPrograms.sh which iterate over the parameters var1, var2, then passed the parameters to my python script test.py and ran all of the jobs at the same time. One limitation persists, I have to be careful with sending for example 1000 jobs at the same time, but this is what I need it.

#!/bin/bash
for var1 in $(seq 1 0.5 3); do
        for var2 in $(seq 1 3); do
                python3 test.py $var1 $var2 &
        done
done

and inside the python script:

import sys
a=float(sys.argv[1])
b=float(sys.argv[2])
def Code(a,b)
    do_something

Solution 2: Inside the python script using multiprocessing along with starmap to pass multiple parameters:

def Code(T,K):
    do_something
RangeT=np.linspace(0.01,0.05,num=10)
RangeK=[2,3,4]
Z=list(itertools.product(RangeT, RangeK))
if __name__ == '__main__':  
    pool = Pool(4)
    results = pool.starmap(Code, Z)
cluster-computing python-3.x python bash performance
3 Answers

You can either use ProcessPoolExecutor from within your Python code:

from concurrent.futures.process import ProcessPoolExecutor

ValuesParams=[1,2,3,4,5]
with ProcessPoolExecutor(5) as pool:
    pool.map(do_something, ValuesParams)

Or you can work at the shell level and simply run your program multiple times with different command-line arguments, for example using parallel:

parallel python myscript.py -- "1" "2" "3" "4" "5"

import sys

i = int(sys.argv[1])
do_something(i)
output file_i.csv
user_7113800

You could use threads

import threading

def something(arg):
    pass
ValuesParams=[1,2,3,4,5]
for i in ValuesParams:
    thread=threading.Thread(target=something,args=(i,))
    thread.start()
user_79301870

You can look at parallel processing with xargs.
A simple example is time head -12 <(yes "1") | xargs -n1 -P4 sleep, which will run 12 sleep 1 commands, 4 parallel. The command will take 3 seconds.
First check how many processens you want running parallel (I think all 5, but maybe you want to start with 3). In your case, when you want programs program1 .. program5 running with at most 3 parallel, use:

printf "%s\n" {1..5} | xargs -I"{}" -P3 bash -c 'program{} > file_{}.csv'

When you want to test above first, use

echo "Start at $(date +%S)"; printf "%s\n" {1..5} | 
xargs -I"{}" -P3 bash -c 'sleep 2; echo "program {} finished at $(date +%S)" | tee file_{}.csv'
user_32201130
Related