Optimize running parallel jobs in bash script

Viewed 34

I have a basic bash script which runs parallel jobs on compute node containing "X" cores. The script goes through a series of folders in a numeric order and execute code within:

#!/bin/bash

no_proc_per_calc=x
no_proc_to_use=X

ini=${v1}
fin=${v2}

counter=1
max_parallel_calcs=$(expr $no_proc_to_use / $no_proc_per_calc)
for ii in $(eval echo "{${v1}..${v2}}"); do
    cd ${ii}
    ./execute &
    echo $PWD
    sleep 5
    cd ../
    mod_test=$(echo "$counter % $max_parallel_calcs" | bc)
    if [ $mod_test -eq 0 ] ; then
        wait
    fi
    counter=$(expr $counter + 1)
done
wait

In this script x is a divisor for both X and v2-v1 (and X/x is usually a divisor of v2-v1, but not necessarily) . The issue is that once the code chooses a block of X/x computations, it will stop until all computations in that block are over, which means if one of calculations takes longer time, X-x node cores will have to remain idle for that duration. Is there a way to continuously to do this type of block calculations, so if a single calculation finishes before the rest the script can move on the next calculation thereby running X/x calculations on the node for most of the computation time.

P.S.: As I am running this submit script on a job cluster, therefore I do not have the option to install new bash libraries. I would prefer if I can solve this problem on a standard out-of-the-box bash installation.

0 Answers
Related