Bash: limit the number of concurrent jobs?

user_164171 0
Viewed 36824

Is there an easy way to limit the number of concurrent jobs in bash? By that I mean making the & block when there are more then n concurrent jobs running in the background.

I know I can implement this with ps | grep -style tricks, but is there an easier way?

concurrency shell bash
14 Answers

It is hard to do without wait -n (for example, shell in busybox does not support it). So here is a workaround, it is not optimal because it calls 'jobs' and 'wc' commands 10x per second. You can reduce the calls to 1x per second for example, if you don't mind waiting a bit longer for each job to complete.

# $1 = maximum concurent jobs
#
limit_jobs()
{
   while true; do
      if [ "$(jobs -p | wc -l)" -lt "$1" ]; then break; fi
      usleep 100000
   done
}

# and now start some tasks:

task &
limit_jobs 2
task &
limit_jobs 2
task &
limit_jobs 2
task &
limit_jobs 2
wait
user_20679260

Bash mostly processes files line by line. So you cap split input file input files by N lines then simple pattern is applicable:

mkdir tmp ; pushd tmp ; split -l 50 ../mainfile.txt
for file in * ; do 
   while read a b c ; do curl -s http://$a/$b/$c <$file &
   done ; wait ; done
popd ; rm -rf tmp;
user_3428820

Wait command, -n option, waits for the next job to terminate.

maxjobs=10
# wait for the amount of processes less to $maxjobs
jobIds=($(jobs -p))
len=${#jobIds[@]}
while [ $len -ge $maxjobs ]; do
    # Wait until one job is finished
    wait -n $jobIds
    jobIds=($(jobs -p))
    len=${#jobIds[@]}
done
user_109817090
Related