How to parallelise a while loop in bash?

user_8714014 0
Viewed 1771

I have the following while loop that runs in parallel. ( The logProcess is a function that I define earlier in my script. )

while read LINE; do
    logProcess $LINE &
done <<< "$ELS_LOGS"
wait

I need to find a way to limit the number of processes running. I know there are parallel processes running. How do I convert the loop to use that command?

parallel-processing bash
2 Answers

May I recommend GNU Parallel to you. Your entire script would become:

parallel -a "$ELS_LOGS" logProcess

If logProcess is a function defined within your script, you will need to export it first before running GNU Parallel, like this:

export -f logProcess

Then if you want, say, 8 running at a time, you simply do:

parallel -j 8 -a "$ELS_LOGS" logProcess

If you want to see what it would do, without actually doing anything:

parallel --dry-run ...

If you want a progress bar, or an ETA:

parallel --eta ...
parallel --bar ...
user_28366210

GNU xargs is also an adequate tool for the job:

xargs -P 20 -d $'\n' -n 1 logProcess

...will run up to 20 concurrent logProcess instances, with each line of the stdin given to xargs passed to a different such instance.

user_141220
Related