How to use all cores with a for loop

user_10666165 0
Viewed 221

I have a script that processes each file in a directory. Each single execution of djvu2pdf takes a lot of time and uses only one of my four CPU cores:

for i in  *.djvu; do djvu2pdf "$i"; done

To speed up the overall processing I want to run four djvu2pdf processes in parallel.

How do I have to change my script that I get four pipelines instead of the presently one pipeline?

parallel-processing bash
2 Answers

Maybe forking will do the trick for you? You can wrap the main part of the processing into a function, for example like this

topdf() {
    local djvu_file=$1
    djvu2pdf "$djvu_file"
}

for i in  *.djvu; do topdf "$i" & done

As you can see there is a "&" character which says that the operation should be sent to background.

user_3960330

With GNU Parallel:

parallel djvu2pdf ::: *.djvu

By default, it will run one job for each CPU core, but you can set more or fewer jobs in parallel with parallel -j N where N is the number you want.

You can also get a progress bar with parallel --bar ...

You can also do a "dry run" to see what it would do without actually doing anything with parallel --dry-run ...

See also here.

user_28366210
Related