I'm trying use GNU parallel in bash to run several instances of a program in parallel, with different arguments in each instance. Furthermore, I'd like to be able to read these arguments from a file, and have the script wait until all parallelized jobs are done. GNU parallel's parallel --semaphore, aka sem, seemed to be an easy way to do this.
MCVE
Using a modified version of the basic example from the sem docs, I created a minimal test-case to illustrate my issue:
while read i; do
echo -n "$i "
sem -j 4 "sleep $i && echo $i finished"
done < args.txt
echo
echo 'Started wait'
sem --wait
echo 'Done waiting'
Here, args.txt is a file that simply contains:
1
2
3
4
Expected vs. actual output
I expected to see an output that looks something like:
user@host:~$ ./test-sem.sh
1 2 3 4
Started wait
1 finished
2 finished
3 finished
4 finished
Done waiting
However, surprisingly, sem --wait does not actually wait for the tasks to get done, and instead, I get an output like this:
user@host:~$ ./test-sem.sh
1 2 3 4
Started wait
Done waiting
user@host:~$ 1 finished
2 finished
3 finished
4 finished
That is to say, the script executes and terminates, and the sem jobs then run and print their respective outputs in the background! Why is this happening? How can I get sem --wait to actually wait after initializing all sem jobs inside the while loop?
Interestingly, it's reading from the file that's causing the problem, not the while loop itself. That is to say, the following works as expected:
i=1;
while (( $i <= 4 )); do
echo -n "$i "
sem -j 4 "sleep $i && echo $i finished"
let i=i+1
done
echo
echo 'Started wait'
sem --wait
echo 'Done waiting'
But my actual use-case has a file with more complex argument combinations, so I'd really like to read the arguments from the file.