Does running scripts via for-loop in Bash file force them to be single-threaded?

user_11633735 0
Viewed 95

I have a Bash script that I submit to a cluster that calls a pipeline of Python scripts which are built to be multithreaded for parallel processing. I need to call this pipeline on all files in a directory, which I can accomplish with a for-loop. However, I am worried that this will run the operations (i.e. the pipeline) on just a single-thread rather than the full range that was intended.

The batch file for submission looks like this:

#!/bin/bash
##SBATCH <parameters>

for filename in /path/to/*.txt; do
    PythonScript1.py "$filename"
    PythonScript2.py "$filename"
done

Will this work as intended, or will the for loop hamper the efficiency/parallel processing of the Python scripts?

parallel-processing cluster-computing scripting bash performance
2 Answers

As originally written, PythonScript2.py won't run until PythonScript1.py returns, and the for loop won't iterate until PythonScript2.py returns.

Note that I said "returns", not "finishes"; if PythonScript1.py and/or PythonScript2.py forks or otherwise goes into the background on its own, then it will return before it is finished, and will continue processing while the calling bash script continues on to its next step.

You could have the calling script put them into the background with PythonScript1.py & and PythonScript2.py &, but this might or might not be what you want, since PythonScript1.py and PythonScript2.py will thus (likely) be running at the same time.

If you want multiple files processed at the same time, but want PythonScript1.py and PythonScript2.py to run in strict order, follow the comment from William Pursell:

for filename in /path/to/*.txt; do
    { PythonScript1.py "$filename"; PythonScript2.py "$filename"; } &
done
user_70954330

If you are running on a single server:

parallel ::: PythonScript1.py PythonScript2.py ::: /path/to/*.txt

This will generate all combinations of {PythonScript1.py,PythonScript2.py} and *.txt. These combinations will be run in parallel but GNU parallel will only run as many at a time as there are CPU threads in the server.

If you are running on multiple servers in a cluster, it really depends on what system is used for controlling the cluster. On some system you ask for a list of server and then you can ssh to those:

get list of servers > serverlist
parallel --slf serverlist ::: PythonScript1.py PythonScript2.py ::: /path/to/*.txt

On others you have to give each of the commands you want to run to the queing system:

parallel queue_this ::: PythonScript1.py PythonScript2.py ::: /path/to/*.txt

Without knowing more about which cluster control system is used, it is hard to help you more.

user_3630280
Related