How do I run a python script in parallel with different arguments in linux?

Question

I have a python script which takes in an argument like so.

conda activate base python myScript.py arg1

I’m connected to my server over ssh so there will be no window system. I want to run 5 different with my script in parallel. So like,

#!/bin/bash
conda activate base
for i in arg1 arg2 arg3 arg4 arg5
do
  python myScript.py $i
done

But this would only let the code run one after the other. How do I run it in parallel?

user_2836621 · Answer

With GNU Parallel

parallel python myScript.py ::: arg1 arg2 arg3 arg4 arg5

GNU Parallel can also do the ssh to one, or more servers for you, so you don't need to login:

parallel -S server 'conda activate base; python myScript.sh' ::: arg1 arg2 arg3 arg4 arg5

Also, add a shebang as the first line of your script like this:

#!/usr/bin/env python

and make your script executable like this:

chmod +x myScript.py

then you can stop telling your shell to use Python every time you want to run it and just use:

./myScript.py arg

user_2422776 · Answer

You could execute them in the background using &:

for i in arg1 arg2 arg3 arg4 arg5
do
  python myScript.py $i &
  # Here ---------------^
done