How to run dependent scripts in python in parallel in shell?

user_13675974 0
Viewed 126

I have 6 scripts called:

A_1.py, B_1.py, C_1.py
A_2.py, B_2.py, C_2.py

How do I run these script with the following dependent logic where C scripts should be run (in parallel) on B is complete while at the same time running A and B scripts in sequential order in shell?

enter image description here

python-3.x shell
2 Answers

Unless I have misunderstood your table, it is no harder than this:

A_1.py   # Start A_1
B_1.py   # then B_1

# We can now start A_2 followed by B_2, along with C_1 in parallel    
{ A_2.py ; B_2.py; } &
C_1.py &

# Now wait for all of those to complete, and start C_2
wait
C_2.py
user_28366210

I've never done this before, but I believe makefiles are the way forward:

A Makefile consists of a set of rules. A rule generally looks like this:

targets : prerequisities
   command
   command
   command

The targets are file names, separated by spaces. Typically, there is only one per rule. The commands are a series of steps typically used to make the target(s). These need to start with a tab character, not spaces. The prerequisites are also file names, separated by spaces. These files need to exist before the commands for the target are run.

So let's assume you want to run 5 commands: cmd1, cmd2, cmd3, cmd4, cmd5. Assume the following dependencies:

  • cmd1 and cmd3 have no dependencies
  • cmd2 depends on output from cmd1 and cmd4
  • cmd4 depends on the output of cmd1
  • cmd5 depends on output of cmd4 and cmd3.

In this case, the Makefile would look like:

all: cmd1.run cmd2.run cmd3.run cmd4.run cmd5.run

cmd1.run : 
   cmd1
   touch cmd1.run

cmd2.run : cmd1.run cmd4.run
   cmd2
   touch cmd2.run

cmd3.run :
   cmd3
   touch cmd3.run

cmd4.run : cmd1.run
   cmd4
   touch cmd4.run

cmd5.run : cmd3.run cmd4.run
   cmd5
   touch cmd5.run

Now you can just run your scripts with the simple command:

$ make

More details on the hocus-pocus: https://makefiletutorial.com/

user_83440600
Related