Execute multiple pyiron jobs with dependencies

user_14437231 0
Viewed 35

I have 4 jobs (A, B, C, D), which I want to start using pyiron. All jobs need to run on a remote cluster using SLURM. Some of the jobs need results from other jobs as input.

Ideally, I would like to have a workflow like:

  • Job A is started by the user.
  • Jobs B and C start automatically and in parallel (!) as soon as job A is done.
  • Job D starts automatically as soon as the jobs B and C are finished.

I realize that I could implement this in Jupyter using some if-conditions and the sleep-command. However, the jobs A, B, and C could run for multiple days and I don't want to keep my Jupyter notebook running for so long.

Is there a more convenient way to realize these job dependencies in pyiron?

pyiron
1 Answers

I guess the easiest way would be to submit the whole Jupyter notebook to the queue using the script job class:

job = pr.create.job.ScriptJob("script")
job.script_path = 'workflow.ipynb'
job.server.queue = 'my_queue'
job.server.cores = 32
job.run()

Here workflow.ipynb would be your current notebook, my_queue your SLURM queue for remote submission and 32 the total number of cores for allocation.

user_69967460
Related