Wait for set of qsub jobs to complete

user_1034413 0
Viewed 40918

I have a batch script which starts off a couple of qsub jobs, and I want to trap when they are all completed.

I don't want to use the -sync option, because I want them to be running simultaneously. Each job has a different set of command line parameters.

I want my script to wait till when all the jobs have been completed, and do something after that. I don't want to use the sleep function e.g. to check if certain files have been generated after each 30 s, because this is a drain on resources.

I believe Torque may have some options, but I am running SGE.

Any ideas on how I could implement this please?

Thanks P.s. I did find another thread Link

which had a reponse

You can use wait to stop execution until all your jobs are done. You can even collect all the exit statuses and other running statistics (time it took, count of jobs done at the time, whatever) if you cycle around waiting for specific ids.

but I am not sure how to use it without polling on some value. Can bash trap be used, but how would I with qsub?

qsub
9 Answers

This works in bash, but the ideas should be portable. Use -terse to facilitate building up a string with job ids to wait on; then submit a dummy job that uses -hold_jid to wait on the previous jobs and -sync y so that qsub doesn't return until it (and thus all prereqs) has finished:

# example where each of three jobs just sleeps for some time:
job_ids=$(qsub -terse -b y sleep 10)
job_ids=job_ids,$(qsub -terse -b y sleep 20)
job_ids=job_ids,$(qsub -terse -b y sleep 30)
qsub -hold_jid ${job_ids} -sync y -b y echo "DONE"
  • -terse option makes the output of qsub just be the job id
  • -hold_jid option (as mentioned in other answers) makes a job wait on specified job ids
  • -sync y option (referenced by the OP) asks qsub not to return until the submitted job is finished
  • -b y specifies that the command is not a path to a script file (for instance, I'm using sleep 30 as the command)

See the man page for more details.

user_37803890 
#!/depot/Python-2.4.2/bin/python

import os
import subprocess
import shlex

def trackJobs(jobs, waittime=4):
    while len(jobs) != 0:
        for jobid in jobs:
            x = subprocess.Popen(['qstat', '-j', jobid], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
            std_out, std_err = x.communicate()
            if std_err :
                jobs.remove(jobid)
                break
        os.system("sleep " + str(waittime))
    return

This is the simple code where you can track the stauts of completion of qsub jobs . Here function accepts list of jobIds (for example ['84210770', '84210774', '84210776', '84210777', '84210778'] )

user_132127660

You can start a job array qsub -N jobname -t 1-"$numofjobs" -tc 20, then it has only one job id and runs 20 at a time. You give it a name, and just hold until that array is done using qsub -hold_jid jid or qsub -hold_jid jobname.

user_37054900

I needed more flexibility, so I built a Python module for this and other purposes here. You can run the module directly as a script (python qsub.py) for a demo.

Usage:

$ git clone https://github.com/stevekm/util.git
$ cd util
$ python
Python 2.7.3 (default, Mar 29 2013, 16:50:34)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import qsub
>>> job = qsub.submit(command = 'echo foo; sleep 60', print_verbose = True)
qsub command is:

qsub -j y -N "python" -o :"/home/util/" -e :"/home/util/" <<E0F
set -x
echo foo; sleep 60
set +x
E0F

>>> qsub.monitor_jobs(jobs = [job], print_verbose = True)
Monitoring jobs for completion. Number of jobs in queue: 1
Number of jobs in queue: 0
No jobs remaining in the job queue
([Job(id = 4112505, name = python, log_dir = None)], [])

Designed with Python 2.7 and SGE since thats what our system runs. The only non-standard Python libraries required are the included tools.py and log.py modules, and sh.py (also included)

Obviously not as helpful if you wish to stay purely in bash, but if you need to wait on qsub jobs then I would imagine your workflow is edging towards a complexity that would benefit from using Python instead.

user_53595310
Related