Create low priority slurm jobs that suspend or requeue if another job is submitted

user_14376959 0
Viewed 352

I am currently running a job on my schools HPC that contains 20 compute nodes. I would like to use all of these in a nice way so that if another student needs a compute node my job will be paused/ suspended and then be requeued when the resources become available again. My thought process is to submit 10 jobs that use two nodes each like so:

#!/bin/bash
#SBATCH --job-name=cpu-detect
#SBATCH --nodes=2
#SBATCH --ntasks=2
#SBATCH --mem=50G
#SBATCH --cpus-per-task=32
#SBATCH --partition=compute
srun conda run -n fires3.7 python detector.py

From what I have seen most students only request one node, so my thoughts are if a request is made, one of my jobs will be stopped and then requeued which will free up two nodes for that student to use, and then once that student is done, the job that was stopped will start again. Is this possible? I could not find too much information online.

slurm hpc
1 Answers

You can use scontrol suspend jobid to suspend your jobs if your architecture/configuration supports it. Your job should also supports this.

Slurm supports premption, the act of "stopping" one or more "low-priority" jobs to let a "high-priority" job run. It all depends on the way, the cluster is configured.

From slurm man page:

suspend <job_list>

Suspend a running job. The job_list argument is a comma separated list of job IDs. Use the resume command to resume its execution. User processes must stop on receipt of SIGSTOP signal and resume upon receipt of SIGCONT for this operation to be effective. Not all architectures and configurations support job suspension. If a suspended job is requeued, it will be placed in a held state. The time a job is suspended will not count against a job's time limit. Only an operator, administrator, SlurmUser, or root can suspend jobs.

You can resume it by scontrol resume jobid

resume <job_list> Resume a previously suspended job. The job_list argument is a comma separated list of job IDs. Also see suspend. NOTE: A suspended job releases its CPUs for allocation to other jobs. Resuming a previously suspended job may result in multiple jobs being allocated the same CPUs, which could trigger gang scheduling with some configurations or severe degradation in performance with other configurations. Use of the scancel command to send SIGSTOP and SIGCONT signals would stop a job without releasing its CPUs for allocation to other jobs and would be a preferable mechanism in many cases.

In my opinion, if your cluster supports suspend, you can indeed write a script in login node (not advising it because login node resources will be shared by all users). A script that runs in background and check for all the job information (using squeue). If there are any pending jobs, you can sent scontrol suspend jobid to your job (or cancel your job using scontrol cancel jobid). Then resume it when needed (if the job is suspended).

But as a responsible user, you don't need to worry about this since you should onlyrequest the resources that you need and run the job that is essential. It is indeed the responsibility of administrators to come up with a fair scheduling policy (by creating different queues like test queue for small jobs (fewer nodes) with less duration, micro queue for small jobs with long duration, large queue for large jobs etc) etc. Different policies could be employed by your institute to provide a fair policy. For example, a user cannot submit more than 2 jobs in a queue.

user_109119320
Related