I sometimes run jobs on the scavenge partition on my HPC system. However, I've never had a job that was preempted, which should happen when a higher priority user requests to use a particular scavenge node.
Instead, my jobs regularly get canceled. I get something similar to following message in my all of my canceled slurm out files:
...
3540500 2969.8072 0.66356586 0.19427035 1.1890431 1.7702431 14.373949 14.373949 14.373949
3541000 2988.6985 0.63768049 0.16779792 1.1237651 1.6317543 14.404362 14.404362 14.404362
3541500 3000.7733 0.65101783 0.1871139 1.1630755 1.7166775 14.423735 14.423735 14.423735
slurmstepd: error: *** STEP 2943114.0 ON p08r03n30 CANCELLED AT 2022-09-16T00:48:13 ***
srun: error: p08r03n30: tasks 0-3: Terminated
srun: Force Terminated StepId=2943114.0
Fri Sep 16 00:48:13 EDT 2022
Completed in 0:01:04:57.589
This makes it difficult because I need to go in and manually requeue these jobs. Since I may only catch them after my first set of jobs have already finished, this occasionally doubles the time needed to run any set of jobs.
I asked my system administrators about this, but the last time I heard from them they weren't quite sure why jobs were getting canceled.
Is there any documentation or setting for why a job would be for terminated rather than preempted (to allow for requeueing in the future)?