Do submitted jobs take a copy the source? Queued jobs?

user_10518066 0
Viewed 132

When submitting jobs with sbatch, is a copy of my executable taken to the compute node? Or does it just execute the file from /home/user/? Sometimes when I am unorganised I will submit a job, then change the source and re-compile to submit another job. This does not seem like a good idea, especially if the job is still in the queue. At the same time it seems like it should be allowed, and it would be much safer if at the moment of calling sbatch a copy of the source was made.

I ran some tests which confirmed (unsurprisingly) that once a job is running, recompiling the source code has no effect. But when the job is in the queue, I am not sure. It is difficult to test.

edit: man sbatch does not seem to give much insight, other than to say that the job is submitted to the Slurm controller "immediately".

slurm
1 Answers

The sbatch command creates a copy of the submission script and a snapshot of the environment and saves it in the directory listed as the StateSaveLocation configuration parameter. It can therefore be changed after submission without effect. But that is not the case for the files used in the submission script. If your submission script starts an executable, if will see the "version" of the executable at the time it starts. Modifying the program before it starts will lead to the new version being run, modifying it during the run (i.e. while it has already been read from disk and saved into memory) will lead to the old version being run.

user_17636140
Related