How to pipe input to SLURM's sbatch?

user_4504711 0
Viewed 298

I have a job script called testjob.sh which I submit as

sbatch testjob.sh

Is it possible to pipe some input to the script? If I ran as a simple command, I would do

echo 1 2 3 4 | ./testjob.sh

however, with SLURM, I tried the following and it does not work:

echo 1 2 3 4 | sbatch testjob.sh

My example script is this:

#!/bin/bash
#SBATCH --export=ALL
#SBATCH --job-name=jobname
#SBATCH -e jobname.err
#SBATCH -o jobname.out
#SBATCH -p RM-shared
#SBATCH -n 1
#SBATCH --time=00:05:00

echo "jobscipt"
while read -r input 
do
        echo "from jobscipt $input"
done

and the outputs:

#this one worked
> echo 1 2 3 4 | ./testjob.sh
jobscipt
from jobscipt 1 2 3 4

#slurm did not take the input
> echo 1 2 3 4 | sbatch testjob.sh
Submitted batch job 3758713
> cat jobname.out
jobscipt

I know it's possible to set arguments to sbatch via setting environmental variables with --export, but how would I pipe an input to it? A workaround to pipe would be a temporary file but that's not particularly elegant.

slurm hpc linux
1 Answers

sbatch runs the job in a different environment ("in the background"), therefore you can't pipe stuff into the scripts.

You can avoid this in two ways:

  • use srun instead of sbatch, which doesn't disconnect the job from your session and piping works more-or-less normally. Still, this doesn't allow you to "queue" the job as with batches.
  • use intermediate file for saving the input, i.e., modify the batch file to read the input from testjob.in:
while read -r input
do
   ...
done < testjob.in

and put your input into the intermediate file before enqueueing the batch:

echo 1 2 3 4 > testjob.in
sbatch testjob.sh
user_10430970
Related