`nohup` issue with submitting `SLURM` job

user_14713401 0
Viewed 74

I have a python code main.py that runs bash script, the bash script inturn submits a job job.bash and obtains its JOBID using echo $JOBID | awk {'print $4'}. If I run python in the terminal, the bash script works and I am able to obtain and echo the JOBID as follows:

#!/bin/bash 
JOBID=`sbatch ~/job.bash  | tee  output.log`
JOBID=`echo $JOBID | awk {'print $4'}`
echo $JOBID

Running above as part of python works in terminal python main.py, but doing nohup python main.py &, the echo does not print or store JOBID.

Any reason for this?

I am submitting a slurm job hence the JOBID is the pid from slurm

(Update Jul 17) Looks like the issue is with the command sbatch ~/job.bash | tee output.log, it doesnt get submitted using nohup and hence JOBID never gets stored and echo'd.

(Update Jul 18) As per the comments from @pynexj adding set -x in the script results:

nohup: ignoring input and redirecting stderr to stdout
+ date
Mon Jul 18 21:46:35 +03 2022
++ sbatch ~/job.bash
++ tee output.log
+ JOBID=
++ echo
++ awk '{print $4}'
+ JOBID=
+ echo

The issue still persists. It appears that nohup is incompatible with sbatch.

Question: Why should nohup prevent submission of slurm job? Its objective is merely to capture terminate signal?

nohup linux bash
1 Answers

If this problem only happens with nohup present, you can get the benefits of nohup without actually using it with:

yourscript </dev/null >file.log 2>&1 & disown -h "$!"

This does the following:

  • Redirects stdin from /dev/null with </dev/null
  • Redirects stdout and stderr to a log file with >file.log 2>&1
  • Tells the shell not to forward HUP signals to the background process with disown -h "$!"

...which is everything nohup does.

user_141220
Related