How to configure the content of slurm notification emails?

user_5332798 0
Viewed 8210

Slurm can notify the user by email when certain types of events occur using options such as --mail-type and --mail-user.

The emails I receive this way contain a void body and a title that looks like :

SLURM Job_id=9228 Name=toto Ended, Run time 07:32:31, COMPLETED, ExitCode 0

I'd like to configure slurm so that the title or even better the body of the email contains other informations in a similar way of what the slurm command squeue --format returns.

(actually I'd like the email to contain the comment I set up using sbatch --comment)

slurm
4 Answers

To customise the email sent by Slurm, you typically write a script and set the value of MailProg to the path to that script in your slurm.conf.

From the doc, MailProg is:

Fully qualified pathname to the program used to send email per user request. The default value is "/bin/mail" (or "/usr/bin/mail" if "/bin/mail" does not exist but "/usr/bin/mail" does exist).

The contrib directory of the Slurm source contains a script written in Perl that you can use and customise: https://github.com/SchedMD/slurm/tree/master/contribs/seff

user_17636140

You might also be interested in Slurm-Mail: https://github.com/neilmunday/slurm-mail

Slurm-Mail allows you to customise HTML emails sent to users and provides more information than the emails sent by Slurm by default.

user_118788670

See my solution here

#!/bin/bash
#SBATCH -J MyModel
#SBATCH -n 1 # Number of cores
#SBATCH -t 1-00:00 # Runtime in D-HH:MM
#SBATCH -o JOB%j.out # File to which STDOUT will be written
#SBATCH -e JOB%j.out # File to which STDERR will be written
#SBATCH --mail-type=BEGIN
#SBATCH --mail-user=my@email.com

secs_to_human(){
    echo "$(( ${1} / 3600 )):$(( (${1} / 60) % 60 )):$(( ${1} % 60 ))"
}
start=$(date +%s)
echo "$(date -d @${start} "+%Y-%m-%d %H:%M:%S"): ${SLURM_JOB_NAME} start id=${SLURM_JOB_ID}\n"

### exec task here
( << replace with your task here >> ) \
&& (cat JOB$SLURM_JOB_ID.out |mail -s "$SLURM_JOB_NAME Ended after $(secs_to_human $(($(date +%s) - ${start}))) id=$SLURM_JOB_ID" my@email.com && echo mail sended) \
|| (cat JOB$SLURM_JOB_ID.out |mail -s "$SLURM_JOB_NAME Failed after $(secs_to_human $(($(date +%s) - ${start}))) id=$SLURM_JOB_ID" my@email.com && echo mail sended && exit $?)

you can also edit this to send seperate stdout/stderr logs or attach them as files.

This snippet is also shared on github-gists

user_105972790

--mail-user=<user> User to receive email notification of state changes as defined by --mail-type . The default value is the submitting user.

And there is an odther parameter is : --mail-type=<type> , it allows you to Notify user by email when certain event types occur. Valid type values are NONE, BEGIN, END, FAIL, REQUEUE , INVALID_DEPEND , STAGE_OUT

user_164071700
Related