Using SnakeMake, how to pass SLURM flags with dashes in their names?

user_2966505 0
Viewed 228

I'm using Snakemake to execute rules on a SLURM cluster.

One of the mandatory flags for this cluster is ntasks-per-node, which in a batch script would be specified as e.g. #SBATCH --ntasks-per-node=5. My understanding is that I need to specify this in a snakemake rule as

rule rule_name:
    ...
    resources:
        time='00:00:30', #30 sec
        ntasks-per-node=1
    ...

However, running this Snakefile I get

SyntaxError in line 14 of .../Snakefile:
keyword can't be an expression

because there are dashes in the name. But as far as I can tell, replacing the dashes with underscores doesn't work. What should I do here?

(I'm using the SLURM profile here if that matters)

snakemake
1 Answers

Try quoting. But more importantly, only the resources that are defined in the RESOURCE_MAPPING variable in the slurm_submit.py will be picked up, and the default cookiecutter does not include an ntasks-per-node argument. Hence, quoting alone won't solve the issue.

There are multiple options.

  1. Edit the slurm_submit.py. Add the ntasks-per-node argument and provide whatever alias(es) you would like to use.

    RESOURCE_MAPPING = {
  "time": ("time", "runtime", "walltime"),
  "mem": ("mem", "mem_mb", "ram", "memory"),
  "mem-per-cpu": ("mem-per-cpu", "mem_per_cpu", "mem_per_thread"),
  "nodes": ("nodes", "nnodes"),
  # some suggested aliases
  "ntasks-per-node": ("ntasks-per-node", "ntasks_per_node", "ntasks")
}

    I would only do this if there actually are situations where you might change this value.

  2. Define an invocation-level configuration. Snakemake's --cluster_config parameter can still be used to provide additional configuration settings. In this case, a file like

    # myslurm.yaml
__default__:
    ntasks-per-node: 1

    Then use it with

    snakemake --profile slurm --cluster_config myslurm.yaml

    This is likely the least work to get going.

  3. Define a global value in the profile. The cookiecutter profile generator provides multiple options to define global options that don't often need to change for the profile.

user_5709180
Related