snakemake - replacing command line parameters with wildcards by cluster profile

Viewed 135

I am writing a snakemake pipeline to eventually identify corona virus variants.

Below is a minimal example with three steps:

LOGDIR = '/path/to/logDir'
barcodes = ['barcode49', 'barcode50', 'barcode51']

rule all:
    input:
        expand([
            # guppyplex
            "out/guppyplex/{barcode}/{barcode}.fastq",

            # catFasta
            "out/catFasta/cat_consensus.fasta",
        ], barcode = barcodes)


rule guppyplex:
    input:
        FQ = f"fastq/{{barcode}}" # FASTQ_PATH is parsed from config.yaml
    output:
        "out/guppyplex/{barcode}/{barcode}.fastq"
    shell:
        "touch {output}" # variables in CAPITALS are parsed from config.yaml


rule minion:
    input:
        INFQ = rules.guppyplex.output,
        FAST5 = f"fasta/{{barcode}}"
    params:
        OUTDIR = "out/nanopolish/{barcode}"
    output:
        "out/nanopolish/{barcode}/{barcode}.consensus.fasta"
    shell:
        """
        touch {output} && echo {wildcards.barcode} > {output}
        """

rule catFasta:
    input:
        expand("out/nanopolish/{barcode}/{barcode}.consensus.fasta", barcode = barcodes)
    output:
        "out/catFasta/cat_consensus.fasta"
    shell:
        "cat {input} > {output}"

If I run the snakemake locally by calling snakemake -p --cores 1 all everything works. Yet my ultimate goal is to use qsub to run the jobs on a cluster. I also want the stderr and stdout from qsub to have meaningful names, which include wildcards and the rule names for each job.

However, if I call snakemake with

snakemake -p --cluster "qsub -q onlybngs05b -e {LOGDIR} -o {LOGDIR} -j y" -j 5 --jobname "{wildcards.barcode}.{rule}.{jobid}" all

I will get the following error:

AttributeError: 'Wildcards' object has no attribute 'barcode'

I have recently read the snakemake documentation where it appears that I could replace the command line parameters (--cluster "qsub -q onlybngs05b -e {LOGDIR} -o {LOGDIR} -j y" -j 5 --jobname "{wildcards.barcode}.{rule}.{jobid}") by a yaml file. Although the documentation is not all that clear to me.

I have created a config.yaml file at /home/user/.config/snakemake which looks like so:

cluster: 'qsub'
q: 'onlybngs05b'
e: '/home/ngs/tempOutSnakemake'
o: '/home/ngs/tempOutSnakemake'
j: 5
jobname: "{wildcards.barcode}.{rule}.{jobid}

But then it appears that snakemake is not properly parsing the config.yaml. I am getting

snakemake: error: ambiguous option: --o=/home/ngs/tempOutSnakemake could match --omit-from, --output-wait, --overwrite-shellcmd

I also tried to replace o in the config file by stdout (kind of the long version of the parameter (-h vs --help for several programs), though it does not work.

Therefore my question is how I can replace the command line parameters --cluster "qsub -q onlybngs05b -e {LOGDIR} -o {LOGDIR} -j y" -j 5 --jobname "{wildcards.barcode}.{rule}.{jobid}" by a config.yaml file that accepts wildcards?

1 Answers

I think the problem is that rule catFasta doesn't contain the wildcard barcode. If you think about it, what job name would you expect in {wildcards.barcode}.{rule}.{jobid}?

Maybe a solution could be to add to each rule a jobname parameter that could be {barcode} for guppyplex and minion and 'all_barcodes' for catFasta. Then use --jobname "{params.jobname}.{rule}.{jobid}"

Related