In snakemake, how do you use wildcards with scatter-gather processes?

Viewed 412

I am trying to use snakemake's scatter-gather functionality to parallelize a slow step in my workflow. However, I cannot figure out how to apply it in situations where I am using wildcards. For example, I have defined the wildcard library in rule all, however, this does not seem to apply to the scatter function in ScatterIntervals:

import re

SCATTER_COUNT = 100
scattergather: 
    split=SCATTER_COUNT

rule all:
    input:
        expand("{library}_output.txt", library=["FC19271512", "FC19271513"])


rule ScatterIntervals:
    input:
        "{library}_baits.interval_list"
    output:
        temp(scatter.split("tmp/{library}_baits.scatter_{scatteritem}.interval_list"))
    params:
        output_prefix = (
            lambda wildcards, output: 
            re.sub("\.scatter_\d+\.interval_list", "", output[0])
        ),
        scatter_count = SCATTER_COUNT
    shell:
        """
        python ScatterIntervals.py \
            -i {input} \
            -o {params.output_prefix} \
            -s {params.scatter_count}
        """


rule ProcessIntervals:
    input:
        bam = "{library}.bam",
        baits = "tmp/{library}_baits.scatter_{scatteritem}.interval_list"
    output:
        temp("tmp/{library}_output.scatter_{scatteritem}.txt")
    shell:
        """
        python ProcessIntervals.py \
            -b {input.bam} \
            -l {input.baits} \
            -o {output}
        """


rule GatherIntervals:
    input:
        gather.split("tmp/{library}_output.scatter_{scatteritem}.txt")
    output:
        "{library}_output.txt"
    run:
        inputs = "-i ".join(input)
        command = f"python GatherOutputs.py {inputs} -o {output[0]}"
        shell(command)
    
WildcardError in line 16 of Snakefile: 
No values given for wildcard 'library'.
1 Answers

Evidently this works like expand, in that you can quote the wildcards that aren't scatteritem if you want DAG resolution to deal with them:

temp(scatter.split("tmp/{{library}}_baits.scatter_{scatteritem}.interval_list"))

The same logic applies for gather.split.

Related