Run parallelized snakemake jobs in a specific order

Question

I'm currently building a snakemake pipeline, where I need to:

1 - Split reference genomes according to chromosome.
2 - Do an unrelevant operation on them.
3 - Merge the chromosomes back into a full assembly.

Here is a snippet of what's happening at step 2:

rule get_consensus:
    input:      
        vcf = rules.filter_snps.output,
        chr_ref = "../data/g1k_v37/chromo/{chr}.fa"
    output:
        hap1=temp("{sample}/chr{chr}/{sample}_chr{chr}_haplo1.fa"),
        hap2=temp("{sample}/chr{chr}/{sample}_chr{chr}_haplo2.fa"),
    threads: 2
    priority: 1
    shell:
        "bcftools consensus -H 1 -f {input.chr_ref} --sample {wildcards.sample} {input.vcf} > {output.hap1} \
         & \
         bcftools consensus -H 2 -f {input.chr_ref} --sample {wildcards.sample} {input.vcf} > {output.hap2}"

To mitigate memory usage I must have snakemake perform the third operation ASAP, meaning it must prioritize the order of jobs in step 2 according to chromosomes, not samples:

wildcards: sample=sample1, chr=01
wildcards: sample=sample1, chr=02
wildcards: sample=sample1, chr=03
...

Sadly, snakemake does just the opposite:

wildcards: sample=sample1, chr=01
wildcards: sample=sample2, chr=01
wildcards: sample=sample3, chr=01
...

This implies my pipeline currently accumulates all of the splitted fastas (until all of them have been generated), before merging any of them using the rule on step 3 (shown below):

rule merge_chromosomes:
    input:
        hap1=expand(rules.get_consensus.output.forw, chr=range(1,23), sample="{sample}"),
        hap2=expand(rules.get_consensus.output.rev, chr=range(1,23), sample="{sample}")
    output:
        hap1=temp("{sample}_s1.fq.gz"),
        hap2=temp("{sample}_s2.fq.gz")
    threads: 4
    priority: 3
    shell:
        "zcat {input.hap1} | gzip > {output.hap1} \
         & \
         zcat {input.hap2} | gzip > {output.hap2} \
        "

My question is:, is there any way to force snakemake to prioritize on the {chr} wildcard instead of the {sample} one, when generating the DAG?

I've tried tweaking with priority:, resources: and --prioritize, to no avail.

My problem pretty much looks like a complete mix of these two past issues:

Run parallelized snakemake jobs in a specific order

0 Answers