Handling parallelization

user_5913101 0
Viewed 123

I'm a bit new on snakemake.

Imagine that I have a rule, like the one below (I've set the number of threads to 10).

Is there any way to make snakemake magically handles the parallelization of the loop for in this rule?

rule MY_RULE:
    input:
        input_file=TRAIN_DATA
    output:
        output_file=OUTPUT_DATA
    threads: 10
    run:        
        for f,o in zip(input.input_file, output.output_file):
            DO_SOMETHING_AND_SAVE(f,o)

Thanks

snakemake
1 Answers

I guess your rule could be re-written as (with additional code to make a small self-contained example):

TRAIN_DATA = ['a.txt', 'b.txt', 'c.txt']
OUTPUT_DATA = ['a.out', 'b.out', 'c.out']

files= dict(zip(OUTPUT_DATA, TRAIN_DATA))

wildcard_constraints:
    x= '|'.join([re.escape(x) for x in OUTPUT_DATA])

rule all:
    input:
        expand('{x}', x= OUTPUT_DATA),

rule MY_RULE:
    input:
        input_file= lambda wc: files[wc.x]
    output:
        output_file= '{x}'
    run:        
        DO_SOMETHING_AND_SAVE(input.input_file, output.output_file)

This will run rule MY_RULE for each input/output pair in parallel. Of course, the details depend on what exactly you want to do before and after MY_RULE...

user_11144530
Related