How to implement splitting of files in snakemake when number of files is known

user_3017393 0
Viewed 114

Context

rule A uses the split command in a shell directive. The number of files generated by rule A depends on a user specified value from the config and is thus known.

In this question there is a difference because the number of output files is unknown, but there is a reference to the dynamic() keyword. Apparently this has been replaced by the use of checkpoint. Is this really the correct way to go in this scenario? There is also something like scattergatter but the example is not clear to me.

Code

chunks = config["chunks"]
sample_list = ["S1", "S2"]

rule all:
    input:
       expand("{sample}_chunk_{chunk}_done_something.tsv", sample=sample_list, 
             chunk=[f"{i}".zfill(len(str(chunks))-1) for i in range(0, chunks)])

rule A:
    input:
        "input_file_{sample}.tsv"
    output:
        # the user defined number of chunks, how to specify these?
    params: chunks=chunks
    shell:
        "split -n {params.chunks} --numeric-suffixes=1 --additional-suffix=.tsv {input[0]} some_prefix_{wildcards.sample}_"

rule B: 
    input:
        "some_prefix_{sample}_{chunk}.tsv"
    output:
        "{sample}_chunk_{chunk}_done_something.tsv"
    shell:
        "#Do something"

Attempts

I tried using a checkpoint with an input function for rule B and using directory() in rule A. However using directory results in SyntaxError in line 253 of MySnakefile: Unexpected keyword directory in rule definition (Snakefile, line 253) and even if that would not throw an error, I don't know how to get chunks into this input function since it is not a wildcard.

How to implement the splitting of an input file best in Snakemake?

snakemake
1 Answers

Since the number of chunks is known beforehand, you can set the number of output files in rule A from the chunks parameter using an array:

rule A:
    ...
    output:
        chunks = ["some_prefix_{{sample}}_{02d}.tsv".format(x+1) for x in range(chunks)]

With chunks = 2, this would expand to chunks = ["some_prefix_{sample}_01.tsv", "some_prefix_{sample}_02.tsv"], matching the synatx of the split output. The {sample} wildcard will be filled-in with Snakemake's standard wildcard replacement.

user_32037190
Related