Can Snakemake run cluster jobs on a user-defined subset of data?

Viewed 15

I was wondering whether there is any possibility to create cluster jobs with snakemake which will run a batch of tasks. Assuming the following input:

image_00.tif
image_01.tif
...
image_99.tif

I would like to create the following output:

meas_image_00.txt
meas_image_02.txt
...
meas_image_99.txt

But instead of creating 100 jobs, I would like to submit 5 jobs with 20 input files each, creating 20 output files. I thought about the group flag but I am not sure whether this is really doing what I want. The only other way I can imagine is to additionally create a batch file as output and ignore that there will be per image outputs. The expected files then would be something like this:

batch_1.summary
batch_2.summary
batch_3.summary
batch_4.summary
batch_5.summary

But I would loose control if the meas_image*.txt are created at all.

Does anybody have an advice how to solve that?

Edit: I played around with an example using rules generated by loops. I just concentrated on creating output in batches for now but something goes wrong. This is my snakefile:

data=list(range(0,100))
samples = ["image_" + str(i).zfill(2) for i in data]
batch_size = 5


def chunks(lst, n):
    """Yield successive n-sized chunks from lst."""
    for i in range(0, len(lst), n):
        yield lst[i:i + n]

def chunklist():
    chunk_dict = {}
    for i, chunk in enumerate(chunks(samples, batch_size)):
        chunk_dict[i] = chunk
    return chunk_dict

cd = chunklist()

rule all:
  input: 'rule_3.out'
  
for i,chunk in cd.items():
    rule:  # batch rule
        name: "my_rule_{}".format(i)
        output: expand("{sample}.txt", sample=chunk)
        group: 'batch_group'
        run: 
            print(output)
            print(chunk)
            for file in chunk:
                print("Creating {} of {}".format(file, chunk))
                touch('{}.txt'.format(file))

for i in cd:
    rule:  # batch rule
        name: "batch_rule_{}".format(i)
        input: expand('{sample}.txt', sample=cd[i])
        group: 'batch_group'
        output: touch('{}.group'.format(i))


rule rule3:
  input: 
    files=expand('{sample}.txt', sample=samples),
    batches=expand('{i}.group', i=range(len(samples) // batch_size - 1))  # should be ceil in general
  output:
    'rule_3.out'
  shell: 'cat {input.files} > {output}'

And this is the problem, I get:

rule my_rule_10:
    output: image_50.txt, image_51.txt, image_52.txt, image_53.txt, image_54.txt
    jobid: 12
    reason: Missing output files: image_51.txt, image_54.txt, image_53.txt, image_52.txt, image_50.txt
    resources: tmpdir=/tmp

image_50.txt image_51.txt image_52.txt image_53.txt image_54.txt
['image_95', 'image_96', 'image_97', 'image_98', 'image_99']
Creating image_95 of ['image_95', 'image_96', 'image_97', 'image_98', 'image_99']
Creating image_96 of ['image_95', 'image_96', 'image_97', 'image_98', 'image_99']
Creating image_97 of ['image_95', 'image_96', 'image_97', 'image_98', 'image_99']
Creating image_98 of ['image_95', 'image_96', 'image_97', 'image_98', 'image_99']
Creating image_99 of ['image_95', 'image_96', 'image_97', 'image_98', 'image_99']

It's strange to me as output is generated by chunk but then differs from the variable chunk... I have no clue what goes wrong. This example should be executable for everybody. I would be happy to get help.

0 Answers
Related