Snakemake rule log/benchmark wildcards dont match output wildcards after checkpoint

user_15381556 0
Viewed 107

I am running a Snakemake workflow with a checkpoint at some point from which I gather the previously unknown number of output files. Snakemake should then create a number of tasks based on the file number with the next rule, using some part of the gathered checkpoints files as wildcards for that rules wildcards. It all works fine, unless I want that rule to also create log and/or benchmark files, at which point is throws:

SyntaxError:
Not all output, log and benchmark files of rule plasmid_spades contain the same wildcards. This is crucial though, in order to avoid that two or more jobs write to the same file.
  File "/path/to/Snakefile", line N, in <module>

These are the relevant parts of the workflow:

WCS = None

...

def gather_checkpoint_output(wildcards):
    ck_output = checkpoints.checkpoint_rule.get(**wildcards).output[0]
    global WCS
    WCS, = glob_wildcards(os.path.join(ck_output, "{wc}", "{wc}.file"))
    return expand(os.path.join(ck_output, "{wc}", "{wc}.file"), wc=WCS)


def gather_some_rule_after_checkpoint_out(wildcards):
    rule_output = checkpoints.checkpoint_rule.get(**wildcards).output[0]
    WCS2, = glob_wildcards(os.path.join(rule_output, "{wc}", "{wc}.file"))
    return expand(os.path.join("some", "{wc}", "path", "output.file"), wc=WCS2)

...

localrules: all
rule all:
    input:
        gather_checkpoint_output,
        gather_some_rule_after_checkpoint_out

...

rule some_rule_after_checkpoint:
    input:
        input = gather_checkpoint_output
    output:
        out_dir = directory(expand(os.path.join("some", "{wc}", "dir"), wc=WCS)),
        output = expand(os.path.join("some", "{wc}", "path", "output.file"), wc=WCS)
    log:
         os.path.join("logs", "some", "path", "{wc}_rule.log")
    benchmark:
         os.path.join("logs", "some", "path", "{wc}_rule_benchmark.tsv")
...

Is the problem, that it evaluates the logs/benchmarks wildcard in the beginning (WCS = None), while the output will be reevaluated with the checkpoint functions? Although, a rules wildcards are based off of the outputs wildcards, I think. I tried lambda functions, expand(), etc., to specifically get the wildcards from (the hopefully reevaluated WCS) for the logs, but that is apparently not permitted. Am I overlookig something obvious here or is the entire construction wrong somehow?

checkpoint snakemake wildcard syntax-error logging
1 Answers

Your issue might be an avatar of the infamous and very common "wrong use of expand" general problem.

In the some_rule_after_checkpoint rule the log and benchmark directives contain a wildcard, while the output directive doesn't. Indeed, you need to be well aware of the fact that the wildcards in the output file name patterns are expanded, resulting in a list of fully resolved file names.

This confuses Snakemake: What value of the wildcard should it use to determine the names of the log and benchmark files if there is no wildcard in the output file names? Wildcard values in a rule are determined by matching this rule's output file name patterns with fully-resolved input file names in a downstream rule.

You should likely not use expand in the outputs of some_rule_after_checkpoint, since the expanding is already done in the input of the all rule.

With non-expanded output file name patterns in some_rule_after_checkpoint, each different file in the expanded input of rule all will trigger one instance of the some_rule_after_checkpoint rule, for which the value of the wildcard will be determined based on a pattern matching between the output file patterns in some_rule_after_checkpoint and the desired "fully resolved" input for all. This wildcard will then enable Snakemake to generate the corresponding log and benchmark file.

user_18787880
Related