How to skip intermediate rules with ancient?

user_10192444 0
Viewed 803

I want to skip intermediate rules if their output file already exists.

For example, I want to skip the second rule if it ran before in this basic Snakefile:

rule third:
    input:
        "b",
        "c"
    output: "d"
    shell: "touch {output}"

rule second:
    input: ancient("b")
    output: "c"
    shell: "touch {output}"

rule first:
    input: "a"
    output: "b"
    shell: "touch {output}"

After touching a, this pipeline works fine.

When I touch a again, I would expect that the second rule will be skipped, as the input is flagged as ancient. This is not what happens, instead the while pipeline is run again.

Flagging c in the third rule instead of b in the second rule as ancient also has no effect.

Update: Using the suggestion, I updated the Snakefile to skip the second rule if it ran before (and is not forced):

ruleorder: cached_c > second

rule third:
    input:
        "b",
        "c"
    output: "d"
    shell: "touch {output}"

rule second:
    input: "b"
    output:
        c = "c"
    shell: '''
    touch {output.c}
    cp {output.c} cache
    '''

rule cached_c:
    input: "cache"
    output: "c"
    shell: "cp {input} {output}"

rule first:
    input: "a"
    output: "b"
    shell: "touch {output}"
snakemake
3 Answers

Ancient indeed prevents a rule from running if the input is changed, but not when snakemake does the changing. You can run snakemake -nr to see the reasons for running a rule, and in the case you present rule second is run because Input files updated by another job: b. I've run into something similar but I thought it was an interplay between temp and ancient, I didn't realize it was more general. However, if you touch b instead of a, rule second will not run. Ancient is preventing it from running because the input file modified time is ignored; Demitry is incorrect in his description.

While caching will work, it adds another file to the disk and in real examples may be less than ideal. Furthermore, in a case where rule third only depends on c, it will still be rerun since the modified time of c is updated with the copying. My go to is an input function that will return an empty string (or dict if using unpack) when the output file already exists. This is a stronger version of ancient that will only generate the output if it doesn't exist. For your example:

import os

rule third:
    input:
        "b",
        "c"
    output: "d"
    shell: "touch {output}"

def second_input(wildcards):
    if os.path.exists("c"):
        return ""
    else:
        return "b"

rule second:
    input: second_input
    output: "c"
    shell: "touch {output}"

rule first:
    input: "a"
    output: "b"
    shell: "touch {output}"

Now touching a will only cause rules first and third to run because a is updated, then b is updated.

user_123737910

Why are you assuming that the second rule would be skipped? The ancient modifier does the opposite:

For determining whether output files have to be re-created, Snakemake checks whether the file modification date (i.e. the timestamp) of any input file of the same job is newer than the timestamp of the output file. This behavior can be overridden by marking an input file as ancient. The timestamp of such files is ignored and always assumed to be older than any of the output files

So instead of always skipping the rule you are always running it, even if nothing has changed.

The fact that the rule has a dependency means that you expect the output of this rule to be reevaluated whenever the input is changed. You may break this dependency manually by creating a separate rule that gets it's input from cache. The cached version has to have no dependencies (thus would never be reevaluated), and you may manually copy the files whenever you decide they are good enough for being cached.

One alternative is to copy the files into the cache directory automatically after each successful run (or before each run if you find the artefacts): https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#onstart-onsuccess-and-onerror-handlers

onsuccess:
    # Copy your files here

Update: The quote from the Snakemake reference that I provided in my message is ambiguous, and could be understood in two ways. It doesn't say however what does Snakemake do with the timestamps of ancient inputs linking them to other rules that may affect this file.

Regarding caching, personally I haven't touched the Between workflow caching, but that is not what I meant. Snakemake has no idea whether the changed source would affect the cached file unless it reevaluates the file and compares the hashes. If you wish to avoid that (and you are pretty sure that the change in the source wouldn't affect the rest of the pipeline), you may create two separate branches to create a target. For example, in your case you need a file called c: I advise you to have two separate rules for that: the first takes b as an input, the other takes cached_b. You need to disambiguate these two rules with rulesorder giving the cached version priority. The strategy how this cached_b is being created is up to you: either manualy copy b to cached_b or a script that is being run onsuccess.

user_112186870

This is actually a bug and the given example should have produced the correct behavior.

The related issue can be found here. The fix was merged today (2020-10-15).

user_101924440
Related