Using expand() in snakemake to input any file from list of directories

user_17785015 0
Viewed 277

I have a rule that takes any and every TSV file (multiple TSVs) from a list of directories defined as tasks. For example:

tasks
    foo
        example1.tsv
        circle.tsv
    bar
        rectangle.tsv
    square
        triangle.tsv
        triangle_1.tsv

I then have a rule in a Snakemake workflow that runs a script on the list of files as such:

task_list = ["bar", "square"]

rule gather_files:
        input:
            tsv=expand("results/stats/{tasks}/*.tsv", tasks=task_list)
        output: 
            "results/plots/visualizations.pdf"
        script:
            "Rscript plot_script.R"

The *.tsv results in errors when I try to run the rule and I know it's not the correct way either. What is the best way to do this? Should I use regex to match any string in {task}/*.tsv? I want to limit the combinations of directories to expand on (tasks) but have no constraints on the filenames in them.

snakemake wildcard python
2 Answers

This is not very elegant but should work

from glob import glob

task_stats = ["foo", "bar", "square"] 

rule combine_files:
        input:
            tsv=[j for i in expand("results/stats/{tasks}/*.tsv", tasks=task_stats) for j in glob(i)]
        output: 
            "results/plots/stats_visualizations.html"
        script:
            "../scripts/plot_all_stats.Rmd"
user_121801130

I've had the same question, and a hacky way that I used to solve this problem was by passing a directory as an input, and a matching pattern as a parameter of the rule:

rule analysis:
    params:
        csvglob = "*.csv"
    input:
        folder="results/stats/{tasks}"
    output: 
        "results/plots/stats_visualizations.html"
    script:
        "../scripts/plot_all_stats.Rmd"

And in my script I read from that parameter as

rootdir = snakemake.input["folder"]
csvglob = snakemake.params["csvglob"]
files = glob(f"{rootdir}/{csvglob}")

I take a similar approach would also work for R.

Downside: - feels hacky. Upside: - quite easy to change the pattern, or manipulate and filter it from within the script.

user_35680920
Related