using .e and .o files from qsub as log for each rule in snakemake

Viewed 245

In my snakemake workflow I am trying to redirect the the .e and .o files produced by qsub to the log directive of each rule. Below is an example of a rule that I have, including the log directive. To make it clear, this rule produces the (temporary) output [snakemakeWorkDir]/{sample}/aligned/{sample}.bam whereas I also would like to have the files

  1. [snakemakeWorkDir]/{sample}/aligned/log/{wildcards.sample}.{rule}.{jobid}.e[0-9]+ and
  2. [snakemakeWorkDir]/{sample}/aligned/log/{wildcards.sample}.{rule}.{jobid}.o[0-9]+

produced

rule bwa_map: 
    input:
        R1 = FQDIR + "{sample}_R1_001.fastq.gz",
        R2 = FQDIR + "{sample}_R2_001.fastq.gz",
        REF = config['ref']    
    output:
        # wrap output in temp
        temp("{sample}/aligned/{sample}.bam")
    conda:
        "envs/NIPTlibPrep.yaml"
    log:
        directory("{sample}/aligned/log")
    shell:
        "mkdir -p {log} | bwa mem {input.REF} {input.R1} {input.R2} "
        "| samtools view -Sb - > {output}"

The command that I am using to submit the snakemake jobs to the cluster is:

snakemake -p -s Snakefile_v5_ngs_bngs05b --cluster "qsub -q onlybngs05b -e {log} -o {
log}" -j 5 --use-conda --jobname "{wildcards.sample}.{rule}.{jobid}"

When I call the snakemake with the command above, my job gets the status Eqw on the queue with the following error message

 04/10/2020 10:37:30 [2003:28804]: error: can't open output file "/home/ngs/NIPT-PearlPPlasma-03-PPx_S3downSample/fastQC/log": No such file or directory

Even though the first command of the shell directive is to create the log directory (I had first tried without the mkdir -p command, and got the same error.)

Another option would be to omit the production of those files at all, and use the 'classic' snakemake way of producing the log files (e.g. using the 2> pipe)

0 Answers
Related