In my snakemake workflow I am trying to redirect the the .e and .o files produced by qsub to the log directive of each rule. Below is an example of a rule that I have, including the log directive. To make it clear, this rule produces the (temporary) output [snakemakeWorkDir]/{sample}/aligned/{sample}.bam whereas I also would like to have the files
[snakemakeWorkDir]/{sample}/aligned/log/{wildcards.sample}.{rule}.{jobid}.e[0-9]+and[snakemakeWorkDir]/{sample}/aligned/log/{wildcards.sample}.{rule}.{jobid}.o[0-9]+
produced
rule bwa_map:
input:
R1 = FQDIR + "{sample}_R1_001.fastq.gz",
R2 = FQDIR + "{sample}_R2_001.fastq.gz",
REF = config['ref']
output:
# wrap output in temp
temp("{sample}/aligned/{sample}.bam")
conda:
"envs/NIPTlibPrep.yaml"
log:
directory("{sample}/aligned/log")
shell:
"mkdir -p {log} | bwa mem {input.REF} {input.R1} {input.R2} "
"| samtools view -Sb - > {output}"
The command that I am using to submit the snakemake jobs to the cluster is:
snakemake -p -s Snakefile_v5_ngs_bngs05b --cluster "qsub -q onlybngs05b -e {log} -o {
log}" -j 5 --use-conda --jobname "{wildcards.sample}.{rule}.{jobid}"
When I call the snakemake with the command above, my job gets the status Eqw on the queue with the following error message
04/10/2020 10:37:30 [2003:28804]: error: can't open output file "/home/ngs/NIPT-PearlPPlasma-03-PPx_S3downSample/fastQC/log": No such file or directory
Even though the first command of the shell directive is to create the log directory (I had first tried without the mkdir -p command, and got the same error.)
Another option would be to omit the production of those files at all, and use the 'classic' snakemake way of producing the log files (e.g. using the 2> pipe)