Match-zero-or-more operator in nextflow glob

user_7562715 0
Viewed 97

I am trying to create a robust glob pattern that will match most of the different naming conventions used for fastq files we receive. However, the version of nextflow I am using (20.10.0) on the HPC doesn't seem to accept what I've written.

Here are some examples of file names:

19_S8_R1_001.fastq.gz    19_S8_R2_001.fastq.gz
F1HD1_S28_R1.fastq.gz    F1HD1_S28_R2.fastq.gz 
SRR3137747_1.fastq       SRR3137747_2.fastq

The pattern I originally wrote to go with the fromFilePairs operator was *_?(R){1,2}?(_001).f?(ast)q?(.gz). Which I tested in a bash environment. Here is the output from testing in the directory with the top two example files:

-bash-4.2$ shopt -s extglob
-bash-4.2$ ls -1 *_?(R){1,2}?(_001).f?(ast)q?(.gz)
19_S8_R1_001.fastq.gz
19_S8_R2_001.fastq.gz

But when I tried to run this with nextflow, it just gave me the error message I put into the ifEmpty operator.

I've eventually got it working, but using this pattern: *_{R1,R2,1,2}{.fastq.gz,.fq.gz,.fastq,.fq,_001.fastq.gz,_001.fq.gz,_001.fastq,_001.fq}, which isn't particularly robust.

Unless I've missed it in the nextflow documentation (and the information I've found about glob), I don't see alternatives to match-zero-or-more operators in nextflow. Any alternative solutions?

Thanks in advance.

nextflow glob
1 Answers

The following glob pattern seems to match some of the more common FASTQ filenames:

Channel
    .fromFilePairs( '*_{,R}{1,2}{,_001}.{fq,fastq}{,.gz}' )
    .view()

Or with a parameterized directory prefix:

Channel
    .fromFilePairs( "${params.input_dir}/*_{,R}{1,2}{,_001}.{fq,fastq}{,.gz}" )
    .view()

Results:

N E X T F L O W  ~  version 21.04.3
Launching `script.nf` [serene_austin] - revision: 5527b9b3c0
[SRR3137747, [/path/to/fasta/SRR3137747_1.fastq, /path/to/fasta/SRR3137747_2.fastq]]
[19_S8, [/path/to/fasta/19_S8_R1_001.fastq.gz, /path/to/fasta/19_S8_R2_001.fastq.gz]]
[F1HD1_S28, [/path/to/fasta/F1HD1_S28_R1.fastq.gz, /path/to/fasta/F1HD1_S28_R2.fastq.gz]]

Another option, which might be more robust (and readable), is to make use of the fact that you can specify more than one glob pattern using a list as argument, and build your list of glob patterns dynamically:

nextflow.enable.dsl=2

params.input_dir = '/path/to/fasta'


def cartesian_product(A, B) {
    A.collectMany{ a -> B.collect { b -> [a, b] } }
}

def extensions = [
    '.fastq.gz',
    '.fastq',
    '.fq.gz',
    '.fq',
]

def suffixes = [
    '*_R{1,2}_001',
    '*_R{1,2}',
    '*_{1,2}',
]

workflow {

    def patterns = cartesian_product(suffixes, extensions).collect { 
        "${params.input_dir}/${it.join()}"
    } 

    Channel.fromFilePairs( patterns ).view()
}

Results:

N E X T F L O W  ~  version 21.04.3
Launching `script.nf` [deadly_payne] - revision: 6d2472ef23
[19_S8, [/path/to/fasta/19_S8_R1_001.fastq.gz, /path/to/fasta/19_S8_R2_001.fastq.gz]]
[F1HD1_S28, [/path/to/fasta/F1HD1_S28_R1.fastq.gz, /path/to/fasta/F1HD1_S28_R2.fastq.gz]]
[SRR3137747, [/path/to/fasta/SRR3137747_1.fastq, /path/to/fasta/SRR3137747_2.fastq]]
user_7518630
Related