Pass path for `publishDir` to NextFlow Processes

Question

I'm using NextFlow with DSL2 syntax. I'd like to define the final results path as a command-line argument. As far as I can tell, however, NextFlow processes don't recognize an input variable in the directive scope (see minimal example below).

Ultimate question: How can I pass a "base directory" to a process that can then append to that path (e.g., publishDir("${results_dir}/MY_PROC_RESULTS/", mode = "copy"))? Also open to other suggestions if this isn't the best approach.

Really appreciate any help!

Here's a minimal example:

MY_WORKFLOW.nf

/*
 * Make this pipeline a nextflow 2 implementation
 */
nextflow.enable.dsl=2

include {SUB_WORKFLOW_WF} from './SUB_WORKFLOW_WF_PROCS.nf'

/*
 * Define the directory to publish final results in.
 */
params.results_dir = "${projectDir}/results"

results_dir = params.results_dir

workflow{
    SUB_WORKFLOW_WF(results_dir)
}

SUB_WORKFLOW_WF_PROCS.nf

workflow SUB_WORKFLOW_WF {
    take:
        results_dir

    main:
        println results_dir // This works
        MY_PROC(results_dir)
}

process MY_PROC {
    /*
     * Workflow fails with ERROR 1 if the following line is included. Fails
     * with ERROR 2 if the following line is excluded.
     */
    println $results_dir
    publishDir("${results_dir}/MY_PROC_RESULTS/", mode = "copy")

    input:
        path(results_dir)

    script:
        """
        echo "Hello"
        """
}

Results

ERROR 1

No such variable: $results_dir

 -- Check script './SUB_WORKFLOW_WF_PROCS.nf' at line: 11 or see '.nextflow.log' file for more details

ERROR 2

Error executing process > 'SUB_WORKFLOW_WF:MY_PROC'

Caused by:
  Not a valid PublishDir entry [org.codehaus.groovy.runtime.GStringImpl] null/MY_PROC_RESULTS

user_751863 · Answer

You get the first error because there is no '$results_dir' variable in the global scope. The second error is a bit more cryptic, but is actually just a syntax error. The 'mode' declaration just needs a colon, like a Groovy Map to separate keys and values:

publishDir("${results_dir}/MY_PROC_RESULTS/", mode: "copy")

So rather than passing around a variable, start by defining your pipeline parameters at the beginning of the script before any 'include' declaration. These params will then be inherited from the including context. For example:

MY_WORKFLOW.nf

nextflow.enable.dsl=2

params.publish_dir = './results'

include { SUB_WORKFLOW_WF } from './SUB_WORKFLOW_WF_PROCS.nf'


workflow{
    
    SUB_WORKFLOW_WF()
}

SUB_WORKFLOW_WF_PROCS.nf

workflow SUB_WORKFLOW_WF {

    MY_PROC()
}

process MY_PROC {
    
    publishDir(
        path: "${params.publish_dir}/MY_PROC_RESULTS",
        mode: 'copy',
    )

    output:
    path "proc_results.txt"

    """
    touch "proc_results.txt"
    """
}

Results:

$ nextflow run ./MY_WORKFLOW.nf 
N E X T F L O W  ~  version 21.04.3
Launching `./MY_WORKFLOW.nf` [shrivelled_montalcini] - revision: d31b4930b9
executor >  local (1)
[2b/08e223] process > SUB_WORKFLOW_WF:MY_PROC [100%] 1 of 1 ✔

$ find ./results/MY_PROC_RESULTS/proc_results.txt 
./results/MY_PROC_RESULTS/proc_results.txt

Pass path for `publishDir` to NextFlow Processes

1 Answers