Select nodes to execute Nextflow workflow with Kubernete

user_11221274 0
Viewed 119

I have access to a cluster with a lot of nodes. I am running my Nextflow workflow using this command:

./nextflow kuberun user/repo -c nextflow.config -profile kubernetes -v my_pvc:/mounted_path -with-report _report.html -with-trace _trace

I would like to run my nextflow workflow on a specific set of nodes. I have have already labeled my nodes of interest:

kubectl label nodes node1 disktype=my_experiment
kubectl label nodes node2 disktype=my_experiment
kubectl label nodes node3 disktype=my_experiment

I am not understanding from Nextflow and Kubernete documentation how to is it possible to schedule my workflow, with the processes splitted between my nodes of interest.

I understand how to do with kubernete only: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/

But not how to the nextflow kuberun command.

Any help is very appreciated, thank you!

nextflow kubernetes-pod kubernetes
1 Answers

Use one or more process selectors and the pod directive to select the nodes using a pod label. For example, the following could be added to your 'kubernetes' profile:

process {

    withName: my_process {
        pod {
            nodeSelector = 'disktype=my_experiment'
         }
     }
    ...
}
user_7518630
Related