I've installed dask using conda. When I create delayed functions and run them over my PBS cluster using dask, how do I ensure that the worker nodes activate the same conda environment before running the delayed functions?
1 Answers
This document gives you good general information about how imports and environments work across a dask cluster. In your PBS case, you may fall into the case where each workers see the same filesystem.
To configure the PBS launching, you should read this, where you can define various settings, including the paths that the workers will be searching. If you also/instead have local storage on the nodes, then you might want to distribute environments using one of the methods in the first link.
Related