NoModuleFoundError: No module named 'distributed'

user_13224846 0
Viewed 2540

I'm trying to implement dask on a cluster that uses SLURM. The client is successfully created and scaled, however, at the line

with joblib.parallel_backend('dask'):

the operation gets the worker timeout error and I get the following error from the slurm jobs:

/usr/bin/python3: Error while finding module specification for 'distributed.cli.dask_worker' (ModuleNotFoundError: No module named 'distributed')

I have checked to make sure that distributed has been installed on the cluster's nodes and I am able to import it into python without any issues. Does anyone know why distributed is causing issues?

joblib dask python
2 Answers

Making a fresh conda environment with dask[complete] seems to have worked.

user_132248460

You do not have the distributed library installed. This commonly happens for a few reasons:

  1. You did pip install dask rather than pip install dask[complete] or conda install dask

  2. You installed into a different python executable running on your machine

    I see that you're using /usr/bin/python3. To be extra safe try /usr/bin/python3 -m pip dask[complete]

  3. Your worker machines don't share the same file system as your login nodes

user_6166160
Related