Dask: Get scheduler address from Client

user_8233329 0
Viewed 110

I'm using dask-mpi to deploy on my HPC. I'd like to get hold of the scheduler address to use with another library. This is simple in the case of a LocalCluster (or any other explicit cluster definition) e.g.:

from dask.distributed import Client, LocalCluster

cluster = LocalCluster()
client = Client(cluster)

address = cluster.scheduler_address

But dask-mpi is initialised without a cluster object:

from dask.distributed import Client
from dask_mpi import initialize

initialize()
client = Client()

Trying to access the underlying client.cluster doesn't help as, in this case, it's set to None.

Is there some other way to get hold of the scheduler address from the client or elsewhere? Thanks!

mpi4py dask-distributed dask mpi python
2 Answers

It's not currently documented (at least I couldn't find it in the docs), but I've found out that a Client object has a scheduler property. The address, and other properties, can be obtained like:

from dask.distributed import Client
client = Client()
address = client.scheduler.address

I would suppose client.scheduler should return a Scheduler object, but I haven't done much more digging, yet.

user_82333290

It's not a direct solution and comes straight from docs, but one way is to specify where to write the scheduler file:

mpirun -np 4 dask-mpi --scheduler-file ~/dask-scheduler.json

The file will contain the connection info:

from dask.distributed import Client
client = Client(scheduler_file='~/dask-scheduler.json')
user_106935960
Related