Is it possible to have default distributed Client per thread?

user_5372214 0
Viewed 45

I have 2 independent remote clusters and running few parallel tasks in different threads that contain Dask DataFrame manipulations. I don't want explicitly use client in each thread - client.compute(df). I want to use default - df.compute().

Is it possible to set default client for each thread?

dask-dataframe dask pandas distributed python
1 Answers

This might not be the solution you are looking for, but one option is to wrap relevant computations in a context manager:

from distributed import Client, LocalCluster
from dask.datasets import timeseries

cluster1 = LocalCluster(n_workers=1, dashboard_address=':0')
cluster2 = LocalCluster(n_workers=1, dashboard_address=':0')

with Client(cluster1) as client:
    print(timeseries().head())

with Client(cluster2) as client:
    print(timeseries().head())
user_106935960
Related