Dask scatter with broadcast=True extremely slow

user_19064142 0
Viewed 127

I have created a single (remote) scheduler and ten worker on different machines on the same network and try to distribute a dataframe from a client. My problem is that it takes 30min to do the scatter.

from dask.distributed import Client
df = pd.DataFrame({ i : range(10) for i in range(10)})
client = Client(scheduler_addr)
future = client.scatter(df, broadcast=True)

This code works but it is too slow to be usable - with broadcast=False it works reasonably fast. I have created Scheduler and Worker both with default arguments. How should it be done instead?

my dask.distributed version is 2022.01.0

dask-distributed dask pandas python
1 Answers

Scatter with broadcasting should be very fast for small objects, but will be slower on large objects.

One way to avoid sending large objects across the network is to store them at a common location and instruct workers to load these objects directly:

df = pd.DataFrame({ i : range(10) for i in range(10)})
df.to_parquet('my_file.parquet')

def run_batch(n):
    df = pd.read_parquet('my_file.parquet')
    ...

client = Client()
futures = client.map(run_something, range(10))

There is also a hack for this use-case, once_per_worker, see this blog post.

user_106935960
Related