Understanding Dask's Task Stream

user_2333690 0
Viewed 745

I'm running dask locally using the distributed scheduler on my machine with 8 cores. On initialization I see:

enter image description here

Which looks correct, but I'm confused by the task stream in the diagnostics (shown below): enter image description here

I was expecting 8 rows corresponding to the 8 workers/cores, is that incorrect?

Thanks

AJ

I've added the code I'm running:

import dask.dataframe as dd
from dask.distributed import Client, progress
client = Client()
progress(client)

# load datasets
trd = (dd.read_csv('trade_201811*.csv', compression='gzip',  
                    blocksize=None, dtype={'Notional': 'float64'})
         .assign(timestamp=lambda x: dd.to_datetime(x.timestamp.str.replace('D', 'T')))
         .set_index('timestamp', sorted=True))
dask python-3.x
1 Answers

Each line corresponds to a single thread. Some more sophisticated Dask operations will start up additional threads, this happens particularly when tasks launch other tasks, which is common especially in machine learning workloads.

My guess is that you're using one of the following approaches:

  • dask.distributed.get_client or dask.distributed.worker_client
  • Scikit-Learn's Joblib
  • Dask-ML

If so, the behavior that you're seeing is normal. The task stream plot will look a little odd, yes, but hopefully it is still interpretable.

user_6166160
Related