I have a rather large dataset across different files that I read in using dask, followed by a machine learning task for which I want to use dask as parallel backend. I've noticed that reading in the files runs much faster using a Client with a higher number of workers instead of one worker with many threads. However, their individual share of memory is then too small to handle the ML task. I would therefore like to change the number of my workers to 1, with the maximum possible number of threads assigned to that new unique worker. Is there a way to do that without completely kiling and restarting my client?
I looked into the docs but couldn't find anything of use. Also happy about a hint where to look for this kind of info next time, if not there.
This is an example of what my current code looks like:
from dask.distributed import Client
import dask.dataframe as dd
from sklearn.linear_model import LogisticRegression
from joblib import parallel_backend
client = Client(n_workers=4, threads_per_worker=2)
df = dd.read_hdf(path_to_file_dir, '/data')
feats = df['feats'].compute()
labels = df['labels'].compute()
dummy = LogisticRegression()
with parallel_backend('dask'):
dummy.fit(feats, labels) # FAILS bc of too high memory consumption