Dask: Multiprocessing works on Mac but not on Windows

user_12208262 0
Viewed 1222

My dask multiprocessing code to apply two different lambda apply functions works perfectly on Mac but cannot run on a Windows 7 server. I can run the code with threading on Windows 7 but it's very slow. When I try scheduler processes on the Windows 7 server, CPU usage on all cores hit 100% and the server freezes.

I can run the code below in Windows using scheduler = 'threads' or by specifying num_workers but it doesn't seem to engage all cores. With num_workers or threading on the Mac, it is much slower than scheduler='multiprocessing'.

Is there some additional code needed for Windows not to overload the cores/workers? 

import pandas as pd
import dask.dataframe as ddf
import multiprocessing

daskdf = ddf.from_pandas(mypandasdataframe, npartitions=4*multiprocessing.cpu_count())


def removecw(df):
    for word in mylist:
        df['A'] = df['A'].apply(lambda x: re.sub(r'\b{}$'.format(re.escape(word)), '', x))
    return df


daskdf = daskdf.map_partitions(removecw, meta = daskdf)
daskdf = daskdf.compute(scheduler='processes')

daskdf.to_csv('outputfilename')

On my Mac I can run the code with the expected result (removing common words from the end of the strings in column A). When I try scheduler processes on the Windows 7 server, CPU usage on all cores hit 100% and the server freezes.

dask scheduler multiprocessing windows python
1 Answers

Windows handles multiprocessing a bit differently than Mac and Linux.

My guess is that your problem is due to trying to pass a function that is difficult to serialize. Maybe it implicitly depends on variables that are not passed in or maybe it is defined dynamically. Here are some things that you might try, but no guarantees:

  1. Define your function in a separate module
  2. Make sure that your function doesn't implicitly depend on any global variables. If you need some additional variable, like mylist then pass it in explicitly (the apply method can take additional keyword arguments)
  3. Try the distributed scheduler on your local machine. https://docs.dask.org/en/latest/scheduling.html#dask-distributed-local
user_6166160
Related