How to pass a pandas dataframe to dask distributed workers?

Viewed 1931

I am trying to pass a big pandas dataframe as a function argument to a worker of dask distributed. What I tried (X is my dataframe):

1 Passing the data directly to function:

def test(X):
    return X
f=client.submit(test, X)
f.result()

2 Saving a dataframe in an initialization function.

def worker_init(r_X):
    global X
    X=r_X
client.run(worker_init,X,y)

3 Scattering the dataframe across all nodes and then using it via futures

def test(X):
    return X
f_X = client.scatter(X, broadcast=True)
f = client.submit(test,f_X)
f.result()

None of the variants work for my case. Variant 1 and 2 work almost the same. The dask-scheduler increases the memory with every task and never frees it until it gets out of memory and the task fails.

Variant 3 does not work because instead of passing a pandas dataframe I get some garbage.

How can I send the dataframe to a worker and do not have a MemoryError on the scheduler?

The full code of variant 3 that is supposed to be memory efficient, but does not even pass a dataframe:

import pandas as pd
import numpy as np
from distributed import Client
client = Client('localhost:8786')
X = np.random.rand(10000,100)
X=pd.DataFrame(X)
f_X = client.scatter(X, broadcast=True)
def test(X):
    return X
f = client.submit(test,f_X)
f.result()[:10]

[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
1 Answers
Related