I am trying to pass a big pandas dataframe as a function argument to a worker of dask distributed. What I tried (X is my dataframe):
1 Passing the data directly to function:
def test(X):
return X
f=client.submit(test, X)
f.result()
2 Saving a dataframe in an initialization function.
def worker_init(r_X):
global X
X=r_X
client.run(worker_init,X,y)
3 Scattering the dataframe across all nodes and then using it via futures
def test(X):
return X
f_X = client.scatter(X, broadcast=True)
f = client.submit(test,f_X)
f.result()
None of the variants work for my case. Variant 1 and 2 work almost the same. The dask-scheduler increases the memory with every task and never frees it until it gets out of memory and the task fails.
Variant 3 does not work because instead of passing a pandas dataframe I get some garbage.
How can I send the dataframe to a worker and do not have a MemoryError on the scheduler?
The full code of variant 3 that is supposed to be memory efficient, but does not even pass a dataframe:
import pandas as pd
import numpy as np
from distributed import Client
client = Client('localhost:8786')
X = np.random.rand(10000,100)
X=pd.DataFrame(X)
f_X = client.scatter(X, broadcast=True)
def test(X):
return X
f = client.submit(test,f_X)
f.result()[:10]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]