dask.delayed KeyError with distributed scheduler

user_12935581 0
Viewed 669

I have a function interpolate_to_particles written in c and wrapped with ctypes. I want to use dask.delayed to make a series of calls to this function.

The code runs successfully without dask

# Interpolate w/o dask
result = interpolate_to_particles(arg1, arg2, arg3)

and with the distributed schedular in single-threaded mode

# Interpolate w/ dask
from dask.distributed import Client
client = Client()
result = dask.delayed(interpolate_to_particles)(arg1, arg2, arg3)
result_c = result.compute(scheduler='single-threaded')

but if I instead call 

result_c = result.compute()

I get the following KeyError:

> Traceback (most recent call last):   File
> "/path/to/lib/python3.6/site-packages/distributed/worker.py",
> line 3287, in dumps_function
>     result = cache_dumps[func]   File "/path/to/lib/python3.6/site-packages/distributed/utils.py",
> line 1518, in __getitem__
>     value = super().__getitem__(key)   File "/path/to/lib/python3.6/collections/__init__.py",
> line 991, in __getitem__
>     raise KeyError(key) KeyError: <function interpolate_to_particles at 0x1228ce510>

The worker logs accessed from the dask dashboard do not provide any information. Actually, I do not see any information that the workers have done anything besides starting up.

Any ideas on what could be occurring, or suggested tools that I can use to further debug? Thanks!

dask-delayed dask-distributed keyerror dask ctypes
1 Answers

Given your comments it sounds like your function does not serialize well. To test this, you might try pickling the function in one process, and try unpickling it in another.

>>> import pickle
>>> print(pickle.dumps(interpolate_to_particles))
b'some bytes printed out here'

And then in another process

>>> import pickle
>>> interpolate_to_particles = pickle.loads(b'the same bytes you had before')

If this doesn't work then you'll know that that's your problem. I would encourage you to look up "how to make sure that ctypes functions are serializable" or something similar, or ask another question with that smaller scope here on Stack Overflow.

user_6166160
Related