I have dask distributed as part of a larger effort with workflow (prefect), and am seeing an error in my dask workers. The worker receives its work, but quickly raises this exception:
TypeError: can not serialize 'Variable' object
The full trace is:
distributed.protocol.core - CRITICAL - Failed to Serialize
Traceback (most recent call last):
File "/opt/conda/lib/python3.7/site-packages/distributed/protocol/core.py", line 25, in dumps
small_header, small_payload = dumps_msgpack(msg)
File "/opt/conda/lib/python3.7/site-packages/distributed/protocol/core.py", line 163, in dumps_msgpack
payload = msgpack.dumps(msg, use_bin_type=True)
File "/opt/conda/lib/python3.7/site-packages/msgpack/__init__.py", line 35, in packb
return Packer(**kwargs).pack(o)
File "msgpack/_packer.pyx", line 286, in msgpack._cmsgpack.Packer.pack
File "msgpack/_packer.pyx", line 292, in msgpack._cmsgpack.Packer.pack
File "msgpack/_packer.pyx", line 289, in msgpack._cmsgpack.Packer.pack
File "msgpack/_packer.pyx", line 225, in msgpack._cmsgpack.Packer._pack
File "msgpack/_packer.pyx", line 283, in msgpack._cmsgpack.Packer._pack
TypeError: can not serialize 'Variable' object
distributed.comm.utils - INFO - Unserializable Message: {'op': 'variable_get', 'timeout': 0, 'name': <distributed.variable.Variable object at 0x7fec9eafaef0>, 'client': 'Client-worker-365168fe-ccf9-11ea-800b-0242ac110004', 'reply': True}
distributed.comm.utils - ERROR - can not serialize 'Variable' object
Traceback (most recent call last):
File "/opt/conda/lib/python3.7/site-packages/distributed/comm/utils.py", line 35, in _to_frames
msg, serializers=serializers, on_error=on_error, context=context
File "/opt/conda/lib/python3.7/site-packages/distributed/protocol/core.py", line 25, in dumps
small_header, small_payload = dumps_msgpack(msg)
File "/opt/conda/lib/python3.7/site-packages/distributed/protocol/core.py", line 163, in dumps_msgpack
payload = msgpack.dumps(msg, use_bin_type=True)
File "/opt/conda/lib/python3.7/site-packages/msgpack/__init__.py", line 35, in packb
return Packer(**kwargs).pack(o)
File "msgpack/_packer.pyx", line 286, in msgpack._cmsgpack.Packer.pack
File "msgpack/_packer.pyx", line 292, in msgpack._cmsgpack.Packer.pack
File "msgpack/_packer.pyx", line 289, in msgpack._cmsgpack.Packer.pack
File "msgpack/_packer.pyx", line 225, in msgpack._cmsgpack.Packer._pack
File "msgpack/_packer.pyx", line 283, in msgpack._cmsgpack.Packer._pack
TypeError: can not serialize 'Variable' object
I have brought this down to a very small test case, which I think is reasonably representative. It raises the same issue:
from distributed.protocol.core import dumps_msgpack
from distributed.variable import Variable
from distributed.client import Client
client = Client(address='10.72.112.29:38786')
var = Variable(name='test', client=client)
msg = {'op': 'variable_get', 'timeout': 0, 'name': var, 'client': client.id, 'reply': True}
d = dumps_msgpack(msg)
This sample produces the error message, but does also show some version mismatches (which I don't think are the issue):
/Users/gdesmarais/git/rightsize/rightsize-venv/lib/python3.7/site-packages/distributed/client.py:1136: VersionMismatchWarning: Mismatched versions found
+---------+---------------+---------------+---------------+
| Package | client | scheduler | workers |
+---------+---------------+---------------+---------------+
| numpy | 1.19.0 | 1.18.1 | 1.18.1 |
| python | 3.7.8.final.0 | 3.7.0.final.0 | 3.7.0.final.0 |
+---------+---------------+---------------+---------------+
warnings.warn(version_module.VersionMismatchWarning(msg[0]["warning"]))
Traceback (most recent call last):
File "<input>", line 7, in <module>
File "/Users/gdesmarais/git/rightsize/rightsize-venv/lib/python3.7/site-packages/distributed/protocol/core.py", line 163, in dumps_msgpack
payload = msgpack.dumps(msg, use_bin_type=True)
File "/Users/gdesmarais/git/rightsize/rightsize-venv/lib/python3.7/site-packages/msgpack/__init__.py", line 35, in packb
return Packer(**kwargs).pack(o)
File "msgpack/_packer.pyx", line 286, in msgpack._cmsgpack.Packer.pack
File "msgpack/_packer.pyx", line 292, in msgpack._cmsgpack.Packer.pack
File "msgpack/_packer.pyx", line 289, in msgpack._cmsgpack.Packer.pack
File "msgpack/_packer.pyx", line 225, in msgpack._cmsgpack.Packer._pack
File "msgpack/_packer.pyx", line 283, in msgpack._cmsgpack.Packer._pack
TypeError: can not serialize 'Variable' object
Am I missing a package or something? I can provide my installation scripts, etc., but they are a bit involved - the client is a local python env, the scheduler and workers are dockers.