Unable to set the nanny listening address for dask-worker

user_14044465 0
Viewed 230

When starting dask-worker, I need to explicitly set the nanny address published to the scheduler and I can't find how to do this. Does anyone know?

What I am trying to do: I am running dask-worker in an Ubuntu VM (WSL 2 / Windows 10) which assigns a virtual network address to the ethernet adapter (through Hyper-V). This means that any published TCP address from the worker to the scheduler needs to be the external host IP whilst any port binding within the VM has to be against the virtual network address.

Here is my command line:

dask-worker --dashboard --nthreads=1 --nprocs=1 --memory-limit=2GB --name=workstation1 --listen-address=tcp://172.24.10.244:42321
i --contact-address=tcp://192.168.2.1:42321 --nanny-port=43321 tcp://192.168.2.2:8786 &

The problem here is that dask-worker sends its contact address as 192.168.2.1 (the ip address of the physical network adapter) but it is also sending out its nanny address to the scheduler as 172.24.10.244 which is not a contactable address from the scheduler's perspective. I would have expected the published nanny ip addr to be lifted out of the contact-address parameter but that is not the case.

Any thoughts on how to fix this?

This is dask / distributed 2.22.0.

Thanks!

dask-distributed dask
2 Answers

You might consider using the --interface keyword instead to set the right network interface. This is a bit more general than listen/contact address.

user_6166160

One workaround for dask not correctly reporting its contact addresses is to use iptables DNAT rewriting to rewrite the incorrectly reported IP from dask to the correct IP on the scheduler. It is admittedly a huge hack. A better solution would be a patch to dask which solves this edge case.

user_1534330
Related