I am some relatively large task on DASK on a machine with (120 GB RAM, 280GB Swap space)
Using this setting, every thing is a bit slow but it works:
client = Client(memory_limit='90GB',n_workers=1,threads_per_worker=14)
I think there is some IO issue when observing the computation. However, if I use
client = Client()
In the end, there are memory error causing everything to break. But I figured it didn't try to use the swap space at all... Is there a way to configure dask to use swap space to avoid memory error?
Any other advise? For example, is it ok if i do?
client = Client(memory_limit='90GB',n_workers=4,threads_per_worker=14)
Since theoretically, I have 120GB RAM+ 280 GB swap > 4*90GB.... Am I making any sense here?