I wanted to use rechunker to rechunk some Zarr files.
The Docs state:
If we create a distributed cluster, then rechunker will use that when it executes.
However I can see this only partly archived. The Dashboard shows that it is calculated within the cluster, but it uses only one (of the 2 available) workers and exceeds the memory limits (but I think this is known, e.g. here) I tried various option changing the scheduler etc. but nothing worked so far for me.
Do I miss something?
Here is a code, reproducing the problems:
with dd.LocalCluster(n_workers=2, memory_limit='14GiB') \
as cluster, dd.Client(cluster) as client:
dask.config.set({'temporary-directory': f'./temp_dask/'})
print('Dashboard Link Data Analysis (Dask): ', client.dashboard_link)
# creating toy data
zstore = zarr.DirectoryStore(f'../data/example_dask.zarr')
drdm = da.random.randint(0, 10, (60, 150_000),
chunks=(60, 4370), dtype='int32')
xar = xr.Dataset(
data_vars=dict(
my_array=(['x', 'y'], drdm)
),
coords=dict(
channeli=np.arange(60),
time=np.arange(150_000)
)
)
xar.to_zarr(store=zstore)
# want to rechunk
source = zarr.open(f'../data/example_dask.zarr', mode='r')
intermediate = f'../data/intermediate.zarr'
target = f'../data/result_dask_rechunked.zarr'
rechunked = rechunk(source,
target_chunks=dict(my_array=(60, 150_000)),
target_store=target,
max_mem='5GiB', temp_store=intermediate, # memory adaption
executor='dask')
rechunked.execute() # tried various scheduler=X things
os.system(f'rm -rf ./temp_dask/')
os.system(f'rm -rf ../data/intermediate.zarr') # removing intermediate
