rechunker is not using dask environment?

Viewed 19

I wanted to use rechunker to rechunk some Zarr files.

The Docs state:

If we create a distributed cluster, then rechunker will use that when it executes.

However I can see this only partly archived. The Dashboard shows that it is calculated within the cluster, but it uses only one (of the 2 available) workers and exceeds the memory limits (but I think this is known, e.g. here) I tried various option changing the scheduler etc. but nothing worked so far for me.

enter image description here

Do I miss something?

Here is a code, reproducing the problems:

with dd.LocalCluster(n_workers=2, memory_limit='14GiB') \
            as cluster, dd.Client(cluster) as client:

    dask.config.set({'temporary-directory': f'./temp_dask/'})
    print('Dashboard Link Data Analysis (Dask): ', client.dashboard_link)

    # creating toy data
    zstore = zarr.DirectoryStore(f'../data/example_dask.zarr')

    drdm = da.random.randint(0, 10, (60, 150_000),
                             chunks=(60, 4370), dtype='int32')

    xar = xr.Dataset(
        data_vars=dict(
            my_array=(['x', 'y'], drdm)
        ),
        coords=dict(
            channeli=np.arange(60),
            time=np.arange(150_000)
        )
    )

    xar.to_zarr(store=zstore)


    # want to rechunk
    source = zarr.open(f'../data/example_dask.zarr', mode='r')
    intermediate = f'../data/intermediate.zarr'
    target = f'../data/result_dask_rechunked.zarr'

    rechunked = rechunk(source,
                        target_chunks=dict(my_array=(60, 150_000)),
                        target_store=target,
                        max_mem='5GiB', temp_store=intermediate,  # memory adaption
                        executor='dask')
    rechunked.execute()  # tried various scheduler=X things

    os.system(f'rm -rf ./temp_dask/')
    os.system(f'rm -rf ../data/intermediate.zarr')  # removing intermediate
0 Answers
Related