Currently I am working on a Dash local cluster on a set of lz4 compressed Parquet files. Reading and writing compressed files works fine but when setting and index the shuffle algorithm will write a lot of uncompressed data to the disk (data is larger than my memory so I use out-of-memory shuffling on the disk).
Shuffling in Dask is done with the partd project which itselfs supports compression with snappy or lz4. However, I am not able to activate compression for the local workers and shuffle files. Is there any way using enviroment variables or dask.settings?
Many thanks
import dask
import dask.dataframe as dd
from dask.distributed import Client
# setup local cluster
client = Client(n_workers=2, threads_per_worker=4, processes=False, memory_limit='16GB')
# load, set index, save
df = dd.read_parquet('Data/Parquet', engine='fastparquet') # <-- is compressed
df2 = df.set_index(use_columns, shuffle='disk') # <-- generates a lot of uncompressed data on the disk
df2.to_parquet('Data/ParquetSorted', engine='fastparquet', compression="lz4") # <-- again compressed
Edit: feature has been implemented in the meantime