I'm trying to parallelize a for-loop in Dask. What it actually does is, it's taking the unique values of the 'desc' column and extract all rows with this value. Then it will save this subframe in a parquet file, named like the 'desc' value.
def save_params(df):
for param in df['desc'].unique():
element = df[df['desc'] == param].drop('desc', axis=1).compute()
element['value'] = pd.to_numeric(element['value'])
parameter_path = target_folder + param + '.parquet'
if not path.exists(parameter_path):
element.to_parquet(parameter_path)
else:
previous = pd.read_parquet(parameter_path)
element = element.append(previous)
element.to_parquet(parameter_path)
df is a Dask dataframe with following structure:
| Datetime | desc | value |
|---|
The function is called by
save_params(df)
The Cluster obviously isn't running the loop in parallel (Why should it?!). What is the correct way to parallelize this?