Parallelize for-loop with Dask

Viewed 65

I'm trying to parallelize a for-loop in Dask. What it actually does is, it's taking the unique values of the 'desc' column and extract all rows with this value. Then it will save this subframe in a parquet file, named like the 'desc' value.

def save_params(df):
    for param in df['desc'].unique():
        element = df[df['desc'] == param].drop('desc', axis=1).compute()
        element['value'] = pd.to_numeric(element['value'])
        parameter_path = target_folder + param + '.parquet'
        if not path.exists(parameter_path):
            element.to_parquet(parameter_path)
        else:
            previous = pd.read_parquet(parameter_path)
            element = element.append(previous)
            element.to_parquet(parameter_path)

df is a Dask dataframe with following structure:

Datetime desc value

The function is called by

save_params(df)

The Cluster obviously isn't running the loop in parallel (Why should it?!). What is the correct way to parallelize this?

0 Answers
Related