Dask groupby existing index column(s)

Hanan Shteingart 0
Viewed 32

I have a data frame index by column A. How do I group by using that column with an additional one, e.g. [A, B] without resetting the index? e.g. what I do today is df.reset_index().groupby([A,B]).apply(f) which I think might be not optimal because I am losing the prior indexing. In my parquet file, A and B are already marked as index. How can I use the existing indexing to make group by A and B more efficient?

Moreover (might be unrelated) I see the CPU usage as very low as if there is a shuffling or io bottleneck.

dask
1 Answers

You can always just do a map_partition job as set-index you guarantee each partition has all the values corresponding to that index.

def groupby_apply_func(df):
    return df.reset_index().groupby([A,B]).apply(f)

#### Assumes that ddf has index A
ddf.map_partitions(groupby_apply_func)
Vibhu Jawa0
Related