We have a 1.5BM records spread out in several csv files. We need to groupby on several columns in order to generate a count aggregate.
Our current strategy is to:
- Load them into a dataframe (using
Daskorpyspark) - Aggregate columns in order to generate 2 columns as key:value (we are not sure if this is worthwhile)
- Save file as
Parquet - Read the
Parquetfile (Daskorpyspark) and run a groupby on the index of the dataframe.
What is the best practice for an efficient groupby on a Parquet file?
How beneficial is it to perform the groupby on the index rather then on a column (or a group of columns)?
We understand that there is a partition that can assist - but in our case we need to groupby on the entire dataset - so we don't think it is relevant.