Best practice for groupby on Parquet file

Viewed 1605

We have a 1.5BM records spread out in several csv files. We need to groupby on several columns in order to generate a count aggregate.
Our current strategy is to:

  1. Load them into a dataframe (using Dask or pyspark)
  2. Aggregate columns in order to generate 2 columns as key:value (we are not sure if this is worthwhile)
  3. Save file as Parquet
  4. Read the Parquet file (Dask or pyspark) and run a groupby on the index of the dataframe.

What is the best practice for an efficient groupby on a Parquet file?
How beneficial is it to perform the groupby on the index rather then on a column (or a group of columns)?
We understand that there is a partition that can assist - but in our case we need to groupby on the entire dataset - so we don't think it is relevant.

1 Answers
Related