Spark - Group by multiple columns VS Group by an id created from the same multiple columns

Viewed 97

Is it a good idea to create an id from multiple columns and then make a group by ? Maybe grouping data by one column (the id) is more efficient?

Example:

my_dataframe = my_dataframe \
  .withColumn("my_key", F.concat(F.col("column1"),
                                 F.col("column2"),
                                 F.col("column3"))) \
  .groupBy(F.col("my_key")) \
  .agg(...)
0 Answers
Related