Is it a good idea to create an id from multiple columns and then make a group by ? Maybe grouping data by one column (the id) is more efficient?
Example:
my_dataframe = my_dataframe \
.withColumn("my_key", F.concat(F.col("column1"),
F.col("column2"),
F.col("column3"))) \
.groupBy(F.col("my_key")) \
.agg(...)