How do you select the 'maximum' struct from each group

user_11313583 0
Viewed 25

I have a dataFrame that contains an id column and a struct of two values order_value

example_input = spark.createDataFrame([(1, (1,2)), (1, (2,1)), (2, (1,2))], ["id", "order_value"])

I would like to keep one record from each id, that is the maximum of the order_value column. Specifically the maximum of the order (first part of order_value) with ties broken by the the maximum of the value (second part of order_value)

How can this be done?

example_input.groupby('id').max() doesn't seem to work as it complains that order_value is not numeric.

my desired output is given by:

example_output = spark.createDataFrame([(1, (2,1)), (2, (1,2))], ["id", "order_value"])
pyspark
1 Answers

Try with array_max function in spark.

Example:

#groupby on id then collect_list to create an array to find max in the array

example_input.groupBy("id").agg(array_max(collect_list(col("order_value"))).alias("order_value")).\
show(10,False)
user_76326950
Related