I am trying to write some performance-mindful code in Spark and wondering whether I should write an Aggregator or a User-defined Aggregate Function (UDAF) for my rollup operations on a Dataframe.
I have not been able to find any data anywhere on how fast each of these methods are and which you should be using for spark 2.0+.
You should write an Aggregator rather than an UserDefinedAggregateFunction as UserDefinedAggregateFunction performs inefficient serialization/deserialization tasks for each row. Rewriting an UserDefinedAggregateFunction to an Aggregator can improve performance from 25%-30% to 100x, as stated in pull request replacing UserDefinedAggregateFunction with Aggregator
Due to those performance issues, UserDefinedAggregateFunction class has been deprecated in Spark 3.0
