Spark: aggregate versus map and reduce

user_1877098 0
Viewed 2464

I'm learning Spark and start understanding how Spark distributes the data and combines the results. I came to the conclusion that using the operation map followed by reduce has an advantage on using just the operation aggregate. This is (at least I believe so) because aggregate uses a sequential operation, which hurts parallelism, while map and reduce can benefit from full parallelism. So when having a choice, isn't it better to use map and reduce than aggregate ? Are there cases where aggregate is preferred ? Or maybe when aggregate can't be replaced by the combination map and reduce ?

As an example - I want to find the string with the max length:

val z = sc.parallelize(List("123","12","345","4567"))
// instead of this aggregate ....
z.aggregate(0)((x, y) => math.max(x, y.length), (x, y) => math.max(x, y))
// .... shouldn't I rather use this map - reduce combination ?
z.map(_.length).reduce((x, y) => math.max(x, y))
apache-spark mapreduce
3 Answers

A little example will can be better than long explanations.

Imagine you have a class Toto with an age field. You have many Toto and you desire to compute sum of ages of every Toto.

final case class Toto(val age: Int)

val rdd = sc.parallelize(0 until n).map(Toto(_))

// map/reduce style
val sum1 = rdd
             // O(n) operations to go througth every Toto's age
             .map(_.age)
             // another O(n) to access data then O(n) operations to sum the n values
             .reduce(_ + _)
// You get the result with 2 pass over your data plus O(n) additions

// aggregate style
val sum2 = rdd.aggregate(0)((agg, e) => agg + e.age, _ + _)
// With one pass over the data, and O(n) additions you obtain the same result

It's a bit more complicate if you take into account access and each operations.

Because aggregate still access then sum the age into the aggregate wich represent O(2.n) operations, O(n) access plus O(n) additions, plus negligeable merged operation between aggregates.

On the other side with map/reduce style, first the map represent O(n) access, then again O(n) access to data to reduce them with an overhead of O(n) addition operations for a total of O(3.n) operations.

Without forgetting the fact that Spark is lazy and all of your transformation will be leverage by a final action.

I presume that using aggregate will save some operations and then will improve application running time. But depending on what you're doing it could be more usefull to express successive map followed by a reduce for readability compare to an aggregate or combineByKey (generalization of aggregateByKey). So i will suppose that it depends on which goals you desire to reach depending the use case.

user_36084760

I believe I can partially answer my own question. I was wrongly assuming that, because a sequential operation is used, aggregate might be hurt in its parallelism. The data can still be parallelized and the sequential op will be executed on each chunk. This doesn't seem less performing than the map operation. So then the question that remains is: why would you use aggregate as opposed to the map-reduce combination ?

user_18770980

Aggregate operation allows to specify a combiner function (to reduce the amount of data sent through the shuffle), which is different to reducer, with map-reduce combination the same function is used to combine and reduce. I know used old Map Reduce terminology but conceptually all shared nothing shuffle based frameworks do this and if you google for mapreduce combiner you will find a lot of explanations of the concept.

user_14858800
Related