I wrote a simple query in spark: spark.range(1000).filter(id => id % 2 == 0).count(). The DAG for it is this:
Could somebody please explain to me why we start with a ShuffledRowRDD right away? Filter definitely doesn't cause a shuffle, MapParitionsRDD I understand, we count the number of elements for each partition, then we should shuffle at the end by bringing the result from each partition to the driver.
So, why is there no shuffle at the end, but one in the very beginning?
And why is there 2 MapPartitionsRDDs, what does that mean?