Why dataset.count() is faster than rdd.count()?

Viewed 8504

I created a Spark Dataset[Long]:

scala> val ds = spark.range(100000000)
ds: org.apache.spark.sql.Dataset[Long] = [id: bigint]

When I ran ds.count it gave me result in 0.2s (on a 4 Core 8GB machine). Also, the DAG it created is as follows:

enter image description here

But, when I ran ds.rdd.count it gave me result in 4s (same machine). But the DAG it created is as follows:

enter image description here

So, my doubts are:

  1. Why ds.rdd.count is creating only one stage whereas ds.count is creating 2 stages ?
  2. Also, when ds.rdd.count is having only one stage then why it is slower than ds.count which has 2 stages ?
1 Answers
Related