DAG Scheduler vs. Catalyst for Spark

user_6933993 0
Viewed 525

I would like to confirm some aspects that from reading all blogs and Databricks sources and the experts Holden, Warren et al, seem still poorly explained. I also note some unanswered questions out there in the net regarding this topic.

The following then:

  • My understanding is that for RDD's we have the DAG Scheduler that creates the Stages in a simple manner.

  • My understanding based on reading elsewhere to-date is that for DF's and DS's that we:

    • Use Catalyst instead of the DAG Scheduler. But I am not convinced.
    • Have Tungsten.

As DAG applies to DF's and DS's as well (obviously), I am left with 1 question - just to be sure:

  • Is it Catalyst that creates the Stages as well?
    • This may seem a silly question, but I noted a question on Disable Spark Catalyst Optimizer here on SO. That would imply no.
    • In addition, as the Spark paradigm is Stage based (shuffle boundaries), it seems to me that deciding Stages is not a Catalyst thing.

Therefore my conclusion is that the DAG Scheduler is still used for Stages with DF's and DS's, but I am looking for confirmation.

Moreover, this picture implies that there is still a DAG Scheduler.

enter image description here

This picture from the Databricks 2019 summit seems in contrast to the statement found on a blog:

An important element helping Dataset to perform better is Catalyst Optimizer (CO), an internal query optimizer. It "translates" transformations used to build the Dataset to physical plan of execution. Thus, it's similar to DAG scheduler used to create physical plan of execution of RDD. ...

I see many unanswered questions on SO on the DAGs with DF's etc. and a lot of stuff is out-of-date as it RDD related. So, as a consequence I asked a round a few of my connection with Spark knowledge on this and noted they were remiss in providing a suitable answer.

apache-spark
2 Answers

Let me try to clear these terminologies for you.

Spark Scheduler is responsible for scheduling tasks for execution. It manages where the jobs will be scheduled, will they be scheduled in parallel, etc. Spark Scheduler works together with Block Manager and Cluster Backend to efficiently utilize cluster resources for high performance of various workloads. DAGScheduler is a part of this.

Catalyst is the optimizer component of Spark. It performs query optimizations and creates multiple execution plans out of which the most optimized one is selected for execution which is in terms of RDDs.

Tungsten is the umbrella project that was focused on improving the CPU and memory utilization of Spark applications.

DAGScheduler is responsible for generation of stages and their scheduling. It breaks each RDD graph at shuffle boundaries based on whether they are "narrow" dependencies or have shuffle dependencies. It also determines where each task should be executed based on current cache status.

user_84916420

Well, I searched a bit more and found a 'definitive' source from the Spark Summit 2019 slide from David Vrba. It is about Spark SQL and shows the DAG Scheduler. So we can conclude that Catalyst does not decide anything on Stages. Or is this wrong and is the above answer correct and the below statement correct? The picture implies differently is my take, so no.

The statement I read elsewhere on Catalyst: ...

An important element helping Dataset to perform better is Catalyst Optimizer (CO), an internal query optimizer. It "translates" transformations used to build the Dataset to physical plan of execution. Thus, it's similar to DAG scheduler used to create physical plan of execution of RDD.

...

was a little misleading. That said, checking to be sure, elsewhere revealed no clear statements until this.

enter image description here

user_69339930
Related