Spark number of tasks vs number of partitions

Viewed 116

I am running Spark on my local machine with i5 and quad core processor i.e. 4 core, 8 threads. I run a simple Spark job to understand the behaviour but got confused how number of partitions and number of tasks are different on Spark UI.

Below operations I did:

  • read CSV file in Spark dataframe with 2 partitions.
  • I have checked number of underlying partitions using df.rdd.getNumPartitions() which is giving 2.
  • Apply withColumn logic to add another column.
  • df1.withColumn("partitionId", spark_partition_id()).groupBy("partitionId").count().show() to get size of each partition.

Questions:

  1. With above operations spark creates 7 JobIds. Why 7? My understanding is, a job is created when action is called and I do not have 7 actions.

    enter image description here

  2. I have 2 partitions, so shouldn't there be 2 tasks running ? Why I see different number of tasks on different stages?

0 Answers
Related