I am running Spark on my local machine with i5 and quad core processor i.e. 4 core, 8 threads. I run a simple Spark job to understand the behaviour but got confused how number of partitions and number of tasks are different on Spark UI.
Below operations I did:
- read CSV file in Spark dataframe with 2 partitions.
- I have checked number of underlying partitions using
df.rdd.getNumPartitions()which is giving 2. - Apply
withColumnlogic to add another column. df1.withColumn("partitionId", spark_partition_id()).groupBy("partitionId").count().show()to get size of each partition.
Questions:
