Stage in Persist has only 1 task running causing spill to disk

We are caching a data set after running the given SQL on it. The flow of the code is

1. Load the dataset into a spark data frame
2. Run SQL
3. Repartition the result dataset using dataset.repartition()
4. and cache the dataset with option MEMORY_AND_DISK.

The SQL to run is :

`select *,CASE WHEN event_type in ('value1','value2') THEN EXISTING_COLUMN_1 ELSE '0' END AS NEW_COLUMN_NAME from events`

The result of the SQL is another data frame- result, that we repartition using result.repartion(300) and cache it by using persist function. The parameter spark.sql.shuffle.partitions is set to 2001.

Now, the problem is, at the time of caching, there are three stages: [![stages][1]][1]

• stage 1 shuffle-writes the data
• stage 2 shuffle-reads it. There is no spill to disk/memory in these stages
• stage 3, in which there is only 1 task running and causing spill to disk and memory. This has become the bottleneck in our application. [![stage3][2]][2] [![enter image description here][3]][3]

Why there is this stage with one task, which is spilling the data to disk/memory? Kindly help us in understanding the issue and where we are going wrong.

Edit 1: Code snippet for Function_SQLTransformation.scala with SQL.

val events=spark.read.format("orc").load("path")
val table=spark.sql("""select *,
                     CASE 
                     WHEN event_type in ('value1','value2') THEN 
                     EXISTING_COLUMN_1 ELSE '0'
                     END AS NEW_COLUMN_NAME from events""".stripMargin)
table = table.repartition(300)
table.createOrReplaceTempView("second_table_name")
table.persist(StorageLevel.MEMORY_AND_DISK)


  [1]: https://i.stack.imgur.com/RcAZt.png
  [2]: https://i.stack.imgur.com/l723E.png
  [3]: https://i.stack.imgur.com/XUE3G.png