The source table has many columns namely
The table is partitioned on year,year_month and tran_datetime
table size : 590000000
each partition has roughly 50000000 rows
the data is being written to S3.
I am trying add index to the above table with below code , but i am getting the bellow error. I am not sure what is causing it.
Spark 2.3
Total Memory:1.2TB and cores: 35
Driver.cores 5
Driver.memory 37G
executor.cores 5
executor.memory 37G
driver.memoryOverhead 5120
executor.memoryOverhead 5120
Code is in the image below.
Can some one please explain me this error, why is this coming and what dose it mean by no space left.