We are using Azure Databricks cluster (E64s_v3 (432GB RAM, 64 Cores); Worker's = 12) to run the scala code. We are trying to parallelize the writing of data by worker node so we can have all the cores working for it. Using "new scala.concurrent.forkjoin.ForkJoinPool(8)" to parallelize the writing of data. We have 2B of records and all of them can be segregated by 50 Ids, our goal is to have 1 worker working on each Id of data with all the cores running so we can have 12 paralleling writings happening with all the cores working. Below is the piece of code that we have used
sTpids.tasksupport = new ForkJoinTaskSupport( new scala.concurrent.forkjoin.ForkJoinPool(8) ) sTpids.foreach(p => { trnMap.write.mode("overwrite").parquet(partitionPath) }
From our observation, the processes is not parallelizing instead all the workers are working on writing 1 Id at a given time and only 1 core per worker is being used. Can you some one help me understand why this is happening ?