Running several spark jobs concurrently from driver

Viewed 1957

Imagine that we have 3 customers and we want do some same work for each of them in parallel.

def doSparkJob(customerId: String) = {
  spark
    .read.json(s"$customerId/file.json")
    .map(...)
    .reduceByKey(...)
    .write
    .partitionBy("id")
    .parquet("output/")
}

We do it concurrently like this (from spark driver):

val jobs: Future[(Unit, Unit, Unit)] = for {
  f1 <- Future { doSparkJob("customer1") }
  f2 <- Future { doSparkJob("customer1") }
  f3 <- Future { doSparkJob("customer1") }
} yield (f1, f2, f3)

Await.ready(jobs, 5.hours)

Do I understand correctly that this is bad approach? Many spark job will push out context of each other from executors and there will be many spilling data to disc appears. How spark will be manage execute task from parallel jobs? How shuffle appears when we have 3 concurrent job from one driver and only 3 executors with one core.

I guess, a good approach should looks like this: We read all data together for all customers groupByKey by customer and do what we want to do.

1 Answers
Related