Scala's Parallel Collections vs. Spark Futures vs. `SparkContext.parallelize()`

Viewed 19

Task

I would like to spark-submit a Spark application with multiple independent jobs. These jobs can be ran in parallel. Ideally, I'd like to not only use the multiple cores of my driver node, but all the resources available in the entire cluster.

I found 3 potential solutions online, but I'm confused as to which one is the more appropriate/idiomatic approach. Can someone explain to me which approach is best suited for my task?

1. Using Scala's native parallel collection

(1 to 100).toList.par
  .foreach(...)

2. Using Spark Futures

import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.duration.Duration
import scala.concurrent.{Await, Future}

val futureA = Future { ... }
val futureB = Future { ... }

Await.result(futureA, Duration.inf)
Await.result(futureB, Duration.inf)

3. Using sc.parallelize()

val rdd = sc.parallelize(Array(1,2,3,4,5,6))
rdd.foreach(...)
1 Answers
Related