Spark, keeping state information through CollectionAccumulator[int]

user3023949 0
Viewed 28

Can we keep state information across workers through use of CollectionAccumulator[Int] ? RDD is huge

Below is sample code snippet

var valueColl:CollectionAccumulator[Int] = spark.sparkContext.collectionAccumulator("myValue")
rdd.foreachPartition(p => {
  val temp:java.util.List[Int] = valueColl.value
  if (!temp.contains(p.value)) {
   valueColl.add(p.value)
  }
}
azure-databricks apache-spark scala
1 Answers

Unfortunately this is not possible. You cannot access the value of an accumulator within an executor process.

From the docs:

Only the driver program can read the accumulator’s value, using its value method.

An accumulator is used to collect data from the executor processes. Each executor contains one or many instances of the accumulator. Each instance of the accumulator only sees the values that have been collected within its own process. Only when the different instances of the accumulator are sent to the driver process, they are reduced to a single final value and can then be used - on the driver.

userf5W80
Related