Too many Accumulators in Spark Job

Viewed 1167

My spark app has 40 Accumulators

object MySparkApp { 
  def main(args: Array[String]): Unit = {
    // initialize SparkContext

    val acc1 = sc.accumulator(0)
    val acc2 = sc.accumulator(0)
    .
    .
    val acc40 = sc.accumulator(0)

    val logRdd = sc.textFile("input/path").map(x => parser.parse(x))
    logRdd.forEach(x => incrementCounter(x, acc1, acc2,..... acc40))
  }
}

This code is very ugly what would be a good way to wrap these accumulators in something like an object and make the code more readable.

1 Answers
Related