Merge multiple RDD generated in loop

Viewed 4473

I am calling a function in scala which gives an RDD[(Long,Long,Double)] as its output.

def helperfunction(): RDD[(Long, Long, Double)]

I call this function in loop in another part of the code and I want to merge all the generated RDDs. The loop calling the function looks something like this

for (i <- 1 to n){
    val tOp = helperfunction()
    // merge the generated tOp
}

What I want to do is something similar to what StringBuilder would do for you in Java when you wanted to merge the strings. I have looked at techniques of merging RDDs, which mostly point to using union function like this

RDD1.union(RDD2)

But this requires both RDDs to be generated before taking their union. I though of initializing a var RDD1 to accumulate the results outside the for loop but I am not sure how can I initialize a blank RDD of type [(Long,Long,Double)]. Also I am starting out with spark, so I am not even sure if this is the most elegant method to solve this problem.

2 Answers
Related