I'm getting error
Dataset transformations and actions can only be invoked by the driver, not inside of other transformations; for example, rdd1.map(x => rdd2.values.count() * x, see SPARK-28702
Here is my code
do {
val size = accumulatorList.value.size()
val current = accumulatorList.value.get(size -1)
joinoutput = current.join(broadvast.value, current.col("A") === broadvast.value.col("A"))
.map {x =>
val gig = x._1.getAs("Name") + "|" + x._1.getAs("PIP")
MyData(
x._2.getAs("Name"),
x._2.getAs("Place"),
x._2.getAs("Phone"),
x._2.getAs("Doc"),
gig
)}
accumulatorList.add(joinoutput)
joincount = joinoutput.count()
}
while (joincount >0 )
This error is coming while I'm joining in do block. Its running fine in my local/cluster but failing while I'm building my code with test cases. Even my test case is running fine in local, but the problem is only coming while building it.
Can anyone suggest please what's wrong happening and what can be done to fix this ?
Thanks