Avoid lazy evaluation of code in spark without cache

Question

How can I avoid lazy evaluation in spark. I have a data frame which needs to be populated at once, since I need to filter the data on the basis of random number generated for each row of data frame, say if random number generated > 0.5, it will be filtered as dataA and if random number generated < 0.5 it will be filtered as dataB.

val randomNumberDF = df.withColumn("num", Math.random())
val dataA = randomNumberDF.filter(col("num") >= 0.5)
val dataB = randomNumberDF.filter(col("num") < 0.5)

Since spark is doing lazy eval, while filtering there is no reliable distribution of rows which are being filtered as dataA and dataB(sometimes same row is being present in both dataA and dataB)

How can I avoid this re-computation of "num" column, I have tried using "cache", which worked, but given my data size is going to be big, I am ruling out that solution. I have also tried using other actions on the randomNumberDF, like :

count
rdd.count
show
first

these didn't solve the problem.

Please suggest something different from cache/persist/writing data to HDFS and again reading it as solution.

References I have already checked :

user_5761558 · Answer

If all you're looking for is a way to ensure that the same values are in randomNumberDF.num, then you can generate random numbers with a seed (using org.apache.spark.sql.functions.rand()):

The below is using 112 as the seed value:

val randomNumberDF = df.withColumn("num", rand(112))
val dataA = randomNumberDF.filter(col("num") >= 0.5)
val dataB = randomNumberDF.filter(col("num") < 0.5)

That will ensure that the values in num are the same across the multiple evaluations of randomNumberDF.

user_1138523 · Answer

besides using org.apache.spark.sql.functions.rand with a given seed, you coud use eager-checkpointing:

df.checkpoint(true)

This will materialize the dataframe to disk

Avoid lazy evaluation of code in spark without cache

2 Answers