Understanding persistence in Spark

user_3613204 0
Viewed 805

I am fairly new to Spark, and was playing around it a bit and have a few doubts regarding it:-

Here is the code that I am trying out:- 

import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.api.java.function.Function;
import org.apache.spark.api.java.function.Function2;

/**
 * Computes an approximation to pi
 * Usage: JavaSparkPi [slices]
 */
public final class JavaSparkPi {

    public static void main(String[] args) throws Exception {

        SparkConf sparkConf = new SparkConf().setAppName("JavaSparkPi").setMaster("local[4]");


        JavaSparkContext jsc = new JavaSparkContext(sparkConf);

        JavaRDD<String> data = jsc.textFile("/Users/sanchay/Desktop/2017-08-08");


        while(true) {

            Thread.sleep(1000);

                    int count = data.map(s -> s.length()).reduce((a, b) -> (a + b));
            System.out.println("count == " + count);

        }

      //  jsc.stop();

    }
}

1) Since I am currently running in the local mode, upon each iteration the time taken is getting reduced (file size is around 150 Mb). This would be because each time the RDD is getting cached. I wanted to know when I would be running this on a cluster of remote nodes, will I experience this same behaviour, because it is not necessary that the same node will get the same partition it computed before.

2) Also,I tried to un-persist the 'data' variable above, however, I noted no difference in the time taken, as in the time reduced from the 1st iteration to the 2nd, implying something somewhere was cached . So, is it still somehow getting cached, or am I missing something obvious here.

rdd apache-spark caching
2 Answers

I think it become faster on later iteration because of OS caching mechanism. Briefly, when you read file, the filesystem module will put its content in some memory location to make faster access next time. You don't cache RDD explicitly so it's not in memory, each run it first read content file again then do map, reduce ops..., but the file content is in somewhere that's fast to access so the later run will get speedup.

user_47068430

NO it won't persist() by default we have to mention which RDD to be persisted without the external method spark won't persist anything

you can persist an RDD like

import org.apache.spark.storage.StorageLevel 

val source = s.map(rec => (rec.split(",")(0), rec)).persist(StorageLevel.MEMORY_ONLY)
 //DISK_ONLY,MEMORY_AND_DISK,MEMORY_AND_DISK_SER etc

Default caching will take place in memory for that we can give that as val rdd1 = a.cache()

user_54395460
Related