Is reading of a file is lazily evaluated in Apache spark?

user_14415712 0
Viewed 252

Consider the below code,

Val df1 = sparkSession.read().json("abc.json")

Val df2 = df1.where("point > 200").limit(10)

df2.count()

My first question is, when the first line is evaluated and my data is read into df1? does that happen when count action is called?

My second question is, does the data that is saved to df1 will be available in df1 till the execution of program ends and I can use it to perform any other operations as well?

Ex -   Val df3 = df1.where("point > 100")

       df3.count()
pyspark apache-spark-sql apache-spark
1 Answers

Spark has two things, operations and actions. All the oprations are lazy and doesn't get execution until you call actions (i.e. count, show, save)

In your case action is count. So once you execute df2.count() following things happens.

  1. spark reads the file and creates dataframe, df1
  2. it applies your logic and creates a new dataframe, df2
  3. finally shows your count.

Second Question No, Data is not saved to a dataframe like normal programming language. This is lazy evolution. So a action triggers all the steps.

In you're case when you call df3.count following things happens.

  1. spark will read file again and create dataframe, df1
  2. you're logic will be applied and new dataframe d3 will be created.
  3. and finally a count will be showed.

If you are planning to have couple of operation on single dataframe you can save that dataframe using cache function. Which will put the dataframe into memory and so operation before that won't be performed again.

so what you should do is.

val df1 = sparkSession.read().json("abc.json")
df1.cache() // all the data are saved to memory (or disk )

val df2 = df1.where("point > 200").limit(10)
df2.count() // However cache happens only after first action. So this line will execute reading file and caching it. 

val df3 = df1.where("point > 100")
df3.count() // This won't read the file again.
user_25047620
Related