Does Spark hold DataFrame in memory when loaded from a file?

user_959788 0
Viewed 693

If I create a Dataframe like so:

val usersDF = spark.read.csv("examples/src/main/resources/users.csv")

Does spark actually load (/copy) the data (from the csv file) into memory, or into the underlying filesystem as a distributed dataset?

I ask because after loading the df, any change in the underlying file's data is not reflecting in queries against the dataframe. (Unless ofcourse the dataframe is again freshly loaded by invoking the above line of code.

I am using interactive queries on Databricks notebooks.

apache-spark-sql apache-spark in-memory
1 Answers

Unless until you perform an action on that file, the file doesn't gets loaded into memory and you will see all the contents of the file till the time it is loaded into memory when an action occurs in the execution plan.

And if an action has already been taken on the file during which any modification has been done to the file, then you will see the cached result of the first execution if it is able to fit in MEMORY.

user_100615880
Related