If you save a DataFrame to disk, will Spark load that data if you use that DataFrame lower in the script?

user_13969 0
Viewed 154

If you load some data, compute a DataFrame, write that to disk and then use the DataFrame later... assuming it isn't still cached in RAM (lets say there wasn't enough), would Spark be smart enough to load the data from disk rather than recompute the DataFrame from the original data?

For example:

df1 = spark.read.parquet('data/df1.parquet')
df2 = spark.read.parquet('data/df2.parquet')

joined = df1.join(df2, df1.id == df2.id)
joined.write.parquet('data/joined.parquet')

computed = joined.select('id').withColummn('double_total', 2 * joined.total)
computed.write.parquet('data/computed.parquet')

Under the right circumstances, when we store computed, will it load the joined DataFrame from data/joined.parquet or will it always re-compute by loading/joining df1/df2 if it isn't currently caching joined?

pyspark apache-spark-sql apache-spark caching python
1 Answers

The joined dataframe points to df1.join(df2, df1.id == df2.id). As far as I know the parquet writer will not cause any changes to that reference therefore in order to load the parquet data you need to construct a new Spark reader with spark.reader.parquet(...).

You can verify the above claim from the DataFrameWriter code (check parquet/save methods) which returns Unit and not modifying somehow the reference of the source dataframe. Finally to answer your question in the above example the joined dataframe will be calculated once for joined.write.parquet('data/joined.parquet') and once for computed.write.parquet('data/computed.parquet')

user_7503760
Related