I want to cache a table(dataframe) in one notebook and use it in another notebook , I am using same databricks cluster for both the notebooks.
Please suggest if this is possible , If yes then how ?
2 Answers
Yes it is possible based on the following setups .
You can register your dataframe as temp table . The lifetime of temp view created by createOrReplaceTempView() is tied to Spark Session in which the dataframe has been created.
spark.databricks.session.share to true
this setup global temporary views to share temporary views across notebooks. ref : link
You can share dataframe between notebooks.
On the first notebook please register it as temp view:
df_shared.createOrReplaceGlobalTempView("df_shared")
On the second notebook please read it from global temp database:
global_temp_db = spark.conf.get("spark.sql.globalTempDatabase")
df_shared= table(global_temp_db + ".df_shared")
Related