If i am using both Scala and python in same notebook in databricks, how spark session would create two session or single session?
If it is different session for Scala and python, how tempview is sharing across them? Eventhough we are not able to share variables among them.
It will work with the same Spark JVM instance defined on the cluster level. And because of this, temp view defined in one language will be visible from another cell that uses different language - that's common way of sharing datasets between different languages. Another way to pass data between languages is to use spark.conf.set & spark.conf.get - but it should be limited to the small amounts of data.
P.S. Look into this answer that shows how to share variables between Pytnon & SQL.
