What's the difference between SparkSession.catalog and SparkSession.sessionState.catalog?

user_8269493 0
Viewed 474

I'm learning Spark, got confused about Spark's Catalog.

I found a catalog in SparkSession, which is an instance of CatalogImpl, as below

  /**
   * Interface through which the user may create, drop, alter or query underlying
   * databases, tables, functions etc.
   *
   * @since 2.0.0
   */
  @transient lazy val catalog: Catalog = new CatalogImpl(self)

And I found that there is a catalog in SparkSession.sessionSate, which is an instance of SessionCatalog.

What's the difference between them?

apache-spark-sql apache-spark
1 Answers

What's the difference between them?

tl;dr None.

The line in CatalogImpl is the missing piece in your understanding:

private def sessionCatalog: SessionCatalog = sparkSession.sessionState.catalog

In other words, SparkSession.catalog creates a CatalogImpl that uses sparkSession.sessionState.catalog under the covers.

user_13053440
Related