Get table name from spark catalog

user_2328609 0
Viewed 982

I have a DataSourceV2Relation object and I would like to get the name of its table from spark catalog. spark.catalog.listTables() will list all the tables, but is there a way to get the specific table directly from the object?

apache-spark-sql apache-spark
1 Answers

PySpark

from pyspark.sql.catalog import Table

def get_t_object_by_name(t_name:str, db_name='default') -> Table:
  catalog_t_list = spark.catalog.listTables('default')
  return next((t for t in catalog_t_list if t.name == t_name), None)

Call as : get_t_object_by_name('my_table_name')

Result example : Table(name='my_table_name', database='default', description=None, tableType='MANAGED', isTemporary=False)

Table class definition : https://spark.apache.org/docs/2.3.0/api/java/org/apache/spark/sql/catalog/Table.html

user_83331030
Related