I have to check whether a table exists in hive using spark(1.6.2) scala
If it doesn't I have to create an empty dataframe and save that as a hive table.
If it exists, then overwrite the existing table.
I need a function that returns boolean based on which I could take the above mentioned decisions(of whether to create new table or overwrite existing one)
1.x:
def tableExists(table: String, sqlContext: SQLContext) =
sqlContext.tableNames.contains(table)
2.x:
def tableExists(table: String, spark: SparkSession) =
spark.catalog.tableExists(table)
2.1.x or later.
You can use spark.catalog.tableExists. Credits go to Huseyin Oktay for pointing that out.
We also can define it with a database name as below.
1.6.x
sqlContext.tableNames("db_name").contains("tbl_name")
2.x:
spark.catalog.tableExists("db_name", "tbl_name")
For spark 3.x Java the following worked for me
boolean isTablePresent = sparkSession.catalog().tableExists({db_name},{table_name});