How to get Create Statement of Table in some other database in Spark using JDBC

user_10097157 0
Viewed 2978

Problem statement:

  1. I have a Impala database where multiple tables are present

  2. I am creating Spark JDBC connection to Impala and loading these tables into spark dataframe for my validations like this which works fine:

     val df = spark.read.format("jdbc")
              .option("url","url")
              .option("dbtable","tablename")
              .load()

  3. Now the next step and my actual problem is I need to find the create statement which was used to create the tables in Impala itself

  4. Since I cannot run command like below as it gives error, is there anyway I can fetch the show create statement for tables present in Impala.

    val df = spark.read.format("jdbc")
              .option("url","url")
              .option("dbtable","show create table tablename")
              .load()
impala apache-spark jdbc scala
1 Answers

Perhaps you can use Spark SQL "natively" to execute something like

val createstmt = spark.sql("show create table <tablename>")

The resulting dataframe will have a single column (type string) which contains a complete CREATE TABLE statement.

But, if you still choose to go JDBC route there is always an option to use the good old JDBC interface. Scala understands everything written in Java, after all...

import java.sql.*
Connection conn = DriverManager.getConnection("url")
Statement stmt = conn.createStatement()
ResultSet rs = stmt.executeQuery("show create table <tablename>")
...etc...
user_6387640
Related