Describe Snowflake table from Azure Databricks

user_5788656 0
Viewed 135

I want to issue a DESC TABLE SQL command for a Snowflake table and using Azure Databricks but I can't quite figure it out! I'm not getting any errors but I'm not getting any results either. Here's the Python code I'm using:

options_vcp = {
  "sfUrl": snowflake_url,
  "sfUser": user,
  "sfPassword": password,
  "sfDatabase": db,
  "sfWarehouse": wh,
  "sfSchema": sch
}

sfUtils = sc._jvm.net.snowflake.spark.snowflake.Utils

sfUtils.runQuery(options_vcp, "DESC TABLE myTable")

I can download the Snowflake table using the "sfDatabase", "sfWarehouse", etc. values so they seem to be correct. I can run the DESC TABLE command in Snowflake and get correct results. But the only output I'm getting from databricks is this:

Out[1]: JavaObject id=o315

Does anyone know how to display this JavaObject or know of a different method to run DESC TABLE from Databricks?

snowflake-cloud-data-platform azure-databricks databricks python
2 Answers

From doc: Executing DDL/DML SQL Statements:

The runQuery method returns only TRUE or FALSE. It is intended for statements that do not return a result set, for example DDL statements like CREATE TABLE and DML statements like INSERT, UPDATE, and DELETE. It is not useful for statements that return a result set, such as SELECT or SHOW.

Alternative approach is to use INFORMATION_SCHEMA.COLUMNS view:

df = spark.read.format(SNOWFLAKE_SOURCE_NAME)
    .options(sfOptions)
    .option("query", "SELECT * FROM information_schema.columns WHERE table_name ILIKE 'myTable'")
    .load()

Related: Moving Data from Snowflake to Spark:

When using DataFrames, the Snowflake connector supports SELECT queries only.

Usage Notes

Currently, the connector does not support other types of queries (e.g. SHOW or DESC, or DML statements) when using DataFrames.

user_50708790
user_149691840
Related