Spark : Can't set table properties start with "spark.sql" to hive external table while creating

Viewed 618

Env : linux (spark-submit xxx.py)

Target database : Hive

We used to use beeline to execute hql, but now we try to run the hql through pyspark and faced some issue when tried to set table properties while creating the table.

SQL

CREATE EXTERNAL TABLE example.a(
  column_a string)

TBLPROPERTIES (
  'discover.partitions'='true',  
  'spark.sql.sources.schema.numPartCols'='1', 
  'spark.sql.sources.schema.numParts'='1', 
  'spark.sql.sources.schema.part.0'='{"type":"struct","fields":[{"name":"column_a","type":"string","nullable":true,"metadata":{}}]}', 
  'spark.sql.sources.schema.partCol.0'='received_utc_date_partition'); 

Error message

Hive - ERROR - Cannot persist
example.a into Hive metastore as table property
keys may not start with 'spark.sql.': [spark.sql.sources.schema.partCol.0, spark.sql.sources.schema.numParts,
spark.sql.sources.schema.numPartCols, spark.sql.sources.schema.part.0];

In line 130-147 in spark source code it seems that it prevent all table properties that start with "spark.sql"

Not sure if I did it wrong or there's another way to set up the table properties for hive table. Any kinds of suggestion is appreciated.

0 Answers
Related