Get length of Impala table varchar columns from Spark

Question

Try to get in Spark length of varchar columns, declared in following Impala table:

show create table dev_dm_rco.f102

Result:

CREATE TABLE dev_dm_rco.f102 (
  rep_id INT,
  msr_prd_id INT,
  ...
  bank_code VARCHAR(9),
  tu_cd VARCHAR(2),
  pl_stmt_num VARCHAR(5)'
)
PARTITIONED BY (
  report_date VARCHAR(50)
)
STORED AS PARQUET
LOCATION ...
TBLPROPERTIES ('spark.sql.create.version'='2.2 or prior', 'spark.sql.sources.schema.numPartCols'='1', 'spark.sql.sources.schema.numParts'='1'...)

First try- using Spark API:

String f102SqlQuery = "select * from dev_dm_rco.f102";

Dataset f102Dataset = SparkService.sql(f102SqlQuery);

for(StructField sf:f102Dataset.schema().fields())
log.info("f102dataset sf.name()="+sf.name()+" sf.dataType().typeName()="+sf.dataType().typeName());

Varchar columns returned as string without length:

f102dataset sf.name()=rep_id sf.dataType().typeName()=integer
f102dataset sf.name()=msr_prd_id sf.dataType().typeName()=integer
...
f102dataset sf.name()=bank_code sf.dataType().typeName()=string
...
f102dataset sf.name()=tu_cd sf.dataType().typeName()=string
f102dataset sf.name()=pl_stmt_num sf.dataType().typeName()=string
f102dataset sf.name()=report_date sf.dataType().typeName()=string

Second try- using show create table:

String f102CreateTable = "show create table dev_dm_rco.f102";
Dataset crtF102Stmt = sqlContext.sql(f102CreateTable);

log.info(f102CreateTable+": "+ crtF102Stmt.collectAsList());

Again all varchar columns returned as string with no length:

show create table dev_dm_rco.f102: 
[[CREATE TABLE `dev_dm_rco`.`f102`
(`rep_id` int, 
 `msr_prd_id` int, 
 ...
 `bank_code` string, 
 `tu_cd` string, 
 `pl_stmt_num` string')
PARTITIONED BY (`report_date` string)
ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'
WITH SERDEPROPERTIES (
  
)
STORED AS
  INPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat'
  OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'
TBLPROPERTIES (
  'transient_lastDdlTime' = '1596083577'
)
]]

Is there any way to get length of varchar fields declared in Impala table, using Spark? Spark version: 2.3.1 Thanks!

Get length of Impala table varchar columns from Spark

0 Answers