Reading Parquet Table's Data by Spark-Sql

Viewed 19

I have one Hive Exteral Table with below properties.

ROW FORMAT SERDE                                                         
'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'          
STORED AS INPUTFORMAT                                                    
'com.uber.hoodie.hadoop.HoodieInputFormat'                             
OUTPUTFORMAT                                                             
'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'
PARTITIONED BY: partition_date,hour

Now I'm Reading this table's data using spark sql.

spark.sql("select * from schemaname.tablename")

below is the schema for the file which is used by hive external table for actual data.

{
"name" : "id",
"type" : [ "string", "null" ]
  }, {
"name" : "system_ts",
"type" : [ "string", "null" ]
  },   {
"name" : "empname",
"type" : [ "string", "null" ]

  },
  .
  .
  .
  ...

Now when I am doing df.show() . It's only showing data for id,system_ts, and partition_column data(partition_date,hour). For remaning Column it's showing Null.

Its showing Correct Data when I am doing query by hive but by spark Getting Null.

can Someone help/Guide me , what is the issue here,and how should i handle this.

0 Answers
Related