I have one Hive Exteral Table with below properties.
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'
STORED AS INPUTFORMAT
'com.uber.hoodie.hadoop.HoodieInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'
PARTITIONED BY: partition_date,hour
Now I'm Reading this table's data using spark sql.
spark.sql("select * from schemaname.tablename")
below is the schema for the file which is used by hive external table for actual data.
{
"name" : "id",
"type" : [ "string", "null" ]
}, {
"name" : "system_ts",
"type" : [ "string", "null" ]
}, {
"name" : "empname",
"type" : [ "string", "null" ]
},
.
.
.
...
Now when I am doing df.show() . It's only showing data for id,system_ts, and partition_column data(partition_date,hour). For remaning Column it's showing Null.
Its showing Correct Data when I am doing query by hive but by spark Getting Null.
can Someone help/Guide me , what is the issue here,and how should i handle this.