Unable to read partitioned hive table created from spark structured streaming using partitioned by

user_13248011 0
Viewed 29
interval2    
  .writeStream 
  .format("parquet") 
  .option("path","/user/usera/interval27e") 
  .partitionBy("year","month","day","mm") 
  .trigger(processingTime='15 seconds') 
  .option("checkpointLocation", "/user/usera/checkpoint27e") 
  .start()

Im able to see partitioned structure in hdfs but when try to read hive table getting null result

spark-structured-streaming
1 Answers

Please can you try with the below code:

val dataWrite =  interval2    
    .writeStream
    .format("parquet")
    .partitionBy("year","month","day","mm")
    .option("checkpointLocation", "/user/usera/checkpoint27e")
    .outputMode(OutputMode.Append).trigger(Trigger.ProcessingTime("15 seconds"))
    .start("/user/usera/interval27e")

  dataWrite.processAllAvailable()
user_123358380
Related