Spark Structured Streaming Application produces empty parquet files to Azure blob

Viewed 233

I am reading json messages from Apache Kafka, then i use Apache Spark to write parquet files in the Azure blob storage.I use method partitionBy to write these parquet files in nested folders. My code is like this:

val sourceDF = spark.readStream
      .format("kafka")
      .option("kafka.bootstrap.servers", brokerList)
      .option("subscribe", sourceTopic)
      .option("startingOffsets", "latest")
      .load()
      



sourceDF
      .select(...)
      .where("somecol"=="something")
      .writeStream
      .format("parquet")
      .option("path", outputPath+"/somepath")
      .option("checkpointLocation", checkpointLocation+"/somepath")
      .partitionBy("date","country")
      .queryName("test")
      .trigger(Trigger.ProcessingTime(interval+" seconds"))
      .start()

I have noticed that the spark application produces empty parquet files . This is a bottleneck for me due to the fact i read these parquet files in a hive import process and an exception is thrown that this is not a Parquet file (too small length: 0)

In general i want to forbid Spark streaming from writing empty files.

1 Answers

dataframe.rdd.isEmpty() //Check for empty dataframe in a if and don't run.

Related