I am reading json messages from Apache Kafka, then i use Apache Spark to write parquet files in the Azure blob storage.I use method partitionBy to write these parquet files in nested folders. My code is like this:
val sourceDF = spark.readStream
.format("kafka")
.option("kafka.bootstrap.servers", brokerList)
.option("subscribe", sourceTopic)
.option("startingOffsets", "latest")
.load()
sourceDF
.select(...)
.where("somecol"=="something")
.writeStream
.format("parquet")
.option("path", outputPath+"/somepath")
.option("checkpointLocation", checkpointLocation+"/somepath")
.partitionBy("date","country")
.queryName("test")
.trigger(Trigger.ProcessingTime(interval+" seconds"))
.start()
I have noticed that the spark application produces empty parquet files . This is a bottleneck for me due to the fact i read these parquet files in a hive import process and an exception is thrown that this is not a Parquet file (too small length: 0)
In general i want to forbid Spark streaming from writing empty files.