Not able to write Data in Parquet File using Spark Structured Streaming

Viewed 4305

I have a Spark Structured Streaming:

val df = spark
      .readStream
      .format("kafka")
      .option("kafka.bootstrap.servers", "localhost:9092")
      .option("startingOffsets", "earliest")
      .option("endingOffsets", "latest")
      .option("subscribe", "topic")
      .load()

I want to write data to FileSystem using DataStreamWriter,

val query = df
          .writeStream
          .outputMode("append")
          .format("parquet")
          .start("data")

But zero files are getting created in data folder. Only _spark_metadata is getting created.

However, I can see the data on console when format is console:

val query = df
          .writeStream
          .outputMode("append")
          .format("console")
          .start()

+--------------------+------------------+------------------+
|                time|              col1|              col2|
+--------------------+------------------+------------------+
|49368-05-11 20:42...|0.9166470338147503|0.5576946794171861|
+--------------------+------------------+------------------+

I cannot understand the reason behind it.

Spark - 2.1.0

2 Answers
Related