Spark Streaming XML files and converting them to DataFrames

Viewed 565

My objective is to:

  1. stream POS log (XML-files) from a Kafka-topic or any other HDFS using Spark
  2. convert the streamed data into a Spark DataFrame
  3. Analyze and store the resultant Dataframe as a csv.

In order to be sure about what I am doing, I am trying to run it locally for now using the spark-shell. The POS Log XML files are being generated using a python script every 1-2 seconds. The batch interval is set as 15 seconds val ssc = new StreamingContext(sc, Seconds(15));

Setting up the XML stream as : val xml_stream = ssc.fileStream(PATH);

How do I efficiently convert the xml_stream into a Spark-DataFrame?

I came across this spark-xml package. However, it does not give much idea about using it on a Dstream generated from XML files.

I am using scala as of now since pySpark only supports textFileStream.

0 Answers
Related