My objective is to:
- stream POS log (XML-files) from a Kafka-topic or any other HDFS using Spark
- convert the streamed data into a Spark DataFrame
- Analyze and store the resultant Dataframe as a csv.
In order to be sure about what I am doing, I am trying to run it locally for now using the spark-shell. The POS Log XML files are being generated using a python script every 1-2 seconds.
The batch interval is set as 15 seconds
val ssc = new StreamingContext(sc, Seconds(15));
Setting up the XML stream as : val xml_stream = ssc.fileStream(PATH);
How do I efficiently convert the xml_stream into a Spark-DataFrame?
I came across this spark-xml package. However, it does not give much idea about using it on a Dstream generated from XML files.
I am using scala as of now since pySpark only supports textFileStream.