Pardon if this question has been asked already. I am trying to save streamed data into HDFS using pyspark. Files are being created successfully on HDFS but these files are empty. Below is the simple code I am using.
Please help on fixing this issue.
from pyspark import SparkContext
from pyspark.streaming import StreamingContext
# Create a local StreamingContextwith two working thread and batch interval of 1 second
sc= SparkContext("local[2]", "NetworkWordCount")
ssc= StreamingContext(sc, 2)
# Create a DStream that will connect to hostname:port, like localhost:9999
linesDStream= ssc.socketTextStream("localhost", 9999)
# Split each line into words
wordsDStream= linesDStream.flatMap(lambda line: line.split(" "))
# Count each word in each batch
pairsDStream= wordsDStream.map(lambda word: (word, 1))
wordCountsDStream= pairsDStream.reduceByKey(lambda x, y: x + y)
# save the content into HDFS
wordCountsDStream.saveAsTextFiles("/home/cloudera/stream_Output/file")
wordCountsDStream.pprint()
# Start the computation
ssc.start()
# Wait for the computation to terminate
ssc.awaitTermination()
Using Cloudera quick start VM which comes with a spark 1.6.2 version.