I followed the link for setting TwitterSource and HDFS sink.
Command used for starting the agent:
bin/flume-ng agent -n TwitterAgent -c conf -f conf/flume-conf.properties -Dflume.root.logger=DEBUG,console
I was successful in doing that, but there is one problem now. The output doesn't get written until stop the flume agent. It gets written to the HDFS once i kill my flume agent.
I have two questions here:
1) How to stop a flume agent - Is there any way other than doing CTRL+C.
2) Can I make the flume-agent write to the HDFS on the go, without having to kill the agent.
Note : After killing the process, the file that is written has .tmp as extension. Is it expected?
EDIT:
TwitterAgent.sinks.HDFS.hdfs.rollSize = 0
TwitterAgent.sinks.HDFS.hdfs.rollCount = 10000
TwitterAgent.sinks.HDFS.hdfs.rollInterval = 600