Currently I am sniffing packets from my local wlan interface like :
sudo tshark > sampleData.pcap
However, I need to feed this data to kafka.
Currently, I have a kafka producer script producer.sh:
../bin/kafka-console-producer.sh --broker-list localhost:9092 --topic 'spark-kafka'
and feed data to kafka like this:
producer.sh < sampleData.pcap
where in sampleData.pcap I have pre-captured IP packet information.
However, I wanna automate the process where it'd be something like this:
sudo tshark > http://localhost:9091
producer.sh < http://localhost:9091
This is obviously just a pseudoalgorithm. What I want to do is, send the sniffing data to a port and have kafka continuously read it. I don't want kafka to read from a file continuously because that'd mean tremendous amount of read/write operations from a single file causing inefficiency.
I searched the internet and came across kafka-connect but I can't find any useful documentation for implementing something like this.
What's the best way to implement something like this?
Thanks!