Is it possible to for multiple kafka connect cluster to read from the same file and write on the same topic with SpoolDirCsvSourceConnector?

EruNaoto 0
Viewed 23

I'm using SpoolDirCsvSourceConnector to load CSV data into one Kafka topic. My CSV input file is around 3-4 Gb and I have only run the connector on one single machine, so throughput is low.

EDIT: I have to consume the .csv file. Provider sends me one big .csv file daily.

Would it possible to run the connector on multiple machines on the same file to increase throughput? These machines may or may not be able to see each other, but they will be able to connect to the same kafka cluster. If possible, I want to avoid splitting the CSV file into parts.

apache-kafka-connect apache-kafka
1 Answers

Unless you split the file into parts, you can only use a single instance of the connector.

Where is the data coming from? Do you have the option of consuming it directly into Kafka instead of via a CSV file? e.g. if it's from a database have you considered reading from the database directly using Kafka Connect instead?

Robin Moffatt0
Related