I am trying to load a 1GB csv file in kafka topic using Spooldir-connector in kafka-connect.
The issue is that the connector tasks fails after I run the command.
If I use "errors.tolerance":"all". The connector processes the data but it is not showing in the topics. I used kafkacat as consumer to see the data in topic.
I used following command.
curl -i -X PUT -H "Accept:application/json" -H "Content-Type:application/json" http://localhost:8083/connectors/source-csv-spooldir-1/config
-d '{"connector.class":"com.github.jcustenborder.kafka.connect.spooldir.SpoolDirCsvSourceConnector",
"topic": "data",
"input.path": "/data/unprocessed",
"finished.path": "/data/processed",
"error.path":"/data/error",
"input.file.pattern":".*\\.csv",
"schema.generation.enabled":"true",
"csv.first.row.as.header":"true",
"errors.tolerance":"all"
}'
also can anyone tell me how to mention key.schema and value.schema in this command?