While using stand alone set up, if the input CSV file is bigger, anywhere from 50MB and above, the Kafka connector to read CSV stops abruptly. Restarting the Kafka connector helps to finish processing but adds some duplicate messages. Is there any threshold to the file size to read or any memory, CPU requirements to process CSV files upto 500MB size?
Kafka SFTP Source Connector stops if CSV file is big ( 50 MB to 500 MB) and needs restart
Umesh N
0
Viewed 32
1 Answers
The default value for behavior.on.error is FAIL. According to the docs:
this way the connector stops whenever an error occurs. Set the value to LOG, so that possible errors are logged and processing jumps to the next file. You should be able to understand the reason for the failure.
Take a look at the following post on OpenCsv behaviour on failures:
Reading large CSV files in Java
You might also activate connector logs. Take a look at the docs:
https://docs.confluent.io/platform/current/connect/logging.html#log4j-properties-file
Related