I am doing batching in my custom Kafka Connect sink connector, so not every call to put() will write data out, but most calls will just add the item to an internal buffer. Now I would like to rely on a timely call to SinkTask.flush(Map<TopicPartition, OffsetAndMetadata> currentOffsets) to flush the buffered items.
However, the default is surprisingly high at 60 seconds, which makes me wonder whether it would be a good idea to reduce it to say 100 milliseconds, or even 500 milliseconds?
So what are the downsides? Increased network traffic for the offsets? Drop in throughput?
Or is there any other way to implement flushing out my buffered items, something like punctuation in Kafka Streams perhaps?