We are working with mongodb kafka connetor on top of open source Apache Kafka connector, for data ingestion of json data from Mongo to HDFS. We have kafka consumer which reads data changes in kafka and writes them on hdfs file.
We want to schedule source connectors at specific time different time.
We need to trigger kafka message based on a scheduled date.
We can handle this scenario using source connector’s configuration properties from confluent with customise the polling interval
link:
==> poll.await.time.ms can be a solution
Otherwise, there is Kafka message scheduler:
https://github.com/etf1/kafka-message-scheduler
Automatically Consume Data From Kafka with the Scheduler
When you create a new scheduler, the vkconfig script takes the following steps:
Creates a new Vertica schema using the name you specified for the scheduler. You use this name to identify the scheduler during configuration.
Creates the tables needed to manage the Kafka data load in the newly-created schema.
from MongoDB Kafka connect official documentation :
Use the following configuration settings to specify aggregation pipelines for change streams and read preferences for change stream cursors.
poll.await.time.ms == The amount of time in milliseconds to wait before checking the change stream cursor for new results.
or use : poll.max.batch.size == Maximum number of documents to read in a single batch when polling a change stream cursor for new data. You can use this setting to limit the amount of data buffered internally in the connector.
