I am using Kafka MongoDB Source Connector [https://www.confluent.io/hub/mongodb/kafka-connect-mongodb] with confluent platform v5.4.1 and MongoDB v3.6 ReplicaSet. Kafka MongoDB source connector was deleted and now when it is recreated again after a month, this when I get the following error.
com.mongodb.MongoQueryException: Query failed with error code 280 and error message 'resume of change stream was not possible, as the resume token was not found. {_data: BinData(0, "825F06E90400000004463C5F6964003C38316266623663632D326638612D343530662D396534652D31393936336362376130386500005A1004A486EE3E58984454ADD5BF58F364361E04")}' on server 40.118.122.226:27017
at com.mongodb.operation.QueryHelper.translateCommandException(QueryHelper.java:29)
at com.mongodb.operation.QueryBatchCursor.getMore(QueryBatchCursor.java:267)
at com.mongodb.operation.QueryBatchCursor.tryHasNext(QueryBatchCursor.java:216)
at com.mongodb.operation.QueryBatchCursor.tryNext(QueryBatchCursor.java:200)
at com.mongodb.operation.ChangeStreamBatchCursor$3.apply(ChangeStreamBatchCursor.java:86)
at com.mongodb.operation.ChangeStreamBatchCursor$3.apply(ChangeStreamBatchCursor.java:83)
at com.mongodb.operation.ChangeStreamBatchCursor.resumeableOperation(ChangeStreamBatchCursor.java:166)
at com.mongodb.operation.ChangeStreamBatchCursor.tryNext(ChangeStreamBatchCursor.java:83)
at com.mongodb.client.internal.MongoChangeStreamCursorImpl.tryNext(MongoChangeStreamCursorImpl.java:78)
at com.mongodb.kafka.connect.source.MongoSourceTask.getNextDocument(MongoSourceTask.java:338)
at com.mongodb.kafka.connect.source.MongoSourceTask.poll(MongoSourceTask.java:155)
at org.apache.kafka.connect.runtime.WorkerSourceTask.poll(WorkerSourceTask.java:265)
at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:232)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:177)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:227)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2020-07-09 09:53:09,353] INFO Watching for collection changes on '<myDBName.myCollectionName>' (com.mongodb.kafka.connect.source.MongoSourceTask:374)
After searching for the cause of this error, I do understand that the resume token is not found in the Oplog as Oplog is memory/size capped and it would have flushed out the old information. I also do understand that to minimize the occurrence of this issues I should increase the Oplog size etc. But I want to know if there any possibility of fixing this from Kafka/Confluent Platform side? Like if I could delete the Kafka topic, KSQL topic as I am creating a stream using the topic 'myDBName.myCollectionNamedata', data associated with the topic, or do some operation in the Kafka Connect so that the MongoDB source connector starts capturing the changes in MongoDB Collections again from the current time discarding the old information?