My requirement is to load 200+ oracle tables into 200+ mongo collections via Kafka. I thought of creating single JDBC source connector, 200+topics, 200+ sink connectors. But confluent team said, kafka might not handle these many topics and sink connectors. They asked me to consolidate. How to write 200+ oracle tables into single topic and write into mongo collection? Can you please help me?
2 Answers
Unless some particular context is missing from your question, then the suggestion that Kafka would not handle 200+ topics and sink connectors sounds incorrect.
You should map each logical table to a Kafka topic. The only reason I can think to consolidate multiple tables into one topic is if it's the same logical entity (for example, a table that's sharded across physical locations such as isolated warehouses). There are other reasons for putting multiple message types in one topic but they may well not apply here.
Depending on the volume of data you'd need to manage the scale of your Kafka Connect worker cluster but that's easily done with distributed mode and you just scale out horizontally to cope with the workload as required.
Disclaimer: I work for Confluent ;)
I am guessing here Confluent might have suggested that approach considering your Kafka cluster size otherwise, what you were doing makes sense.
But if you want to use only single topic you can try leveraging the kafka headers which can hold the table-name for an event.
Sugaan, Please find the sample code as requested.
Producer:-
ProducerRecord<String, JsonNode> record = new ProducerRecord<>(topic, key, value);
record.headers().add("test", "test1".getBytes());
producer.send(record
Consumer:-
final ConsumerRecords<String, byte[]> consumerRecords = consumer.poll(Duration.ofMillis(100000));
for (ConsumerRecord<String, byte[]> record : consumerRecords) {
record.headers().forEach(
(header -> System.out.println(header.key() + " " + header.value())));
}
Please let me know if you have any additional questions.
Related