Set kafka message key to source database name in Debezium Postgresql

mitix 0
Viewed 23

We are trying to collect changes from a number of Postgresql databases using Debezium. The idea is to create a single topic with a number of partitions equal to the number of databases - each database gets its own partition, because order of events matters. We managed to reroute events to a single topic using topic routing, but to be able to partition events by databases I need to set message key properly.

Qestion: Is there a way we can set kafka message key to be equal to the source database name?

My thougts:

  1. Maybe there is a way to set message key globally per connector configuration?
  2. Database name can be found in the message, but its a nested property payload.source.name. Didn't find a way to extract value from a nested propery.

Any thoughts?

Thank you in advance!

debezium cdc apache-kafka-connect apache-kafka
1 Answers

You'd need to write/find a Connect transform that can extract nested fields and set the message key, or if you don't mind duplicating data within Kafka topics, you can use Kafka Streams / KsqlDB, etc to do the same.

Overall, I don't think one topic + one partition per database is a good design for scalability of consumers. Sure, it'll keep order, but it's not much overhead to simply create one topic per database with only one partition. Then make consumers read all topics using a regex pattern rather than needing to assign to specific/all partitions in one topic.

OneCricketeer0
Related