Kafka architecture: Sync user data from SSO to many microservices

Thiago Freitas 0
Viewed 20

I have one microservice that is for register users (SSO) and 40 more microservices that consume user from SSO (All microservice have partial copy of user data in your database, like username, email, phone, name, etc). Today, we have used webhook to synchronize data. Now we're studying how to run this with kafka. The doubt is: Having than 40 more microservices, I need to create one topic per microservice or is possible to create one topic "user" and all microservices read the changes from this? If I can create one topic, how to guarantee that one microservice can not read same data 2 or more times?

If necessary know who programming language we are using is python

apache-kafka python
1 Answers

need to create one topic per microservice

Unclear why you think that's necessary

possible to create one topic "user" and all microservices read the changes from this

Absolutely. Or, only "one service" as suggested below to create a KTable of users.

how to guarantee that one microservice can not read same data 2 or more times

You can use a unique group.id per service, and you need to enable appropriate settings to enable EOS semantics, since by default Kafka can only provide at-least-once delivery, meaning you could get data "2 or more" times...

However, the better way to do this is to use Interactive Queries (or kSQLDB REST API for Python folks) to create a CQRS pattern rather than have a "database per service" approach.

Or Kafka Connect can also be used to populate those partial databases.

OneCricketeer0
Related