Using Kafka to share state between microservices

user_4770659 0
Viewed 33

upfront: I have read some articles now about Kafka in combination with microservices and understand that there are multiple ways you could use it. I just wanted to ask a small question because I am still not quite sure if I really got the whole idea:

So let's start with a small example (keep in mind that this is just an example): Let's say I have a service storing users and another one storing blog posts. Let's say when I have to persist a new blog post I have a userId but I would like to also store the username with the blog post. We obviously don't want to make sync calls to the user service so we look for another solution, in this case one with Kafka.

If I got it right the user service should always publish updates to a Kafka queue when a user is updated in it's db, what I am still not sure about is how the blog service would extract the latest username from that queue.

What I have seen so far: The event are change events and the blog service rebuilds the db on its own end by reading all events. The issue I see here is with years of events this could take a while if a new instance spins up. The other thing I saw was reactive Kafka queries and that it is somehow possible to query that internal state of Kafka to get the latest info.

So finally my questions:

  1. Is my approach to publish updates on the user service (that is owning the user data) to some kind of queue in some kind of way, or would this be done in another way?
  2. What exactly are now the possibilities the blog service could use to get that username by the userId it has (from that queue)?
event-sourcing apache-kafka-streams microservices apache-kafka
2 Answers

I am also new in that world of kafka but I can express my opinion.

1-OnUserUpdate: lest's assume that you have the blog already stored with username, and the user updated his username, in this case kafka fit the need , the user service can push an event with key userId(to guarantee the order) and value a data containing the username and then the blog service will consume the message and update user blogs easily. And you can menage policy retention, there is no need to store events for ever, you can set one month or one week...

2-OnBlogCreate: here it is a little bit tricky, how will you get the userName, in this case I don't think it is a good idea to request kafka, it means you will have all usernames in kafka and that is not the main goal of kaka even it could be done. Topics are data append only, so as you said if you store data in kafka how will you manage updates on users ?

user_76562410

Your user topic can be consumed into a KTable with Kafka Streams / KsqlDB. This would create a compacted topic behind the scene such that any new userID event would describe an upsert into the table. A null value for any ID would delete from the table.

Your blog topic then joins/queries its userID against the keys of that table to get some "UserBlog" unioned event. At a high level, it works no differently than doing a remote database lookup.

https://kafka.apache.org/32/documentation/streams/developer-guide/dsl-api.html#kstream-ktable-join

Your other options include "database per service" or CQRS microservice patterns, with Kafka as only an event bus and consumer services reading events into their own unique databases (which could be done with Kafka Connect)

user_23086830
Related