Storing chat messages for a messaging microservice

user_17460590 0
Viewed 302

I am making a real-time based chat microservice and this is my first time making such application. I am kind of confused on how to store the conversations because I think storing them in a database (Cassandra or whatever) is not an optimal solution for the long run.

I've read about Apache Kafka and found that this could be a solution to my problem. Thus, I'm not quite sure if I'm there yet. I just need to know If Kafka is enough without the need of an external database (Just talking about the data related to the messages though, not the users or any other type of data I may need). Because I've read that Kafka provides in its topics the option to use a retention time of "-1" which basically means forever, but AS LONG AS THE SERVER IS RUNNING while I'm not sure I can keep the same server running forever, could someone please clarify this for me ?

microservices apache-kafka message
2 Answers

I'm not sure I can keep the same server running forever

In practice, you have more than one server. The Kafka server process (known as a broker) is designed to run as a cluster. Data is written to one and replicated to others. If one server becomes unavailable the data is then served from another.

I think storing them in a database (Cassandra or whatever) is not an optimal solution for the long run.

I'm not sure that's a correct assumption on which to base your usage of Kafka. Both RDBMS and NoSQL stores can—and are—used as long term stores of data for numerous applications including chat. It depends on your use case, the volumes you envisage, and perhaps also how else you're going to use the data.

user_3506130

read about Apache Kafka and found that this could be a solution to my problem.... need to know If Kafka is enough

For storage, sure. For message delivery to clients, no.

Some extreme examples come up immediately when considering using any message broker

  • you have one topic per "chat conversation", and your chat app becomes really popular - you now have hundreds of thousands of topics, which Kafka cannot handle. A database can be filtered/sharded/partitioned by many fields, including user id
  • chat messages must be ordered, therefore you could really one use one partition for all topics ; with a database, you can always order by timestamp at query time
  • If you used a "firehose" approach of one topic with all messages from everybody, then redistribute the data downstream, then infrequent clients are going to lag behind the most frequent user-client because there are more messages to process.

Most importantly, chat apps are frontend applications; Kafka is a backend technology and doesn't "push" to frontend services. For mobile/browser use-cases, you wouldn't have access to embed a Kafka consumer in them. A REST API layer would have to be added to request messages, just as it would with a database.

A better design would be to generate message events rather than store message data - E.g. "At time T, user X send 'M' message with content C to location/users L"... A topic with that data does not need to be persisted forever, just long enough to be consumed. And still, with all that information, you then need to consider topic partitioning for scalability, then you can filter and dump that into some more persistent location because your user-facing application wouldn't be consuming every single one of those events.

I'm not sure I can keep the same server running forever

Someone will have to. Use a hosted solution, if you cannot.

user_23086830
Related