Put data first in Kafka or Database?

user_1786646 0
Viewed 3559

What are te pro's and con's when putting data first in Kafka and then in a database, or the other way?

An example: A user does a REST (POST) call to store let's say products. Normally i would pick up this call in the backend and save the body to a database (after validation and all..). Is it a best practice to pick up this call and store de data in Kafka fist and then save it to the database (in this case, the database is a kafka consumer).

Or is it better to save it in the database first, then send it to kafka?

Thanks

apache-kafka database
4 Answers

lets take example of both scenarios with your use case, api call for storing a product lets say PRODUCT1 :

you database: product_table(product_id , product_name,product_info)

API pseudo code:

  1. valiadteProductInfo
  2. save - either first in kafka or in DB

APPROACH 1 -

saving to kafka first means that you might see that result in the DB some time later, you will return the product id to the user and if user wants to populate the product id its not visible. for me this is not the right approach as you will then need to handle many things on the UI side for such delay.

APPROACH 2 - saving to db first and kafka second there are two scenarios: 1. kafka push is sync in code- in this case in sending to kafka fails , which in your business case if very critical as other microservice is dependent. it is not the right approach , but if its ok that for <0.001 % of the time if push fails and then you delete the product from DB and return exception to the user. I think its completely ok on this.

  1. kafka push is by polling the db for changes and put changes to kafka (read about EventSourcing for this) : in this case you will get 100% guarantee , but some small delay . this also you can use
user_19872660

I'd prefer to put Kafka as it has guarantee that message will not lost and it is durable. But if u will put 1st put to db then Kafka there is risk that your service can go down between writing to db and kafka.

user_25640320

This completely depends on your requirement.

  1. If you want the functionality to be:

-on failure of push to kafka topic log exception and exit.

-irrespective of kafka push is success or not save data at your end.

-make the consumer save it to DB. I am assuming that when you are sending message you'd want to manipulate data in your listener method. So, it depends what state of data you want to persist in your DB.

  1. Also if you are using Kafka you'd be calling another microservice, is your table which you want to update exposed to both the services, i.e, if the services share the database (ideally they won't).

  2. If database is not shared and you still want to persist that data you'd need to save before or after making pushMessage call to kafka, because that is the extent of check you can have if the message is successfully sent or not. pushMessage will have on failure method, you can throw and exception there and based on failure futher save data or exit.

user_40221590

You will want to at first create various topics, which will act as queues in Kafka for your data. You will then have consumers of that data that will write into your database. This will allow for your system to be re-entrant should one component fail.

Additionally, should you have any other consumers of the data, it is a simple as creating a consumer of the kafka queue and exposing it to your consumer via the common interface agreed upon (REST, SOAP, RPC etc).

Furthermore, you can scale performance by having multiple readers per topic that can then federate their requests to the database.

user_57519020
Related