How to avoid race conditions in Kafka

user_961738 0
Viewed 1316

Imagine this business logic. I'm buying a tshirt from a store that runs microservices and uses Kafka. When I submit my order the web back end saves my order and kicks out two async requests:

  • Payment Service that handles authorizing my payment (credit card stuff).
  • User Service that handles creating my user account (email confirmation step).

Both of those services write to their own kafka topics when they've finished. The web back end subscribes to these topics and when an event comes in it updates the order. Once the order back end has confirmed my user and payment information, it will send that order off to be fulfilled.

How do you prevent race conditions when both events come back at the same time?

  1. If two processes both update the order at the same time, it never sees the complete order and you risk having complete orders that never get fulfilled.

  2. If two processes both update and then immediately requery, they could both see a complete order and then we fulfill the order twice.

Thanks!

apache-kafka python
1 Answers

Whatever method you choose you will need some sort of synchronization point or total ordering. A few approaches are below.

You could serialize your writes via transactions to some DB under the hood and only execute the fulfillment after both events have arrived (check for both events on either event write). Up to you to use pessimistic/optimistic locking here and if it is feasible for your DB selection and expected loads.

Or you could create 1 topic and partition these writes by some unique identifier so that both events go to the same partition thus to the same Kafka consumer. This avoids race conditions assuming your consumers are single threaded. To do that just add that identifier (user-id, tx id etc) to the Kafka key to make any event sent with that key go to the same partition and avoid multi-partition race conditions. This approach has the downside of having multiple event types in 1 topic but I have seen this firehose pattern used effectively before to provide ordering.

user_21888930
Related