CQRS - out of order messages

user_10335281 0
Viewed 1331

Suppose we have 3 different services producing events, each of them publishing to its own event store.

Each of these services consumes other producers services events. This because each service has to process another service's events AND to create its own projection. Each of the service runs on multiple instances.

The most straight forward way to do it (for me) was to put "something" in front of each ES which is picking events and publishing (pub/sub) them in queues of every other service.

This is perfect because every service can subscribe to each topics it likes, while the event publisher is doing the job and if a service is unavailable events are still delivered. This seems to me to guarantee high scalability and availability.

My problem is the queue. I can't get an easily scalable queue that guarantees ordering of the messages. It actually guarantees "slightly out of order" with at-least once delivery: to be clear, it's AWS SQS.

So, the ordering problems are:

  • No order guaranteed across events from the same event stream.
  • No order guaranteed across events from the same ES.
  • No order guaranteed across events from different ES (different services).

I though I could solve the first two problems just by keeping track of the "sequence number" of the events coming from the same ES. This would be done by tracking the last sequence number of each topic from which we are consuming events This should be easy for reacting to events and also building our projection. Then, when I pop an event from the queue, if the eventSequenceNumber > previousAppliedEventSequenceNumber + 1 i renqueue it (or make it invisible for a certain time).

But it turns out that using this solution, it will destroy performances when events are produced at high rates (I can use a visibility timeout or other stuff, the result should be the same).

This because when I'm expecting event 10 and I ignore event 11 for a moment, I should ignore also all events (from ES) with sequence numbers coming after that event 11, until event 11 shows up again and it's effectively processed.

Other difficulties were:

  • where to keep track of the event's sequence number for build the projection.
  • how to keep track of the event's sequence number for build the projection so that when appling it, I have a consistent lastSequenceNumber.

What I'm missing?

P.S.: for the third problem think at the following scenario. We have a UserService and a CartService. The CartService has a projection where for each user keeps track of the products in the cart. Each cart's projection must have also user's name and other info's that are coming from the UserCreated event published from the UserService. If UserCreated comes after ProductAddedToCart the normal flow requires to throw an exception because the user doesn't exist yet.

event-sourcing microservices distributed-computing cqrs
2 Answers

What I'm missing?

You are missing flow -- consumers pull messages from sources, rather than having sources push the messages to the consumers.

When I wake up, I check my bookmark to find out which of your messages I read last, and then ask you if there have been any since. If there have, I retrieve them from you in order (think "document message"), also writing down the new bookmarks. Then I go back to sleep.

The primary purpose of push notifications is to interrupt the sleep period (thereby reducing latency).

With SQS acting as a queue, the idea is that you read all of the enqueued messages at once. If there are no gaps, then you can order the collection then start processing them and acking them. If there are gaps, you either wait (leaving the messages in the queue) or you go to the event store to fetch copies of the missing messages.

There's no magic -- if the message pipeline is promising "at least once" delivery, then the consumers must take steps to recognize duplicate messages as they arrive.

If UserCreated comes after ProductAddedToCart the normal flow requires to throw an exception because the user doesn't exist yet.

Review Race Conditions Don't Exist, by Udi Dahan: "A microsecond difference in timing shouldn’t make a difference to core business behaviors."

user_547340

The basic issue is assuming we can get messages IN ORDER... This is a fallacy in distributed computing... I suggest you design for no message ordering in your system.

As for your issues, try and use UTC time in the message body/header created by the originator and try and work around this data point. Sequence numbers are going to fail unless you have a central deterministic sequence creator (which will be a non-scalable, single point of failure).

Using Sagas/State machine is a path that can help to make sense of (business) events ordering.

user_24610730
Related