RabbitMQ topic exchange message ordering

user_1810154 0
Viewed 492

in the RabbitMQ specification there can be found:

Section 4.7 of the AMQP 0-9-1 core specification explains the conditions under which ordering is guaranteed: messages published in one channel, passing through one exchange and one queue and one outgoing channel will be received in the same order that they were sent. RabbitMQ offers stronger guarantees since release 2.7.0.

but what if there is binding which goes like Exchange 1 -> Exchange 2 -> Queue 1.

Is the ordering still guaranteed?

We assumed it did but we found in our application that it might not be the case. We use spring-rabbit-2.1.6-RELEASE (which uses amqp-client-5.4.3).

The publishers, binding and consumers are following:

Client 1 publishes to Exchange 1 -> Exchange 2 -> Queue 1 - consumed by Client 2
                                 -> Queue 2 - consumed by Client 3

We can see that Client 1 publishes 3 messages in following order:

  • Message 1
  • Message 2
  • Message 3

But the both Client 2 and Client 3 receive the messages in following order:

  • Message 3
  • Message 1
  • Message 2

EDIT 1 (Spring configuration)

For the publisher (Client 1) there is following XML configuration used (no extra properties set on rabbit's ConnectionFactory):

<rabbit:connection-factory channel-cache-size="1" cache-mode="CHANNEL" id="respConnFactory" addresses="..." virtual-host="..." username="..." password="..." executor="connExec"/>
<!-- the executor has no meaning for such usingas mentioned by Gary -->

The publishing is done via:

AmqpTemplate::send(String exchange, String routingKey, Message message)

in a dedicated thread.

Client 2 uses default spring configuration with SimpleMessageListenerContainer.

Client 3 isn't actually our application so I don't know the real setup. That was them who reported us a bug that the messages aren't ordered properly.

Of course there is still possibility that we logged the message publishing with some bug. But I triple checked it - it's from a single thread and there is sequence number in each message's custom header which is incremented correctly on Client 1.

EDIT 2

I did further analysis in order to find out how often the wrong message sorting happens. Here are the result:

I took the logs and data +-2 hours around the incident (4 hours in total) and there were 42706 messages sent and only 3 of them had wrong sorting on Client 2. All 3 messages were sent within interval of 7 ms.

Then I randomly took another time window of length 14 hours. There were 531904 messages sent and all of them received by Client 2 in correct order. The average message rate is ~11 messages per second.

The messages aren't distributed evenly so the 3 messages within 7 ms isn't anything especial - quite an opposite. It's common that within 3-5 ms there are multiple messages sent.

From this analysis I assume there was something weird going on on the rabbit cluster. Unfortunately I don't have the logs from it anymore.

The chance of some kind of race condition is from my point of view very low.

Thank you,

Frank

spring-rabbit spring-amqp amqp rabbitmq java
2 Answers

Spring AMQP uses a cache for channels; in a multi-threaded environment, there is no guarantee that the same thread will always use the same channel; hence ordering is not guaranteed.

With the current releases, the solution is to use scoped operations which will guarantee that a series of publications will occur on the same channel and guarantee order.

In the next release (2.3, available later this year), we have also added the ThreadChannelConnectionFactory which does the same thing.

user_12407630

It happened again and we were able to figure it out.

The whole time it was rabbit health indicator who was responsible for channel recreation and therefore for wrong order sorting. There was a job which periodically called the health endpoint.

As Gary correctly mentioned:

Spring AMQP uses a cache for channels; in a multi-threaded environment, there is no guarantee that the same thread will always use the same channel; hence ordering is not guaranteed.

The health status is checked from different thread and it uses the producer's channel.

As short term solution this will work:

management.health.rabbit.enabled=false

The sorting is guaranteed if the producer is really single thread and the connection factory is setup as in the description.

Another (and maybe proper) solution is to create separate ConnectionFactory and don't use the auto-configuration for rabbit health check.

@Bean("rabbitHealthIndicator")
public HealthIndicator rabbitHealthIndicator(ConnectionFactory healthCheckConnectionFactory) {
    RabbitTemplate rabbitTemplate = new RabbitTemplate(healthCheckConnectionFactory); // make sure it's a different connection factory than the one with guaranteed sorting
    return new RabbitHealthIndicator(rabbitTemplate);
}

That did the trick.

Cheers and thank you Gary for your help.

Frank

user_18101540
Related