RabbitMQ C# Client recovering from node failure on cluster with quorum queues

user_8118024 0
Viewed 1048

I've set-up a three node RabbitMQ cluster with a quorum durable queue.

I'm trying to find out how to implement a robust way to keep things running both on the producer and the consumer side (two .NET Core processes) in case of node failure.

I'm using the following options on the ConnectionFactory class:

var factory = new ConnectionFactory
{
    HostName = hostname,
    AutomaticRecoveryEnabled = true,
    TopologyRecoveryEnabled = true,
    VirtualHost = vhost
};

however, after starting the producer & consumer test processes (which attempt to flood the queue), whenever I stop the master node on the cluster, the clients never recover from this situation, and an OperationInterruptedException is thrown on each call to BasicPublish (on the producer) or BasicAck (on the consumer).

The clients connect to the cluster using a random ip choosen from the three nodes (as given from round-robin dns resolution).

I've read somewhere that for durable classic non-mirrored queues this is the expected behavior, but what about quorum queues? Shouldn't they be a more efficient version of mirrored queues (although with some limitations)?

Is there a way to recover from a single node failure without implementing all the reconnection logic in my clients?

rabbitmq client cluster-computing c#
2 Answers

From what I can see in ConnectionFactory class, you can specify list of HostNames while creating a connection (not on the step of declaring factory). Have you tries this one?

// Summary:
//     Create a connection using a list of hostnames using the configured port. By default
//     each hostname is tried in a random order until a successful connection is found
//     or the list is exhausted using the DefaultEndpointResolver. The selection behaviour
//     can be overriden by configuring the EndpointResolverFactory.
//
// Parameters:
//   hostnames:
//     List of hostnames to use for the initial connection and recovery.
//
// Returns:
//     Open connection
//
// Exceptions:
//   T:RabbitMQ.Client.Exceptions.BrokerUnreachableException:
//     When no hostname was reachable.
public IConnection CreateConnection(IList<string> hostnames);
user_112043910

The clients connect to the cluster using a random ip choosen from the three nodes (as given from round-robin dns resolution).

The problem is likely to be caused by DNS cache. E.g. producer/consumer DNS-resolved the hostname to a single ip address and then cached it (there is a DNS cache in .NET), causing the RabbitMQ client to always connect to the same RabbitMQ node (which could be shut down). There are at least 3 approaches for making the failover process more robust:

  • pass multiple hostnames or ip addresses of the RabbitMQ cluster nodes to the RabbitMQ client
  • put RabbitMQ nodes behind a level-4 load balancer and configure the RabbitMQ client to connect to the load balancer instead of connecting directly to the cluster nodes
  • disable DNS cache
user_6800320
Related