WARN: There are more than one servers on the network broadcasting the same node id

user_8479386 0
Viewed 1313

I have a cluster of three ActiveMQ brokers running on separate machines. Now, I am seeing a warning repeatedly stating following

2020-06-17 10:40:07,378 WARN  [org.apache.activemq.artemis.core.client] AMQ212034: There are more than one servers on the network broadcasting the same node id. You will see this message exactly once (per node) if a node is restarted, in which case it can be safely ignored. But if it is logged continuously it means you really do have more than one node on the same network active concurrently with the same node id. This could occur if you have a backup node active at the same time as its live node. nodeID=03451127-a9c9-11ea-992a-005056ad92be

Here is a snippet of master's broker.xml:

<connectors>
   <connector name="nettyartemis">tcp://10.5.100.1:61616</connector>
</connectors>

<discovery-groups>
   <discovery-group name="my-discovery-group">
      <local-bind-address>10.5.100.1</local-bind-address>
      <group-address>231.7.7.7</group-address>
      <group-port>9876</group-port>
      <refresh-timeout>10000</refresh-timeout>
   </discovery-group>
</discovery-groups>

<cluster-connections>
   <cluster-connection name="my-cluster">
      <connector-ref>nettyartemis</connector-ref>
      <retry-interval>500</retry-interval>
      <use-duplicate-detection>true</use-duplicate-detection>
      <message-load-balancing>STRICT</message-load-balancing>
      <max-hops>1</max-hops>
      <discovery-group-ref discovery-group-name="my-discovery-group"/>
   </cluster-connection>
</cluster-connections>


<broadcast-groups>
   <broadcast-group name="my-broadcast-group">
      <local-bind-address>10.5.100.1</local-bind-address>
      <local-bind-port>5432</local-bind-port>
      <group-address>231.7.7.7</group-address>
      <group-port>9876</group-port>
      <broadcast-period>2000</broadcast-period>
      <connector-ref>nettyartemis</connector-ref>
   </broadcast-group>
</broadcast-groups>    

<ha-policy>
   <replication>
      <master/>
   </replication>
</ha-policy>

Here's a snippet of one of the slave's broker.xml:

<connectors>
   <connector name="nettyartemistwo">tcp://10.5.100.2:61616</connector>
</connectors>

<discovery-groups>
   <discovery-group name="my-discovery-group">
      <local-bind-address>10.5.100.2</local-bind-address>
      <group-address>231.7.7.7</group-address>
      <group-port>9876</group-port>
      <refresh-timeout>10000</refresh-timeout>
   </discovery-group>
</discovery-groups>

<cluster-connections>
   <cluster-connection name="my-cluster">
      <connector-ref>nettyartemistwo</connector-ref>
      <retry-interval>500</retry-interval>
      <use-duplicate-detection>true</use-duplicate-detection>
      <message-load-balancing>STRICT</message-load-balancing>
      <max-hops>1</max-hops>
      <discovery-group-ref discovery-group-name="my-discovery-group"/>
   </cluster-connection>
</cluster-connections>

<broadcast-groups>
   <broadcast-group name="my-broadcast-group">
      <local-bind-address>10.5.100.2</local-bind-address>
      <local-bind-port>5432</local-bind-port>
      <group-address>231.7.7.7</group-address>
      <group-port>9876</group-port>
      <broadcast-period>2000</broadcast-period>
      <connector-ref>nettyartemistwo</connector-ref>
   </broadcast-group>
</broadcast-groups>

<ha-policy>
   <replication>
      <slave/>
   </replication>
</ha-policy>

Any idea suggestion why I am getting this warning?

activemq-artemis jms
1 Answers

When a broker instance is first started it initializes its journal. One of the things the broker does during this initialization phase is to generate a UUID that will be used to identify the broker uniquely for things like clustering. This is called the "node id."

Typically when users see There are more than one servers on the network broadcasting the same node id it indicates that they've manually copied one broker's journal to another broker. This is usually done when users are initially configuring a cluster of brokers because they want to copy the configuration instead of starting from scratch on each node. However, instead of just copying the broker.xml to the other node the entire journal is copied as well, and since the journal contains the unique "node id" then both brokers end up using the same ID.

The solution in this situation is to delete the journal (stored in the data directory by default) from one of the brokers logging this message. The journal will be re-initialized and a new node id will be created once the broker is restarted.

This WARN message can also be logged if you have HA configured and both the master and the slave are active at the same time. A master and a slave naturally share the same node ID because they have the same journal (either via shared storage or replication). However, only one of the brokers is ever supposed to be active. If both brokers are active that's called "split-brain." This situation can be very problematic because both brokers will be operating independently on the same journal data. This can lead to duplicated messages as well as seemingly lost messages, and it can be extremely difficult (if not impossible) to recover the integrity of the data.

In a shared-store configuration the shared storage itself mitigates the risk of split-brain due to the shared file locks on the journal. Only one broker is physically allowed access to the journal data.

However, in a replicated configuration the risk of split-brain is much higher specifically because both the master and the slave have their own copy of the data. If the network connection between the master and the slave fails then the slave has no real way to know for sure if the master actually died or if it was just a network problem. This is why the documentation recommends using at least 3 live/backup pairs. This allows a proper quorum to be established so the live cluster members can vote to determine proper failover.

I also see that you haven't set <check-for-live-server>true</check-for-live-server> on your master which could lead to split-brain in the simple case where failover has happened and you restart the master broker without first shutting down the slave. Without <check-for-live-server>true</check-for-live-server> the master broker will simply start without checking if another broker (e.g. its backup) is broadcasting its node id.

user_83819460
Related