RDQM IBM Pacemaker prob - HA queue manager is not running where it should be

user_12393773 0
Viewed 604

I'm facing some strange RDQM behavior. We have 3 servers (node1: primary, node2, and node3: secondary). 4 QMs of 1G each, are created on the primary with a preferred location node1 and node2. Due to some problems (connection problems, I think), 1 QM is switched to primary on node3. The pacemaker indicates that node3 is its master and the other nodes are Slaves. I tried to restart node3 but as soon as it is accessible, the QM switches back to node3.

I tried with difficulty to reproduce the problem with other QM but it's impossible. What do you think is the origin of the problem?

ibm-mq
2 Answers

There are a number of possible causes for a high availability queue manager to not run on the node you may expect it to, a common cause is failed resource actions. If you run the crm status command, you may see a "Failed Resource Actions" section which may detail a failed resource action preventing the queue manager from running on its preferred node.

The rest of my answer assumes that you did have a failed resource action (i.e. you see a "Failed Resource Actions" section).

Reading the text surrounding it sometimes gives you a hint that you have an underlying issue that you need to fix. Sometimes you can find more clues about underlying issues from the syslog or dmesg at the time of the failed action. If the failed resource action has an "exitreason", try searching for parts of the text in the syslog and dmesg.

Once you have resolved any outstanding issues (if there were any), clear the failed resource action(s) by running crm resource cleanup RESOURCE, replacing "RESOURCE" with the name of the resource that failed (e.g. 'p_fs_haqm1' or 'haqm1'. N.b. the failed resource action's name will be prefixed by the name of the resource). There may be multiple failed resources, so you will need to issue the command for each of them. Note, if the underlying issues weren't fixed, then the action may fail again, which will be seen by reissuing crm status.

For more information visit https://www.ibm.com/support/knowledgecenter/SSFKSJ_latest/com.ibm.mq.tro.doc/q133450_.htm (remember to "Change version or product" to your MQ version) where you will find a section titled "Pacemaker scenario 2: An RDQM HA queue manager is not running where it should be", which goes into a lot more detail than my answer.

user_112960460

I was finally able to solve the problem, please confirm the information below.

It's a network problem. The quorum is lost and we had to intervene manually to solve the problem.

Here are the actions are taken:

  • drbdadm disconnect mq6 # on all servers
  • crm resource cleanup mq6 # on the primary node
  • drbdadm connect mq6 # on all servers

I was able to solve the problem that way.

user_123937730
Related