RabbitMQ poor performance

user_1300730 0
Viewed 3924

We are facing bad performance in our RabbitMQ clusters. Even when idle.

Once installed the rabbitmq-top plugin, we see many processes with very high reductions/sec. 100k and more!

Questions:

  • What does it mean?
  • How to control it?
  • What might be causing such slowness without any errors?

Notes:

  • Our clusters are running on Kubernetes 1.15.11
  • We allocated 3 nodes, each with 8 CPU and 8 GB limits. Set vm_watermark to 7G. Actual usage is ~1.5 CPU and 1 GB RAM
  • RabbitMQ 3.8.2. Erlang 22.1
  • We don't have many consumers and producers. The slowness is also on a fairly idle environment
  • The rabbitmqctl status is very slow to return details (sometimes 2 minutes) but does not show any errors
rabbitmqctl rabbitmq erlang
1 Answers

After some more investigation, we found the actual reason was made up of two issues.

  1. RabbitMQ (Erlang) run time configuration by default (using the bitnami helm chart) assigns only a single scheduler. This is good for some simple app with a few concurrent connections. Production grade with 1000s of connections have to use many more schedulers. Bumping up from 1 to 8 schedulers improved throughput dramatically.
  2. Our monitoring that was hammering RabbitMQ with a lot of requests per seconds (about 100/sec). The monitoring hits the aliveness-test, which creates a connection, declares a queue (not mirrored), publishes a message and then consumes that message. Disabling the monitoring reduced load dramatically. 80%-90% drop in CPU usage and the reductions/sec also dropped by about 90%.

References

Performance:

Monitoring:

user_13007300
Related