How to understand a role of a queue in a distributed system?

user_1947415 0
Viewed 510

I am trying to understand what is the use case of a queue in distributed system.
And also how it scales and how it makes sure it's not a single point of failure in the system?

Any direct answer or a reference to a document is appreciated.

  1. Use case:
    I understand that queue is a messaging system. And it decouples the systems that communicate between each other. But, is that the only point of using a queue?

  2. Scalability:
    How does the queue scale for high volumes of data? Both read and write.

  3. Reliability: How does the queue not becoming a single point of failure in the system? Does the queue do a replication, similar to data-storage?

My question is not specified to any particular queue server like Kafka or JMS. Just in general.

distributed-system message-queue
2 Answers

my take:

  1. I would be careful with "decouple" term - if service A calls api on service B, there is coupling since there is a contract between services; this is true even if the communication is happening over a queue, file or fax. The key with queues is that the communication between services is asynchronous. Which means their runtimes are decoupled - from practical point of view, either of systems may go down without affecting the other.

  2. Queues can scale for large volumes of data by partitioning. From clients point of view, there is one queue, but in reality there are many queues/shards and number of shards helps to support more data. Of course sharding a queue is not "free" - you will lose global ordering of events, which may need to be addressed in you application.

  3. A good queue based solution is reliable based on replication/consensus/etc - depends on set of desired properties. Queues are not very different from databases in this regard.

To give you more direction to dig into:

  • there an interesting feature of queues: deliver-exactly-once, deliver-at-most-once, etc
  • may I recommend Enterprise Architecture Patterns - https://www.enterpriseintegrationpatterns.com/patterns/messaging/Messaging.html this is a good "system design" level of information
  • queues may participate in distributed transactions, e.g. you could build something like delete a record from database and write it into queue, and that will be either done/committed or rolledback - another interesting topic to explore
user_69976990
Related