Why single Broker setup performs better with single topic partition rather than multiple partitions

user_7241513 0
Viewed 175

We are exploring Kafka for coordination across multiple tasks in a Spark job. Each Spark task acts as both a producer AND consumer of messages on the SAME topic. So far we are seeing decent performance, but I am wondering if there is a way to improve it, considering that we are getting the best performance by doing things CONTRARY to what the docs suggest. At the moment we use only a single Broker machine with multiple CPUs, but we can use more if needed.

So far we have tried the following setups:

  1. Single topic, single partition, multiple consumers, NOT using Group ID: BEST PERFORMANCE
  2. Single topic, single partition, multiple consumers each using its own Group ID: 2x slower than (1)
  3. Single topic, single partition, multiple consumers, all using the same Group ID: stuck or dead slow
  4. Single topic, as many partitions as consumers, single Group ID: stuck or dead slow
  5. Single topic, as many partitions as consumers, each using its own Group ID or no Group ID: works, but a lot slower than (1) or (2)

I don't understand why we are getting best performance by doing things against what the docs suggest.

My questions are:

  1. There's a lot written out there about the benefits of having multiple partitions, even on a single broker, but clearly here we are seeing performance deterioration.
  2. Apart from resilience considerations, what's the benefit of adding additional Brokers? We see that our single Broker CPU utilization never goes above 50% even in times of stress. And its easier to simply increase the CPU count on a single VM rather than manage multiple VMs. Is there any merit in getting more Brokers? (for speed considerations, not resilience)
  3. If the above is YES, then clearly we can't have a broker per each consumer. Right now we are running 30-60 Spark tasks, but it can go up to hundreds. So almost inevitably we will be in a situation that each Broker is responsible for tens of partitions, if each task were to have a partition. So based on the above tests, we are still going to see worse performance?

Note that we are setting up the producer to not wait for acknowledgment from the Brokers, as we'd seen in the docs that with many partitions that can slow things down:

producer = KafkaProducer(bootstrap_servers=[SERVER], acks=0)

Thanks for your thoughts.

kafka-consumer-api apache-kafka apache-spark
1 Answers

I think you are missing an important concept: Kafka allows only one consumer per topic partition while there may be multiple consumer groups reading from the same partition. It seems that you have a problem with committing the offsets or too many group re-balancing problems.

Here are my thoughts;

  1. Single topic, single partition, multiple consumers, NOT using Group ID: BEST PERFORMANCE

What actually happens here is -> one of your consumers is idle.

  1. Single topic, single partition, multiple consumers each using its own Group ID: 2x slower than (1)

Both consumers are fetching and processing the same messages independently.

  1. Single topic, single partition, multiple consumers, all using the same Group ID: stuck or dead slow

Only one member of the same group can read from a single partition. This should not give results different than the first case.

  1. Single topic, as many partitions as consumers, single Group ID: stuck or dead slow

This is the situation where each consumer is assigned to different partitions. And, this is the case where we expect to consume as fast as we are.

Single topic, as many partitions as consumers, each using its own Group ID or no Group ID: works, but a lot slower than (1) or (2)

Same remarks on the first and second step.

There's a lot written out there about the benefits of having multiple partitions, even on a single broker, but clearly here we are seeing performance deterioration.

Indeed, by having multiple partitions, we can parallelize the consumers. If the consumers have the same group id, then they will consume from different partitions. Otherwise, each consumer will consume from all partitions.

Apart from resilience considerations, what's the benefit of adding additional Brokers? We see that our single Broker CPU utilization never goes above 50% even in times of stress. And its easier to simply increase the CPU count on a single VM rather than manage multiple VMs. Is there any merit in getting more Brokers? (for speed considerations, not resilience) If the above is YES, then clearly we can't have a broker per each consumer. Right now we are running 30-60 Spark tasks, but it can go up to hundreds. So almost inevitably we will be in a situation that each Broker is responsible for tens of partitions, if each task were to have a partition. So based on the above tests, we are still going to see worse performance?

When a new topic is created, one of the brokers in the cluster is selected as partition leader, where all read/write operations are handled. So, when you have many topics, it will automatically distribute the workload between the brokers. If you have a single broker with many topics, all producers/consumers will be producing/consume from/to the same broker.

user_11561110
Related