What are benefits of using more than 1 partition in Kafka?

user_14316975 0
Viewed 857

How do you make a decision on the need to use more than one partition? Why would you ever put a topic on more than one? Since in most cases order in which messages are produced is important in my opinion there is no reason to make things complex and distribute messages among several partitions and then have to implement complex algorithm to fetch that data.

Please advice on advantages of using more than 1 partition per topic.

Please note I am not asking about scalability, because you can configure a single partition to be replicated into any number of clones. More of a design question, why would anyone store a single topic on more than 1 partition?

kafka-producer-api apache-kafka
2 Answers

Partitions can be helpful for multiple scenarios. One of the main reason why you should have more than one partition in a topic is scalability. Remember, that a partition can be replicated to "any number of clones" but only one partition leader will communicate with your applications.

As you have mentioned, ordering guarantees are often essential for business use cases and it is correct that the ordering can only be ensured within a partition. However, usually the data needs to be ordered by a business entity like a customer. So you would design your topic with multiple partitions but ensuring that all data for a particular customer will always be stored in the same partition. This can be achieved with a customized Partitioner.

You can of course argue that all each partition could be designed as an individual topic with only one partition. Topics are a bit like tables in a DWH where all data share the same schema, so you would collect all data with the same schema (belonging to the same business entity) into one topic. Then again, having more than one partition can increase your throughput.

user_122089100

The first thing to note, is that the order of messages is often not important, or a partial ordering is sufficient. One example would be to push the order state changes of a webshop into a topic. For most processing you only need to have the messages belonging to one order to be ordered, you can parallelly process different orders runnnig at the same time.

Now to the scalablity: as you know, you can have as many replicas for a partiton as you want, but what does that do? It only helps your read performance. For writes, that single parition have a single leader, and only that leader can write to the partition. Also, it has to write the data to all the replicas further loading the leader. Another scalablity issue is that a partiton is tied to a single node. That means that if you have a single partition, you have to have enough storage on your node to accomodate all your data.

These are the main reasons that come to my mind, but there might well be other things.

user_119534650
Related