I am quite new to Kafka, and have a question with regards to the relationship/mapping between Producer, Topic, Broker and Partition in the case where I have a single Producer, single Topic and a single Broker, does it make sense to create multiple partitions for the Topic here on the single Broker here? If yes, how does this help in terms of parallelism/performance?
Thanks.
Even if you have a single Producer, single Topic and a single Broker, it makes sense to create multiple partitions for the Topic in terms of parallelism/performance in the context of consumers. If you have multiple consumers in a single consumer group and multiple partitions in the topic, then it is guaranteed that consumers will receive data from different partitions which will give you parallelism and performance boost while processing from kafka.
First thing to understand is that a topic partition is a unit of parallelism in Kafka Cluster. On both Producer and Broker, the writes are happening in parallel so that you can perform expensive operations (compression etc), and at the consumer end each partition data is given to a single consumer thread.
In your scenario you would be benefited if you are having multiple partitions on a topic and these multiple partitions being consumed by multiple consumers within a single consumer group. That way you can achieve maximum throughput in your application. If you only use a single consumer thread for multiple partitions it would be of no use. Basically More partitions could lead to Higher throughput if you manage your cluster resources cleverly.
In addition to the previous answers it is important to remember that consuming from multiple partitions does not preserve the order of the messages/events. You might have to consider this fact if your application depends on the correct order of messages.
