I am starting to learn Kafka for enterprise solution purposes.
During my readings, some questions came to my mind:
Are all the partitions on a topic of the same size?
What will happen if a Kafka message is sent on a topic and the topic is overloaded. There is no memory for any new messages on the topic?
Are all the partitions on a topic of the same size?
No, they are usually not all of the same size. Mainly, it depends how you distribute the messages accross partitions. If you are using keys in your messages they will be hashed and used to partition your data. Without a key in a message the data will be assigned round robin accross partitions.
In addition, it also depends on how many messages the producer is wrapping into a batch before sending it to the broker and which compression type you are applying.
However, you can say that all replicas of a TopicPartition have the same size as soon as they are in-sync.
What will happen if a Kafka message is sent on a topic and the topic is overloaded. There is no memory for any new messages on the topic?
A topic itself cannot be overloaded, but rather the entire Kafka broker. If a broker has no free disk space left it will shut itself down and you will in that situation not be able to write any more data into a TopicPartition if the leader was on that broker. However, if only one broker goes down and you have more than one replication configured in your topic those replicas will take over on another broker.
To make sure that this never happens you can apply the volume based retention policies that is available in Kafka if you set your cleanup.policy to delete and set the retention.bytes such that with the number of topics/replicas you will not exceed your storage limit.
