We are using Kafka 2.5.1 version cluster. Recently noticed that one of the topic partition data size is uneven. One specific partition is having 300% more size compare to remaining partitions. This created uneven disk utilization in the cluster.
Verified the consumer lag and looks normal like other partitions Also we are using DefaultPartitioner and "metadata.max.age.ms" config set to default, which is 300000ms(5 mins)
How did we make the partition data evenly distributed?
As an operator, there's nothing that can be done; especially since consumers rely on that partition assigment and message ordering.
Kafka doesn't guarantee non-null keys will be evenly distributed. You need to modify your producer such that the partitioning scheme is evenly distributing the data.
For example, by default, if 80% of your records have the same hash(key), then they all get sent to the same partition, causing that partition to be 4x larger than the other 20% combined. You can write your own partitioner to change that behavior, but if you have clients in multiple languages all producing to the same topic, you'll end up with a lot of duplicate work
To add to the answer by @OneCricketeer, I would suggest to first figure out what are your event keys, and which key is the one creating such a huge dis-balance in your topic... To do that I would:
inspect data in the "big" partition, and in one "normal" partition. You can look at the events in the kafka data log using this command, for example:
/usr/bin/kafka-dump-log --print-data-log --files /apps/kafka-data/tracking.ap.client.traffic.keyed-1/00000000000004735501.log
[of course, substitute with your own kafka logs dir, topic name, partition number and log file name]
You will see something like this:
offset: 4736628 position: 676217 CreateTime: 1622043813 isvalid: true keysize: 8 valuesize: 506 magic: 2 compresscodec: NONE producerId: -1 producerEpoch: -1 sequence: -1 isTransactional: false headerKeys: [] key: 123 payload: {"CID":"123",<your own event content ....>, ...}
The key of the event is the "key: 123" part.
compare key distribution in your two partitions, and try to find the key value that is the culprit in your "big" partition - you could write a simple awk/bash script to count number of events in the log by distinct key values, or it might be pretty obvious right away
now that you know what are the keys and which specific key (or maybe a couple of keys) are the problem - you can start thinking about how to solve the problem. It might be very easy - if there is no hard requirement for events grouping, and you can change the key to be a UUID or some other well distributed field value. Or it might be hard - if it was designed this way for a reason and you have to keep all events for the same key, say CustomerID, in the same partition - if say you are doing some work in your consumer that relies on having all events for the same key to be processed in the same consumer thread (some in-memory aggregations, for example). In that case you may have to re-think your application/consumer logic if it is important to re-distribute the data in the topic evenly (and it usually is)
