That is pretty old issue, but decided to share it anyway, maybe someone is still facing that.
Tests done on a vanilla 3 node cluster. Decent CPU, 64G RAM, SSD drive.
When running kafka-producer-perf-test.sh like below:
$ ./kafka-producer-perf-test.sh --topic test_topic --throughput 30000 --num-records 3000000 --record-size 1024 --producer-props acks=all bootstrap.servers=server1:9092,server2:9092,server3:9092
on kafka 2.3.1 I can see the following results (summary line of the above command):
3000000 records sent, 29996.700363 records/sec (29.29 MB/sec), 8.52 ms avg latency, 292.00 ms max latency, 5 ms 50th, 9 ms 95th, 86 ms 99th, 229 ms 99.9th.
where I focus on the average latency, which is 8.52 ms here.
Now, repeating the same test on kafka 2.5.1 gives much higher scores
3000000 records sent, 27373.261798 records/sec (26.73 MB/sec), 962.79 ms avg latency, 3993.00 ms max latency, 6 ms 50th, 10 ms 95th, 128 ms 99th, 251 ms 99.9th.
So the average latency increased from 8.52 ms to 962.79 ms which is something that I would not expected to see.