I have a single instance of kafka installed on a VM with 8 cores and 32GB RAM.
I write to it (produce) from 10 different machines and consume from one machine, all of which are in the same network.
The size of the data that I produce is ~ 35MBit / s.
For some reason, most of the time I can't consume more than ~ 10MBit /s (for limited periods of time I do manage to consume all the produced data), even though the kafka AND the consumer servers are mostly idle (therefore I don't think it's a retention problem).
Could kafka be ignoring some of the produced data?
Some parameter values that might be useful for analysis:
num.network.threads=32
num.io.threads=16
message.max.bytes=2147483647
num.partitions=10
log.retention.ms=120000 (2 minutes)