When i produce 100KB size messages 45 records to broker at the rate of 4.5MBps with no Consumer groups it perfectly producing.
When i add 100 consumer groups i cant able to produce at 4.5MBps.
When i reading articles and blogs i found that default Socket Buffer size is 100KB, and for high network bandwidth we have to increase it to get the throughput.
So i increased it from 100KB to 1MB (Socket_receive_buffer_bytes, Socket_send_buffer_bytes) and my producer throughput increased a bit.
So i increased it to 4MB till now i cant able to produce at 4.5MBps.
is that any other metrics need to change with respect to socket buffer?
How the Socket Buffer Works?????
in Kafka documentation when i connection created the socket buffer is set for that.
Will the message is read from Kafka and put in the socket send buffer by network thread?
Or the Socket send buffer is per consumer?
How to Configure Socket buffer size for no delay consuming and producing in network end?
There network thread is idle most of the time. Response and request queue -No spike there Request queue wait time is very low Then what would be bottle neck here?