In my case, my producer uses librdkafka, topic.metadata.refresh.interval.ms is by default 5mins.
On broker side connections.max.idle.ms is 2 hours, we have pretty low qps, but We want 99.9% of request within rtt 20ms, but if the broker close the connection, the producer will re-establish the channel and send the message, which will get a high latency. It seems the metadata refresh won't work.
How do I address this problem?
For each metadata refresh librdkafka will pick a suitable broker connection to send the request on, it will not send on all broker connections and it will not send on a truly random connection, so the topic metadata refresh can't be used to keep all connections alive.
The Kafka protocol does not really have a cheap or side-effect free way of "heartbeating" (not to be confused with the HeartbeatRequest) an idle connection.
With broker >=v0.11 it is possible to send a Metadata request for the cluster information only (not requesting any topics, thus cheap). I suggest creating a feature request github issue for librdkafka if this is something you want.
But if you don't want idle connections reaped, why not set connections.max.idle.ms really high?
