Lenses MQTT Source Connector doesn't send PINGREQ to MQTT Broker when idle in its keep-alive time

user_7118272 0
Viewed 172

PROBLEM -

I have created an MQTT Source Connector using Lenses. The connector works fine till the data is being published on my MQTT Mosquitto broker and works seemlessly. Now, when i stop publishing the data and there is no data sent to the mqtt source connector, after about 4-5 mins , if i start publishing the data again , the data doesn't come in my source connector even though the connector is still in running state. For resolving this i need to restart my connector everytime which is bad.

METHODS I HAVE ALREADY TRIED -

  1. Even though the client id is unique, i still changed it everytime to see if that was the issue. But it didn't work.

  2. Tried increasing the keepalive interval to 10 mins but that didn't work too.

  3. There were no error logs in kafka connect logs for this for a long time, but once in 15-20 tries i received this Socket Connection Error

UPDATE

Upon digging more into the issue, I found out that my source connector (behaving as an mqtt client) was not sending any PINGREQ packets to my Mosquitto MQTT broker when it was idle in its keep-alive time interval while other client conencted to my mqtt broker were sending their PINGREQ packets when they were idle and hence the connection between the source connector and the mqtt broker was getting dropped.

Do I need to specify any property explicitly in my MQTT Source Connector properties file to send a PINGREQ packet to the MQTT broker in the keep-alive time or does the connector handles that itself ?

mqtt apache-kafka-connect apache-kafka
1 Answers

After a long research , I found out the default keep-alive time which Lenses connector was using (5000) was in seconds. 5000 seconds was way too big, so the mqtt-broker was disconnecting the client even before the first PINGREQ was sent by the client in its keep-alive time. I reduced the keep-alive time to 5 seconds by adding the line connect.mqtt.keep.alive=5 in my connector properties file. This resolved the issue.

user_71182720
Related