I have an MQTT(Mosquitto) service running in a docker container for a production site(hosted on AWS EC2). The reason for running it within a container is to ensure portability since we use this service in multiple places.
Recently, we have been observing that MQTT over time, starts eating up a lot of resources(CPU and Memory), due to which it turns out that we start losing information that has been transmitted over MQTT via QoS 1. To resolve this issue, the following approaches have been tried out:
- Delete the persistence file that the MQTT broker uses
- Delete the complete MQTT volume
- Restart the container
The above steps seem to work for the time being, but do not completely address the issue. We also went a step further to increase the number of resources available to MQTT(EC2 CPU and Memory), but similar to the above steps, it seems to just push back the problem and not address it.
Could you suggest ways in which we can address this issue? Furthermore, it would be helpful, if anyone can suggest how to profile the MQTT Broker as well, for heap usage, topics, clients, disconnects/connects, etc.
Currently, we use the following MQTT configuration file:
# mount to /mosquitto/config/mosquitto.conf
# sys_interval 60
persistence true
persistence_location /mosquitto/data/
# This causes every change to trigger retained data save
autosave_interval 60
queue_qos0_messages true
store_clean_interval 20
max_queued_messages 100000
max_inflight_messages 10000
allow_anonymous false
password_file /etc/mosquitto/conf.d/passwd
acl_file /etc/mosquitto/conf.d/acl
connection_messages true
log_dest stdout
persistent_client_expiration 1h
Note: The clients that we use to connect to the MQTT Broker, are written in NodeJS and Python. And with all the clients we pass the clean-session option as true.
Running the following command on the Mosquitto server:
mosquitto --help
The response received is:
mosquitto version 1.4.12 (build date 2017-06-01 13:03:46+0000)
mosquitto is an MQTT v3.1.1/v3.1 broker.