MQTT retained messages are the system’s way of ensuring new subscribers always get the last known state of a topic, even if they weren’t online when that state was last published.
Let’s see it in action. Imagine you have a smart thermostat.
Scenario:
- Thermostat (Publisher): Publishes its current temperature to
home/livingroom/thermostat/temperaturewith the "retain" flag set.mosquitto_pub -h mqtt.example.com -t "home/livingroom/thermostat/temperature" -m "22.5" -r
- Dashboard (Subscriber 1): Is already subscribed to
home/livingroom/thermostat/temperature. It receives "22.5" immediately.mosquitto_sub -h mqtt.example.com -t "home/livingroom/thermostat/temperature"
- Thermostat (Publisher): The temperature changes to 23.0. It publishes again, retaining the message.
mosquitto_pub -h mqtt.example.com -t "home/livingroom/thermostat/temperature" -m "23.0" -r
- Dashboard (Subscriber 1): Receives "23.0".
- Dashboard (Subscriber 2): Comes online after the last publish and subscribes to
home/livingroom/thermostat/temperature.mosquitto_sub -h mqtt.example.com -t "home/livingroom/thermostat/temperature"
Crucially, Subscriber 2 doesn’t have to wait for the thermostat to publish again. The MQTT broker (like Mosquitto) stores the last retained message for that topic. When Subscriber 2 connects and subscribes, the broker immediately sends "23.0" to it.
This solves the "cold start" problem for clients. Without retained messages, a new subscriber would get nothing until the next time the actual device published its state. For devices that don’t publish constantly (e.g., sensors only reporting on change), this would mean new clients always start with stale or no data. Retained messages provide an instant, up-to-date snapshot.
How it Works Internally:
When a publisher sends a message with the retain flag set to 1 (true) to a topic, the MQTT broker:
- Stores the message: It keeps a copy of this message, including its payload, QoS level, and topic name.
- Replaces existing retained message: If there was already a retained message for that topic, it’s overwritten by the new one.
- Delivers to new subscribers: Any client that subscribes to that topic after the retained message was set will receive that stored message immediately upon subscription.
- Does NOT deliver to existing subscribers (by default): Existing subscribers that are already connected and subscribed won’t receive the retained message again unless they explicitly unsubscribe and re-subscribe, or if the broker is configured to re-send retained messages on re-connection (which is an advanced, often undesirable, setting).
The broker maintains a separate list of retained messages for each client connection. When a client connects, it tells the broker which topics it’s interested in. The broker then checks its retained message store: if there’s a retained message for a subscribed topic, it sends it to the client.
Configuration and Control:
- Publishing: The
retainflag is set by the publisher. Inmosquitto_pub, it’s-r. In many libraries, it’s a boolean parameter likeretain=True. - Clearing a Retained Message: To effectively "clear" a retained message, you publish an empty payload to the topic with the retain flag set. This tells the broker to remove the stored message for that topic.
mosquitto_pub -h mqtt.example.com -t "home/livingroom/thermostat/temperature" -m "" -r
- Broker Configuration: Brokers like Mosquitto have settings to control retained message behavior. For example,
max_inflight_messagesandmax_queued_messagescan indirectly affect how many retained messages can be held, andmax_retained_messageslimits the total number of topics that can have retained messages.- In
mosquitto.conf:
This limits the broker to storing a maximum of 1000 distinct retained messages across all topics.max_retained_messages 1000
- In
The "last will and testament" (LWT) message is another special type of message that uses the retain flag. When a client connects, it can specify an LWT message and topic. If the client disconnects uncleanly (e.g., power loss), the broker will publish the LWT message to the specified topic, and it will be published with the retain flag set. This allows other clients to know that a specific device has gone offline and what its last known state was.
The nuance that often trips people up is how retained messages interact with QoS levels. A retained message will be stored by the broker at the QoS level it was published with. However, when a new subscriber connects, the broker will attempt to deliver that retained message to them at the subscriber’s requested QoS level, up to the QoS level the retained message was originally published at. So, if a message was published with QoS 1 and retained, and a new subscriber requests QoS 0, they get it at QoS 0. If the new subscriber requests QoS 2, they will only get it at QoS 1 (the maximum original QoS).
Understanding how to use retained messages effectively is key to building robust IoT systems where devices might come and go, and new clients need immediate, relevant data.
The next hurdle is understanding how to manage retained messages in a large-scale deployment, particularly when dealing with dynamic topic structures or devices that might go offline permanently.