Page History

...

• edge client = publishing client

• host client = subscribing/consuming client

Why do we use MQTT QoS 0 for Sparkplug Birth and Data messages? 

...

• How to ensure data is not lost when using QoS 0?

• Sparkplug Sessions and Primary Host ID

• Sparkplug sessions and use of the Primary Host Id ensure the edge client (e.g., MQTT Transmission) is connected to the MQTT server at the same time as the host client (e.g., MQTT Engine), so we know we are good to publish data messages when in an established session.

• MQTT Store and Forward at the edge

• Sparkplug’s use of MQTT Store and Forward at the edge allows the edge client to buffer data locally if the host is offline or the edge connection to the MQTT server is down.

• MQTT Keep Alive

• MQTT client-server connections will fail ungracefully. When this happens the client still has an open socket on its side and is publishing data that will never make it to the MQTT server. MQTT messages will be lost if published after a connection fails ungracefully, but before the client realizes the connection has failed. MQTT provides a mechanism to identify failed connections like this - the MQTT Keep Alive. The MQTT Keep Alive requires is an interval of time (measured in seconds) that the client send a Ping Request to the server every 1.5x the client’s keep alive (typically 30 seconds). The server must respond must not allow to elapse without transmitting a packet to the MQTT Server. In the absence transmitted data during this interval the client will transmit a Ping Request control packet and wait for the server to reply with a Ping Response in 1. 5x the client’s keep alive setting. When either side fails to get the appropriate request/response, it determines the connection has failed and takes the appropriate actionWhen the server fails to receive any transmission from the client within 1.5x the Keep Alive value, it will disconnect the network connection with the client. Similarly if the client does not receive the Ping Response within a reasonable amount of time after sending a Ping Request is should close the network connection with the server. In the case of the Sparkplug edge client, it will typically be configured to store all tag changes locally in the MQTT Store and Forward History Store until the connection to the server is reestablished. To mitigate the possibility of data loss in the MQTT keep alive Keep Alive window, Cirrus Link has implemented a rolling buffer that covers covering 2x the client’s keep alive and it contains Keep Alive, containing all tag changes that happened in that window. If the edge client loses connection to the MQTT server, it will flush the rolling buffer once the connection is restored to replay any potentially lost MQTT messages.

• Sparkplug Message Sequence Numbers

• If any message is lost due to some random network issue, but the connection never fails the keep alive Keep Alive (unlikely), a message would actually be lost. This lost message would be identified by the host client because a message sequence number would be missing.

To summarize, data loss with QoS 0 messages can occur in the MQTT keep alive Keep Alive window when a connection fails ungracefully. To mitigate this risk, Sparkplug uses Primary Host Id, and MQTT Store and Forward at the edge with a rolling buffer*. This will not guarantee zero data loss, but it will provide the highest possibility of delivering high speed OT data at scale with a minimal possibility of data loss. 

...