I have data that is toggling between stale and healthy at my subscribing MQTT Client.

There are two common causes for this issue – colliding MQTT Client IDs or colliding Edge Node IDs.

Colliding MQTT Client IDs occur when there are two or more MQTT clients connecting to an MQTT broker using the same Client ID. The broker uses the Client ID to identify the client and the current state of the client and therefore this ID must be unique per client and broker.

Colliding Edge Node IDs occur when there are two or more Edge Nodes publishing with a topic namespace that does not have a unique combination of Group ID and Edge Node ID. In a Sparkplug compliant system, it is this combination of IDs that identifies the Edge Node.

Let’s start by confirming the connection status of the Edge Nodes with your MQTT Distributor or Chariot broker instance to identify which issue you have.

MQTT Distributor

From the Ignition UI connected to your instance of MQTT Distributor, navigate to Status > Diagnostic > Logs.

Set the debug level for the io.moquette.spi.impl.ProtocolProcessor logger to TRACE and set the filter of the Logs view to ProtocolProcessor.

If we can see in the logs that the MQTT broker is continually forcefully disconnecting an existing connection to allow another client with the same Client ID to connect, as shown below, you have Colliding Client IDs. If not, we have a Colliding Edge Node IDs issue.

Chariot 

From the Chariot UI navigate to the Alerts in the left menu bar. Select Types and enable the alerts for MQTT_DISCONNECT

Under Live Alerts, if we can see in the logs that Chariot is logging the DUPLICATE_CLIENT_ID description, as shown below, you have Colliding Client IDs. If not, we have a Colliding Edge Node IDs issue.

Colliding Client ID

To resolve the colliding Client IDs you will need to review your system configurations on the physical Edge Nodes identified and remove the conflicts.

In the logs if you see different IP addresses for the Edge Nodes attempting to connect with the same Client ID, then the same MQTT Client ID has been set on different physical Edge Nodes. Review the configuration for physical Edge Nodes with these IP addresses. 

If using MQTT Transmission, there are two additional scenarios to consider if the logs show the same IP address for the Edge Nodes attempting to connect with the same Client ID.

1. The MQTT Client ID is set on a single physical Edge Node device where a single Transmitter is dynamically picking up multiple virtual Edge Nodes.
2. The MQTT Client ID is set on a single physical Edge Node where multiple transmitters are configured for one or more virtual Edge Nodes.

In either of these two setups, the MQTT connection for each virtual Edge Node requires a unique Client ID. The Client ID in the the MQTT Transmission Configuration should be left blank allowing MQTT Transmission to auto-generate unique Client IDs for each Edge Node connection.

Colliding Edge Node ID

MQTT Transmission uses the Sparkplug B specification which defines the topic namespace to publish data as spBv1.0/group_ID/message_type/edge_ID/[device_ID]

Within a distributed system, the combination of any GROUP_ID and EDGE_NODE_ID must be unique because these are used as 'addresses' in the system to identify the edge node. It is a bit like having two houses with the same postal address. It isn't possible for other MQTT clients in the system to tell where messages are coming from and when sending messages to them, they will both receive the messages.

The topic used for tags published to the MQTT Server is a combination of the MQTT Transmission 'Transmitter' configuration as well as the arrangement of tags in the Ignition tag tree. In the Transmitter configuration, the Tag Path points to the folder where the tag tree will start and the next three folders will be picked up as the group_ID, edge_ID and device_ID.

If you have not carefully managed your tag tree structure, you can create duplicate group_id and edge_id combinations.

In the video below we have two transmitters with a tag tree of Company/Location/Process Area/Line/PLC where Line is the Edge Node device. With the tag path for the transmitter set to My Company, we can see that both edge nodes will publish on the following namespaces:

spV1.0/Lakeside/message_type/Finished Goods/Line1

spV1.0/Lakeside/message_type/Finished Goods/Line2

Since the group_id (Lakeside) and edge_id (Finished Goods) does not uniquely define the edge node, data from these two transmitters will sent with the same topic resulting in the next message sequence number expected by the MQTT client being incorrect.

As a result, the MQTT Client will mark the data as stale and request a rebirth from the transmitter. Depending on the frequency of the published data this manifests as the data from different edge nodes toggling between stale and healthy. If you have multiple MQTT Clients subscribing to the namespace, this will likely create a firestorm of rebirth requests across the system.