MQTT Transmission History Store - Rolling History Buffer

When a Transmitter is configured to use an MQTT Transmission History Store, data is written to the History Store once MQTT Transmission has determined that there is a connection failure which can be up to 1.5 times the configured keep alive. As such any data published during this time can be lost.

In order to cover data loss during a keep alive timeout scenario, the MQTT Transmission History Store includes a Rolling History Buffer that can be configured in the Advanced Properties configuration section. When the Rolling History Buffer is enabled, all tag changes will be written to the History Store regardless of connection status.

• Enable Rolling History Buffer
  • Enable the Rolling History Buffer to store data in a rolling buffer to cover data loss in a keep alive timeout scenario
• Rolling History Prune Interval
  • Interval at which to attempt to prune the rolling history buffer in seconds
• Rolling History Max Age
  • The maximum age allowed of the data in the rolling history buffer in seconds
• Rolling History Prune Quantity
  • The quantity of the oldest data over the Max Age to prune on each prune interval.

The ideal settings for your Rolling Buffer configuration will be dependent on the rate that your tags are changing. For example, consider these settings:

• Rolling history max age - 60s (2x the 30s keep alive timeout)

• Rolling history prune interval - 60s

• Rolling history prune quantity - 10

The rolling history prune interval is how frequently it checks to see if there is any data that’s older than the max age (60s). If there is, in this example, the prune quantity (10 metrics) will be pruned from the buffer for every prune interval (60s). If there’s 100 metrics in the buffer that are older than 60s, after pruning once, only the oldest 10 will be dropped and there will still be 90 old metrics there.

On a reconnect, these older metrics will be published since they’re still in the buffer. Having a lot of old tags in the buffer will increase the number of duplicates that you get. Keeping a proper balance between the tag change rate and these settings can be important for minimizing duplicates.

From 4.0.17 onward, you can find your tag changes per second by checking Enable Tag Tracking tag and watching the Tags Per Second tag under your Transmitter Info > Transmitters > TransmitterName in the MQTT Transmission tag provider. More details about that can be found in this doc.