Prerequisites
Summary
The Inductive Auotmation platform and MQTT modules can be resilient to failures when configured to use redundancy. Redundant Ignition systems can be set up and configured to act as failover backups for primary/master Ignition instances. This tutorial will provide step by step instructions for installing a set of Ignition systems with redundancy on the host/primary Ignition instance as well as redundancy on the MQTT enabled edge nodes. For this tutorial we will show how to set up a total of six Ignition systems. These will be:
- Ignition Primary
- An Ignition system running as master with MQTT Distributor and MQTT Engine installed. This is what remote Ignition systems will send data to in normal operation.
- Ignition Primary Backup
- An Ignition system running as backup with MQTT Distributor and MQTT Engine installed. This is what remote Ignition systems will send data to when Ignition Primary fails.
- Ignition Edge 1
- An Ignition system running as master with MQTT Transmission installed. This will send data to Ignition Primary in normal operation. If Ignition Primary is in a failed state, this will send data to Ignition Primary Backup.
- Ignition Edge 1 Backup
- An Ignition system running as backup with MQTT Transmission installed. This will send data to Ignition Primary in the event that Ignition Edge 1 fails. If Ignition Primary is in a failed state and Ignition Edge 1 is in a failed state, this will send data to Ignition Primary Backup.
- Ignition Edge 2
- An Ignition system running as master with MQTT Transmission installed. This will send data to Ignition Primary in normal operation. If Ignition Primary is in a failed state, this will send data to Ignition Primary Backup.
- Ignition Edge 2 Backup
- An Ignition system running as backup with MQTT Transmission installed. This will send data to Ignition Primary in the event that Ignition Edge 2 fails. If Ignition Primary is in a failed state and Ignition Edge 2 is in a failed state, this will send data to Ignition Primary Backup.
Additional Edge Nodes could be added to this infrastructure. It is also important to note that the Ignition Edge Nodes with MQTT Transmission could also be instances of Ignition Edge MQTT depending on your requirements (https://inductiveautomation.com/whats-new-ignition-edge). There are additional considerations when setting up a real world system using redundancy. These topics are not covered in this tutorial but should be taken into consideration.
- Network paths
- It may make sense to have edge nodes support multiple network paths to the MQTT servers. For example, Ethernet, cellular, and satellite could all exist as supported network paths on a single Edge gateway. This will help ensure additional reliability by supporting failover of networks.
- Primary Ignition and Primary Ignition backup placement
- This tutorial was created by modeling this exact environment using Amazon AWS EC2 instances in the cloud. Reliability could be improved by putting Ignition Primary and Ignition Primary Backup in different AWS availability zones or even different AWS regions. This would allow the primary Ignition with MQTT Distributor and MQTT Engine to continue to operate even in the case of AWS failures.
- MQTT Servers
- Additional MQTT Servers can be added so MQTT connections from remote edge nodes remain established. Additional Chariot MQTT Servers can be used to make the system more robust (http://www.cirrus-link.com/mqtt-servers/).
- History enablement in MQTT Transmission
- MQTT Transmission supports caching of data in the case that it can not establish a connection to any of the configured MQTT Servers. Once a connection is reestablished, it will begin reporting and flush the stored historical values to prevent data loss in catastrophic failures.
Upon completion of this tutorial you will have a functional system with redundancy/failover support for both remote edge nodes as well as the primary Ignition system that the remote edge nodes are reporting to.
Architecture
Tutorial
Step 1: Download and Install Ignition on Six Systems
Ignition is an Industrial Application Platform that can be used to create SCADA and HMI solutions. A fully functional Ignition system can be downloaded and run in trial mode.
Go to the Inductive Automation download page and download the desired Ignition installer for Windows, Linux or MacOS;
https://inductiveautomation.com/downloads/ignition
Once the Ignition installer has been downloaded, follow the instructions provided by Inductive Automation to install and startup Ignition.
(Note: For this test infrastructure, MQTT Distributor will be installed as an Ignition module on both the Primary Ignition Gateway as well as the Primary Ignition Gateway Backup. Make sure to either turn off firewalls or at a minimum allow inbound connections to TCP/IP port #1883 and port #8883, as remote MQTT Clients will need to be able to establish a TCP/IP socket connection to these ports).
Step 2: Download and Install the Cirrus Link MQTT Modules
Go to the Inductive Automation download page again and scroll down to the Third Party modules section. Find the Cirrus Link modules section and download the MQTT Distributor, MQTT Engine, MQTT Transmission modules.
https://inductiveautomation.com/downloads/ignition. For each of the Ignition instance, install the following MQTT Modules.
- Ignition Primary
- MQTT Distributor and MQTT Engine
- Ignition Primary Backup
- MQTT Distributor and MQTT Engine
- Ignition Edge 1
- Ignition Edge 1 Backup
- Ignition Edge 2
- Ignition Edge 2 Backup
Step 3: Configure the MQTT Modules
Once Ignition is installed, the MQTT Modules are installed, and everything is running we can configure the systems. Since we are going to have a backup for each master system, we only need to do most of the configuration for the master systems. Later, we can sync the configurations to the backups automatically. We'll start by configuring the modules and configure the redundancy settings in the next step.
- Ignition Primary - MQTT Distributor
- No modifications to the default parameters are required. However, it is important to make sure the Operation System allows inbound connections on port 1883 and there are no firewalls blocking inbound connections on this port from the remote edge nodes.
- Ignition Primary - MQTT Engine
- The only change from defaults is to set a Primary Host ID. MQTT uses Quality of Service (QOS) levels to ensure messages get delivered. However, this only ensures delivery between a single MQTT client and the MQTT server. In other words, it doesn't ensure delivery from one MQTT client to another MQTT client. Sparkplug introduces the notion of a Primary Host ID which is used to ensure client to client communications. The only requirement is that it match exactly on both the MQTT Engine and MQTT Transmission configurations.
- Ignition Edge 1 and Ignition Edge 2 - MQTT Transmission (Configure the same on both Ignition instances)
- As with the MQTT Engine configuration, the Primary Host ID must be set on the General tab as shown below.
- Delete the existing default MQTT Transmission Server.
- Create a new MQTT Server configuration by clicking the link below.
- Configure MQTT Transmission to point to the Primary Ignition. Configure as shown below making sure to change the URL to reflect your network settings. For example, if you Ignition Primary is at 192.168.1.100 the MQTT Server URL would be: tcp://192.168.1.100:1883. After setting the parameters as shown below. Click the 'Save Changes' button at the bottom.
- Verify the MQTT Server has been created and is shown in the list of MQTT Servers as shown below.
- Repeat the process of creating a MQTT Server but instead point it to the Ignition Primary Backup MQTT Server. These are the parameters to use:
- Name: primary-backup
- URL: tcp://primary-backup:1883
- Change 'primary-backup' in the URL to reflect the network address of the Ignition Primary Backup server.
- Server Type: MQTT Distributor
- Server Set: Default
- Username: admin
- Password: changeme
- When complete, verify both MQTT Servers appear in the list as shown below.
- Finally, make sure to set up the same MQTT Transmission configuration in the Ignition Edge 2 instance.
Step 4: Configure Redundancy
The following configuration shows all of the redundancy settings that were used in setting this environment up using Amazon's AWS EC2 instances (virtual machines). The configuration will vary based on your network configuration. Additional Ignition redundancy resources can be found at the following links:
https://docs.inductiveautomation.com/display/DOC/Setting+Up+Redundancy
https://support.inductiveautomation.com/usermanuals/ignition/index.html?redundancy_settings.htm
- Ignition Primary
- Select Redundancy on the left navigation bar. Then set the Mode to 'Master' and set the Standby Activity left to 'Warm' as shown below.
- Set up the Redundancy Network Settings. The settings here are specific to your network setup. On many LAN configurations none of these changes are required. What is shown below was the configuration for setting up all of these components in Amazon's AWS EC2 instances. The changes were:
- Uncheck 'Auto-detect network interface'
- Set the 'Network Bind Interface' to the public IP address of the Ignition Primary EC2 instance. On a LAN this would be the primary network interface address of the Ignition Primary machine.
- Uncheck the 'Autodetect HTTP Address' tickbox.
- Specify two explicit HTTP addresses for clients to use. These were the public IP addresses of the Ignition Primary and Ignition Primary Backup EC2 instances. On a LAN, these would be the primary network interface addresses of the Ignition Primary and Ignition Primary Backup machines. Also note the HTTP port is 8088 which is the default Ignition HTTP port.
- Set the Master Node Address. Note in the configuration below a hostname is being used. This should be the primary network interface address of the Ignition Primary Gateway.
- Finally, click the 'Save Changes' button.
- Ignition Primary Backup
- Select Redundancy on the left navigation bar. Then set the Mode to 'Backup' and set the Standby Activity left to 'Warm' as shown below.
- Set up the Redundancy Network Settings. The settings here are specific to your network setup. On many LAN configurations none of these changes are required. What is shown below was the configuration for setting up all of these components in Amazon's AWS EC2 instances. The changes were:
- Uncheck 'Auto-detect network interface'
- Set the 'Network Bind Interface' to the public IP address of the Ignition Primary Backup EC2 instance. On a LAN this would be the primary network interface address of the Ignition Primary Backup machine.
- Uncheck the 'Autodetect HTTP Address' tickbox.
- Specify two explicit HTTP addresses for clients to use. These were the public IP addresses of the Ignition Primary and Ignition Primary Backup EC2 instances. On a LAN, these would be the primary network interface addresses of the Ignition Primary and Ignition Primary Backup machines. Also note the HTTP port is 8088 which is the default Ignition HTTP port.
- Set the Master Node Address. Note in the configuration below a hostname is being used. This should be the primary network interface address of the Ignition Primary Gateway.
- Set the Master Node Address. Note in the configuration below a hostname is being used. This should be the primary network interface address of the Ignition Primary Gateway.
- Finally, click the 'Save Changes' button.
- Ignition Edge 1
- Select Redundancy on the left navigation bar. Then set the Mode to 'Master' and set the Standby Activity left to 'Warm' as shown below.
- Set up the Redundancy Network Settings. The settings here are specific to your network setup. On many LAN configurations none of these changes are required. What is shown below was the configuration for setting up all of these components in Amazon's AWS EC2 instances. The changes were:
- Uncheck 'Auto-detect network interface'
- Set the 'Network Bind Interface' to the public IP address of the Ignition Edge 1 EC2 instance. On a LAN this would be the primary network interface address of the Ignition Edge 1 machine.
- Uncheck the 'Autodetect HTTP Address' tickbox.
- Specify two explicit HTTP addresses for clients to use. These were the public IP addresses of the Ignition Edge 1 and Ignition Edge 1 Backup EC2 instances. On a LAN, these would be the primary network interface addresses of the Ignition Edge 1 and Ignition Edge 1 Backup machines. Also note the HTTP port is 8088 which is the default Ignition HTTP port.
- Set the Master Node Address. Note in the configuration below a hostname is being used. This should be the primary network interface address of the Ignition Edge 1 Gateway.
- Finally, click the 'Save Changes' button.
- Ignition Edge 1 Backup
- Select Redundancy on the left navigation bar. Then set the Mode to 'Backup' and set the Standby Activity left to 'Warm' as shown below.
- Set up the Redundancy Network Settings. The settings here are specific to your network setup. On many LAN configurations none of these changes are required. What is shown below was the configuration for setting up all of these components in Amazon's AWS EC2 instances. The changes were:
- Uncheck 'Auto-detect network interface'
- Set the 'Network Bind Interface' to the public IP address of the Ignition Edge 1 Backup EC2 instance. On a LAN this would be the primary network interface address of the Ignition Edge 1 Backup machine.
- Uncheck the 'Autodetect HTTP Address' tickbox.
- Specify two explicit HTTP addresses for clients to use. These were the public IP addresses of the Ignition Edge 1 and Ignition Edge 1 Backup EC2 instances. On a LAN, these would be the primary network interface addresses of the Ignition Edge 1 and Ignition Edge 1 Backup machines. Also note the HTTP port is 8088 which is the default Ignition HTTP port.
- Set the Master Node Address. Note in the configuration below a hostname is being used. This should be the primary network interface address of the Ignition Primary Gateway.
- Set the Master Node Address. Note in the configuration below a hostname is being used. This should be the primary network interface address of the Ignition Primary Gateway.
- Finally, click the 'Save Changes' button.
- Ignition Edge 2
- Select Redundancy on the left navigation bar. Then set the Mode to 'Master' and set the Standby Activity left to 'Warm' as shown below.
- Set up the Redundancy Network Settings. The settings here are specific to your network setup. On many LAN configurations none of these changes are required. What is shown below was the configuration for setting up all of these components in Amazon's AWS EC2 instances. The changes were:
- Uncheck 'Auto-detect network interface'
- Set the 'Network Bind Interface' to the public IP address of the Ignition Edge 1 EC2 instance. On a LAN this would be the primary network interface address of the Ignition Edge 1 machine.
- Uncheck the 'Autodetect HTTP Address' tickbox.
- Specify two explicit HTTP addresses for clients to use. These were the public IP addresses of the Ignition Edge 1 and Ignition Edge 1 Backup EC2 instances. On a LAN, these would be the primary network interface addresses of the Ignition Edge 1 and Ignition Edge 1 Backup machines. Also note the HTTP port is 8088 which is the default Ignition HTTP port.
- Set the Master Node Address. Note in the configuration below a hostname is being used. This should be the primary network interface address of the Ignition Edge 1 Gateway.
- Finally, click the 'Save Changes' button.
Ignition Edge 1 Backup
- Select Redundancy on the left navigation bar. Then set the Mode to 'Backup' and set the Standby Activity left to 'Warm' as shown below.
- Set up the Redundancy Network Settings. The settings here are specific to your network setup. On many LAN configurations none of these changes are required. What is shown below was the configuration for setting up all of these components in Amazon's AWS EC2 instances. The changes were:
- Uncheck 'Auto-detect network interface'
- Set the 'Network Bind Interface' to the public IP address of the Ignition Edge 1 Backup EC2 instance. On a LAN this would be the primary network interface address of the Ignition Edge 1 Backup machine.
- Uncheck the 'Autodetect HTTP Address' tickbox.
- Specify two explicit HTTP addresses for clients to use. These were the public IP addresses of the Ignition Edge 1 and Ignition Edge 1 Backup EC2 instances. On a LAN, these would be the primary network interface addresses of the Ignition Edge 1 and Ignition Edge 1 Backup machines. Also note the HTTP port is 8088 which is the default Ignition HTTP port.
- Set the Master Node Address. Note in the configuration below a hostname is being used. This should be the primary network interface address of the Ignition Primary Gateway.
- Set the Master Node Address. Note in the configuration below a hostname is being used. This should be the primary network interface address of the Ignition Primary Gateway.
- Finally, click the 'Save Changes' button.