Distribute and Manage your SW using the IoT paradigm
How to simplify SW deployment in distributed environments
Software distribution is getting more challenging as software grows in complexity and security requirements become stricter every year. Especially in an industrial setting in which SW components are deployed and managed in many isolated, geo-distributed physical sites, it is complex to automate and secure SW distribution. There are many solutions out there, each designed for a specific scenario with specific assumptions. Our challenge is to identify the right tools and the right approach for the task. This article suggests adopting the IIoT (Industrial IoT) paradigm for solving the problem of complex software distribution in an industrial setting. I will demonstrate that there are many commonalities between the management of modern micro-services solutions that are running inside an isolated environment and typical IIoT scenarios. Therefore, in many situations, it will be beneficial to adopt the IIoT design approach and use IoT frameworks for distributing SW. The approach presented below is not a silver bullet. For some complex use cases such as large Kubernetes clusters deployed on on-premises data centers, the IoT paradigm might not fit, yet for many simpler situations adopting IoT design patterns will be a quick win.
From simple devices to independent edge
IoT frameworks typically involve many small devices that the framework must control. This is the responsibility of the Device Management (DM) module of every commercial IoT framework. Each device has some metadata, such as state and configuration that DM maintains on a large scale. The DM module is also responsible for onboarding new devices and upgrading their SW while maintaining the highest security standards. In legacy IoT settings the SW (a.k.a. firmware as it’s usually embedded in HW) is small, sometimes without even an OS. In such cases, the software is packaged as a complete image tailored for the device, and the software upgrade process just downloads the whole image and reboots the device. In modern IIoT, more responsibilities are delegated to the “edge”. Today it is common to see devices that run software which executes complicated calculations such as Machine Learning (ML) models or statistical analytics. We see a trend where the edge becomes increasingly independent, which results in quicker response times and much less bandwidth. This trend also means that the devices run more complex software, so instead of installing small images on devices, we see now devices running docker images similar to the backend. The fact that the edge becomes more independent does not change the scale of management. IoT DM must maintain metadata for every device no matter how smart it is. It must ensure secure provisioning and updates behind firewalls and disconnected networks. That makes it interesting as a software distribution mechanism.
Industrial SW management challenges
Modern Micro-Services based SW introduces challenges when it comes to SW distribution because every service (by design) is an independent entity. With many possible combinations of HW and SW versions, multiplied by many physically isolated geo-distributed sites, SW Configuration Management (CM) becomes a huge topic. Obviously, I cannot describe it in full here, but for the purpose of this paper, I would like to highlight a few basic challenges:
- Visibility: The system’s owner needs to know the details and state of each SW component installed. With many components, installations, sites, HW and SW versions, etc., that becomes be a complicated task.
- Networking: Industrial settings are typically isolated from the internet with segregated private networks and several layers of firewalls. With that, we must assume that it is impossible to open a port and listen for external notifications, even from a trusted control system. Yet, we need to be able to push SW updates from a central location.
- Identity: Industrial Automation is key for any industrial digital transformation. We should assume that there is no human involvement in almost any process, thus identity management which by legacy is designed for human identities should now be implemented to handle automatic entities. Every device must have an identity with permissions and security policies. In all communications between the site and the cloud (such as an SW update process), both parties should be confident about the identity of the other party.
Using IoT paradigm
By considering every managed on-premises SW component as a “device” or a “module” within a “device”, we can leverage IoT frameworks to fully manage these components from a central location. Not all IoT frameworks available today support containerized software, but many do as the “Intelligent Edge” becomes increasingly popular. As an example, I will use the Azure IoT Edge framework to demonstrate the different capabilities that are required to centralize Software deployment and configuration management.
Secure Bidirectional Communication
Maybe the most basic capability of any IoT solution is to be able to communicate with devices both upstream (device to cloud) for sending telemetries from the device to the cloud, and downstream (cloud to device) for sending commands from the cloud to the device. There are several communication protocols that IoT solutions use to achieve this, and the choice is typically made based on the computation power of the device, and the level of connectivity required. LWM2M, MQTT, AMQP, and HTTPS are only a few examples of the communication protocols being used by IoT frameworks. The details and properties of each protocol and how it achieves bidirectional secure communication are beyond the scope of this paper, but the immediate result is that if we choose to use an IoT solution as a framework for our SW management we can assume secure bidirectional communication across firewalls and isolated networks.
Centralized view
Another paramount capability of IoT frameworks is the ability to monitor devices from a central location at the scale of millions. IoT device management (DM) provides the means to collect, store, query, and visualize device metadata and state. In Azure IoT, Device Twins and Module twins are the core data element for, metadata, configurations, and conditions. With that, we can store and monitor the version and state of each SW component installed. Azure IoT provides visualization tools and integrations such as the integration with Azure Monitor that enables a centralized view and alerts based on the data captured in the twins. Azure Digital twin is another Azure service that enables semantic processing of the information captured in the twins. Such semantic processing can capture relationships between SW components such as compatibility and configurations validity. The scale and complexity of device management required in typical IoT scenarios is huge, thus using an IoT infrastructure for SW distribution provides all the management capabilities required.
Automated SW Deployment at scale.
IoT solutions assume an exceptionally large number of devices. As such they are designed for automation and integration. When you push an update, you must identify the right devices from millions of deployed devices and then push the right update to all the devices that qualify. Such automation is a necessity in industrial settings, because like in IoT scenarios industrial settings also include an exceptionally large number of deployed components. Azure IoT Edge supports deployment at scale both from the Azure portal as well as from CLI and DevOps tools. You can define deployment templates as code that automatically identifies the targets and push the right piece of SW.
Security at the core
Malicious software injection is one of the most popular attack vectors against industrial software. There are many types of mitigations against such attacks. At the core there are a few principles to follow:
- Attestation: Ensures the integrity of software bits for example by using digital signatures to validate the source, the authenticity, and integrity of the SW package to install.
- Authentication: Use mutual strong authentication between the backend and the device so both parties can trust each other.
- Authorization: Apply the least privilege principle so components of a system should have access only to the minimum set of resources and data needed to perform their roles.
- Secure provisioning and key management: Use the tools & protocols provided by the IoT framework to register the new device and store its secrets.
- Secure Communication: Use only a certificate-based secure communication channel based on industry-proven security protocols.
- Audit every SW update and management operation.
All these principles are at the core of device provisioning and management of all the leading IoT solutions. Thus, adopting the IoT paradigm for SW distribution in an industrial setting is an immediate security win.
Containers at the core
The right combination between SW and HW is key both in IIoT and industrial settings. Thus, the SW distribution infrastructure must support any hardware and any operating system or in other words: run containers. It is not a surprise that Azure IoT Edge modules are implemented as Docker-compatible containers. Docker offers many options for creating containers, and those options apply to IoT Edge modules, too. So, for example, you can give modules access to host storage, map ports or restrict module memory and CPU usage. Azure IoT Edge uses Moby, the open-source project behind Docker, as its Docker-compatible engine. That opens to container orchestration. For example, Kubernetes can talk to Azure IoT Hub to deploy containers in remote edge devices. The advantage of this integration is the ability to use standard Kubernetes tools and deployment mechanisms to orchestrate edge modules at scale.
It makes complete sense to use Kubernetes for orchestrating edge computing modules. Azure IoT Edge heavily relies on container runtime and containers to perform local processing. Each IoT Edge device may run dozens of containers that work in tandem to handle data processing and business logic. These containers are pushed through an IoT Hub where multiple edge devices are registered. By using tags, containers may be deployed to more than one edge device at a time. Azure IoT Hub acts like a typical worker node of Kubernetes. Once Azure IoT control plane instructs to run an IoT edge module as a container on the target device, it packages the container image as a module and hands it over to the remote edge device. Since each edge device may run more than one container, they may be compared to Kubernetes pods. When an IoT Hub is registered with Kubernetes through Virtual Kubelet, with an IoT Edge provider, the master nodes will treat IoT Hub as a node. When a deployment is targeted at IoT Hub, Kubernetes control plane simply hands over the scheduling part to IoT Hub. Developers and operators can use familiar manifests declared in YAML that is pushed through the kubectl CLI.
Summary
IIoT software distribution faces similar challenges with typical software distribution in an industrial setting. Both have many components running in geo-distributed isolated networks, both must comply with the highest security standards and be managed and monitored from a central location. Some will claim that the industrial setting is a private case of IIoT, thus it makes sense to use IIoT frameworks and infrastructures to manage software in an industrial setting.
Final note: To be able to continue and publish new stories Medium now requires writers to have a minimum number of followers, so please help me continue to publish and press the ‘follow’ button next to my name.
