How Unifying a Fragmented Embedded Product Lifecycle Can Save OEMs Time & Money

July 21, 2023

Blog

The lifecycle of a connected embedded device, from development to disposal at end of life, can stretch over 10 or more years and has a number of distinct stages: the Foundries.io analysis counts eight of them. For many OEMs today, the transitions between these stages give rise to a high degree of organizational and technical fragmentation.

This fragmentation runs counter to any product manufacturer’s business interests. In a rational world, OEMs would tend to design products to be easily and cheaply secured, maintained, and updated in the field. The product is a single economic and physical unit over its entire lifetime: this means that the cost of a product’s maintenance in the field, as well as its disposal cost, cannot somehow be isolated from the business case that justifies the product’s purchase price. On the contrary, the whole-life costs of the product have to be reconciled against its sale price (and if applicable, any service revenues made from product users) when calculating the profitability of a product.

Unfortunately, organizational fragmentation weighs heavily against this rational approach. In practice, upstream decisions and actions by the development team have downstream effects on test, production, and maintenance teams. Conflict between these teams and their work output can cause delays in the product reaching the market, create a higher risk of security vulnerabilities and liabilities, and compromise processes for updating products. Security is often only addressed in earnest during the latter stages of product development, creating issues further down the line.

Such convolution and conflict can be avoided. While some organizational fragmentation is inevitable – it is natural that development operations should be run as a separate unit from maintenance operations, for instance – the technology exists to unify the underlying engineering basis of a connected embedded device across its entire lifecycle.

By exploring the eight stages of an embedded product’s lifecycle, this article will demonstrate the value of a new generation of embedded DevSecOps platforms that build tools for meeting the requirements of development, lifetime security, maintenance, and fleet management into a product right from the moment an OEM starts to design it.

The Value and Costs of the Eight Stages

The eight stages model of a product’s lifecycle that we’ve summarized applies more or less universally across all types of connected embedded devices and all market segments.

1. Central component selection – the commitment to a specific instance of the microprocessor, FPGA, microcontroller, GPU or other device type around which the hardware implementation will be built. This choice has a profound impact on the type of software the product can run and the applications it supports.
2. Preparation – in this phase, developers will build out a trial software development flow on the component chosen in stage 1. This work verifies the operation of the chip vendor’s board support package, and basic functions such as flashing the initial boot image(s), configuring the secure or high assurance boot path, and making a secure network connection. 
3. Development – the preparation stage provides the basic hardware basis for the product. Stage 3 is when developers build the application, in which all the value of the product resides. At this stage, developers learn to expect the unexpected. Reacting to changed market or technical circumstances can lead to demands for change in the tools, software platforms, or security processes established in the preparation stage, or even in the SoC chosen in Stage 1.
4. Prototyping – after the first three stages, the OEM knows that the SoC performs most of the desired functions on its evaluation kit. In the prototyping stage, developers will build application software to run on a form-, fit-, and function-specific prototype. At this stage, the production team will engage with the development team to implement DFM (design for manufacturing). This means that the platform development team maintains concurrently a new BSP for the custom hardware alongside the original evaluation kit’s BSP.
5. Testing – now the test team can verify that the application functions as envisaged, and that future updates of firmware and application software can be tested and validated before deployment. 
6. Tooling – the systems required to support volume production are now implemented. At this stage, the OEM might discover that the tooling used for preparing prototypes in the laboratory do not also support production. Security flows for functions such as root-of-trust and key management are now of critical importance. Production engineers frequently find that development and prototyping flows cannot be used to build a production image.
7. Production – the make-or-break moment at which the OEM discovers whether, after all the work to date, the product design will be realized in volume. It is particularly important to establish whether production flows for securing the device, managing its root-of-trust, and provisioning cloud service credentials can be linked to the firmware update mechanism with which the product will be supported for the rest of its life in the field.
8. Maintenance – legal liability, brand reputation management, and government regulation are all putting increasing pressure on OEMs to run regular updates of products in the field, both to limit exposure to security vulnerabilities and to maintain the product’s operation in line with the buyer’s reasonable expectations.

The cost of the first seven stages, and the amount of value that they generate, are determined by internal factors, such as the skill and dedication of the developers, and the effectiveness with which they are managed.

The eighth stage is different. The outside world affects how often and how severely a product is assailed by cyber attacks. External factors also dictate the frequency with which changes in the environment, such as new cloud credential requirements for example, necessitate product updates to be deployed out in the field. When the response to a common vulnerabilities and exposures (CVE) notice is botched or a product falls victim to a cyber threat, an OEM can be exposed to massive legal and engineering costs that wipe out the product’s profitability. Effective maintenance can provide an OEM with extremely strong protection against this risk.

All too often, however, the effectiveness of a product’s maintenance operation in Stage 8 is compromised by decisions made at earlier stages.

The truth is that product managers need to be prepared for the moment when a new CVE notice appears, and senior executives demand answers to the questions:

• What did we do to avoid falling victim to a vulnerability, and when? 
• Was our action successful? 

This is the moment many OEMs discover that a product’s development compromised the company’s maintenance and fleet management operations.

Fragmentation in Engineering Processes

This exposure to lifetime risk is a common consequence of the fragmentation in the development process: different engineering competences, tools, structures, and management processes are required in development and prototyping stages from those required in testing, production, and maintenance.

The resulting organizational separation bakes different incentives and time horizons into the structure of an OEM’s product units. Each team can therefore end up making decisions from its own narrow perspective, rather than that of the whole product. For instance, in the preparation stage (Stage 2), it is helpful to secure the SoC, because leaving security to a later stage will take longer and could entail costly and time-consuming reworking of firmware and software development efforts.

But at this stage, developers are measured and rewarded for the speed with which they provide an operational platform on which developers can start building the application. So their incentive is to omit security.

Likewise, in the development phase, the creation and population of a software bill-of-materials (SBOM) ensures that there is a real-time log of the software components of every variant of the product. An SBOM is becoming an important element of compliance with specifications for software used by governments, and institutions such as the US Cybersecurity and Infrastructure Security Agency are strongly advocating for the maintenance of comprehensive SBOMs.

But the creation of an SBOM slows rather than advances the product’s progress through the development stage (Stage 3), so of course the development team has no incentive to create it – hence many products from OEMs today lack a comprehensive, up-to-date SBOM.

In general, product development processes have evolved in the first seven stages of the product lifecycle to neglect the interests and needs of the eighth stage. And the engineering culture in many embedded device OEMs reinforces a certain disregard among product development teams for the need to secure, maintain and update the fleet of production units in the field.

DevSecOps Platforms: Security and Maintenance Built In from the Start

Today, however, the emergence of government rules on device security, and the high financial and reputational costs of security failures, are driving more embedded device OEMs to find a way to support the maintenance stage of the product lifecycle from the start of a product’s development.

A promising approach is the concept of the DevSecOps (development/security/operations) software platform. This kind of platform provides a customizable Linux operating environment for a wide range of SoCs and processors, and supports an integrated flow for board enablement, security, development, maintenance, and fleet management throughout the product lifecycle.

The use of an integrated DevSecOps platform provides a bridge between every stage of the lifecycle. For instance:

• Support for SoCs from the world’s leading chip manufacturers means that the selection, preparation, and development stages can happen concurrently and in lock-step.
• Applications developers can deploy their software on the SoC’s evaluation kit and on custom hardware simultaneously, while keeping all work aimed at readiness for manufacturing synchronized with the development flow.
• Development on the evaluation kit and the pre-production flow will work with the same backend build, secure and deploy frameworks used in the test, tooling and production stages.
• An SBOM is automatically generated and updated for every instance of the product’s firmware and application code. In production, an SBOM is then generated for every production unit shipped from the factory. The provision of a node-by-node SBOM enables product managers to respond immediately and easily to queries about a product’s exposure to any CVE notice.
• Production tooling is compatible with development tools, so that production teams can recreate prior builds in minutes, migrate old designs to new firmware, provide detailed immutable logs for updates over the life of the device, and manage the deployment of application code down to the node.
• All of these prior stages are fully integrated into tools for fleet management and secure over-the-air update delivery and implementation.

By implementing product development on a DevSecOps platform such as the FoundriesFactory product from Foundries.io, OEMs can eliminate the security and product maintenance risks that arise from the natural fragmentation between organizational silos.

A unified platform with common tooling from Stage 1 to Stage 8 also creates more opportunity to develop and deploy new features and exploit more revenue generation opportunities in the installed base of connected devices.

So the use of a DevSecOps platform can turn the maintenance Stage 8 of the product lifecycle from a potential source of financial and reputational risk into a revenue generation opportunity.