Fifteen or 20 years ago, many embedded systems consisted of an 8-bit processor with a few KB of memory and perhaps a couple hundred lines of code. Today, the explosion of embedded systems in automobiles, ATMs, cell phones, and other devices has dramatically changed that dynamic. It’s not uncommon to see multiple 32- and 64-bit multicore processors on an Ethernet network, running multiple applications totaling hundreds of thousands of lines of code in these systems.

As these devices grow in complexity, it has become a significant burden for project teams to confirm that software requirements have been tested and tests can be traced back to requirements. In many cases, even with extensive testing, teams might not know whether they have successfully met all of the product requirements.

A high level of complexity and the need for greater system reliability also bring quality to the forefront. Older embedded systems were either simple enough or not important enough to spend time finding, tracking, and fixing software defects. Today, defects in embedded systems can be just as common and even more important than those in enterprise applications. And because of the greater complexity, it becomes more important to be able to trace defects back through test cases and to requirements so that teams have increased transparency into issues and the features they affect.

Today’s complex embedded systems can have hundreds or thousands of requirements, with as many or more test cases. Tracking and executing these test cases, and using that information to make sure requirements have been satisfied, becomes a real concern for teams that have to gain better control over their processes. To address this challenge, teams need an automated way to link artifacts so that data about one artifact becomes information for others. Application Life-cycle Management (ALM) software can help.

The value of traceability

Traceability is the practice of linking requirements to downstream artifacts like risk, test cases, defects, and even source code. Links enable two-way communication of change and progress between related artifacts.

For example, once a product has requirements, those validating it need to write test cases that ensure the product under development is meeting the design requirements. If test cases fail, the resulting defects are recorded in an issue tracking system.

Traceability enables product teams and stakeholders to understand and derive valuable information from the relationship between product development artifacts. By looking at defects, it is possible to determine which requirements haven’t yet been satisfied. Teams can use this information as an important guide to determine whether a product is ready to ship, and also to triage defects to satisfy the most important requirements first.

But effective traceability can do a whole lot more for a project. If safety risks are identified as a part the product, these can be linked to nonfunctional requirements that can then be traced to downstream artifacts. Source code can be traced to defects and back to requirements, closing the loop between design, development, and testing. In the same vein, when defect fixes are checked back in to source control, and tests run to confirm the fixes, the team and other product stakeholders know very clearly that the corresponding requirements have been satisfied.

Traceability also provides essential project information that often can’t be obtained in any other way. It provides testers with an easily understandable and reportable measure of product quality. By knowing which requirements remain unsatisfied, and whether they have issues logged against them, testers can estimate the time remaining to product completion. Last, traceability enables teams to better understand the work remaining, and in which functional areas of the product that work remains.

Managing requirements, test cases, and defects using Microsoft Word or Excel is challenging enough. But tracing requirements through test cases to defects and back to requirements is impossible without a real tracking system.

Building traceability into an embedded project

How does a product team go about building traceability? It starts with requirements. Once functional requirements are defined, risk analysis and mitigation begins, and testers generate test cases that will enable them to determine if the product under development meets those requirements.

As testers run test cases, the cases that pass indicate that the related requirements have been satisfied. If test cases fail, testers record defects that are associated with those test cases and, by linkage, to the requirement under test. The defects are associated with the related test cases and test runs, providing a link back to the testing stage.

Ideally, this linking is done automatically. While it is possible to manually create and maintain links between project artifacts, the work involved is detailed and constant. Whenever a requirement or test case changes, links have to be manually reestablished. The effort needed to manually create and maintain links between requirements, test cases, and defects is excessive, especially if those links have to be examined and updated almost daily.

In most projects, testers execute test cases multiple times, in different test runs. Tests are rerun when an initial run fails, and the fix needs to be verified. Tests are also run additional times for regression purposes, as the embedded software product scope grows to meet more requirements.

Tests may also be performed manually, or in an automated fashion using a testing harness. In practice, most teams do some of both. Manual testing is typically done the first time, and recorded using an automated tool. Subsequent tests are often run automatically, unless the functionality changes significantly. Test results provide the basis for traceability information, through either defects or successful test results. Both can trace back to requirements and other artifacts.

With automated testing and traceability, testing teams have the opportunity to perform at a high level, accomplishing testing within the product schedule and providing valuable information on quality and requirements fulfillment to product stakeholders. This makes testing more relevant to all stakeholders, especially in the latter stages of product development.

Ideally, this starts with requirements, but must also incorporate test management, defect tracking, and source code management. For products with safety implications, it should also incorporate risk management and mitigation. ALM software, such as TestTrack from Seapine Software, offers the ability to create, manage, and link artifacts from the beginning through the end of a design and development project. Figure 1 illustrates how a traceability matrix can help teams easily determine how defects or requirement changes can affect other parts of a project.

 

 

Traceability and safety-critical projects

Traceability has traditionally been used for large development projects with hundreds of engineers and testers, and thousands of requirements. This has been primarily the realm of commercial and military aviation and space systems, where the need for information to combat complexity overcame the cost of that information. These systems can take years to develop and bring to market, making the information gained through traceability especially valuable as team members leave and new ones arrive.

But with automation, such as that provided by ALM software, the cost and effort of traceability can be driven down to the point where it makes sense for smaller projects, and with shorter schedules. Automatic traceability between requirements and downstream artifacts, and back upstream from defects to requirements, makes identifying and communicating development hurdles and issues faster and more transparent.

Projects, such as smartphones, automotive systems, and smart industrial control equipment, benefit from better traceability. As more testing teams incorporate automated ALM methods to collect and disseminate traceability information, product quality will continue to improve, even as systems become more complex.

Peter Varhol is the solutions evangelist at Seapine Software and has authored dozens of articles and spoken during many industry conferences and webcasts. His past roles include technology journalist, software product manager, software developer, and university professor. He has advanced degrees in computer science, applied mathematics, and psychology. Contact Peter at [email protected].

Seapine Software www.seapine.com/testtrack.html

Follow: Twitter Facebook Google+ LinkedIn YouTube