Test data can spew rapidly from a manufacturing environment, and unless a plan to manage and analyze it is in place, opportunities for improvement may be missed. Joe shares some lessons in creating a comprehensive test data strategy and offers suggestions for building a test data management system.

Data can be like gold to today’s manufacturing and engineering companies, some of which are unwittingly sitting on a mine waiting to be tapped. Just as there are benefits to managing and leveraging money more effectively, managing data correctly can have rewarding payoffs as well. The ROI in data handling has great potential.

The test community can take comfort in this observation: Nearly everyone struggles with test data issues. It’s rare to find a company that can’t improve how it cares for data.

On one end of the spectrum, a few companies discard all their data in favor of only a pass or fail result, depriving themselves of an opportunity to improve and save money. On the other end, some companies get overly enthusiastic and record too much data without a plan, creating a black hole from which information can never be usefully extracted. Most firms are in the middle, collecting some data manually and some electronically, but planning to invest time and resources to enhance the process.

Data is meant to be transformed into useful information to fuel key decisions and can be more productive with processes that analyze data systematically and proactively. Well-managed and processed data has the potential to save time and money, avoid waste, and improve customer satisfaction. The benefits of data analysis have been proven in statistical process control and Six Sigma protocols.

Creating a test data strategy

Libraries of books about calculation and processing methods exist, but few sources address holistic data management. To meet each organization’s unique needs for managing data, some helpful points should be considered when creating the data strategy.

With computer-based automation growing, it’s important to mull over which measurements and test results to store and which not to store. Storage is nearly free, and data can always be deleted later, so there is no reason not to save as much as possible. If there is any chance data can be useful, go ahead and store it, taking into account the needs of other interested groups such as the design and marketing departments, which might need data for their own priorities. On the other hand, be wary of collecting more data than what can be stored or moved, potentially bogging down any efforts to search the data. Discard or quarantine data if it is certain not to be useful to anyone or is redundant. Routinely archive data into another location if its usefulness has passed to keep data volume reasonable.

Take time to study processes and test systems for less obvious data that would otherwise be overlooked. Automation and test products (hardware as well as software) are increasingly involved with data for input and output. For example, configuration files might govern automated test setups or a table of MAC addresses might be assigned to each unit produced. Those and other examples of nonmeasurement information must be dealt with and could prove useful in analysis or in the event of a system crash.

Digging around in formats

As data is uncovered, yet another new challenge will probably emerge. Most software and hardware products impose their own data format for input or output and might also suggest how to organize data. Combinations of just a few different products could create a “Tower of Babel” scenario in which many incompatible file types are produced, making the data very time-consuming to navigate or manage.

A well-designed test product allows users to specify or select a preferred file format. But with that power comes additional responsibility. Some basic formats such as text files are easy to create and read but provide no common structure; they are an excellent choice in the right environment. The other extreme is the popular XML format, which combines the ease of text files with rules of construction. Microsoft has widely embraced XML and built tools to comprehend it; some other editing and reporting products now open and explore XML if it’s well formed. But this format can be daunting to anyone who isn’t savvy with the pitfalls and confusing tags and can be cumbersome to read, as illustrated in Figure 1. Invest extra time to explore the options before choosing or creating a file or data format to meet current and future needs.

Consolidating the haul

Once decisions are made about what data to collect and which format to use, consider how to consolidate the data from various machines in test and manufacturing. The first challenge is copying data from the local machine to a centralized location. The corporate network folders might be a good starting point, but don’t rely on manually copying data. Research and invest in automated data copying to make sure data is positively grabbed from the sources and secured during transfer and storage to the central location.

With data residing in a central location, a database is often a foremost requirement in the minds of many engineers. In a database, tables can be designed to contain and index the data from raw files and can establish a relationship between tables. But the proliferation of databases means that some engineers spend their careers as database administrators, which can become wearisome after dealing with the hidden costs of database creation and management. And although preparing and modifying data for a database can be an excellent way of purifying and reorganizing data, it also can be the most troublesome task in data analysis. Ask if a unified database with all data is needed or just a well-structured set of files with useful formats and reporting tools.

Unique data, but similar needs

Companies often think they have unique needs in managing data, but in reality, they don’t. Most businesses wrestle with many of the same issues.

For companies that build and test products or subassemblies, data requirements vary but usually fall within several types. Most companies test products for adherence to specifications, and some deal with formal compliance, regulatory, or industry specifications. Test routines often comprise a set of ordered steps and measurements to be verified one by one. Many companies serialize products and subassemblies; others run units in batches or lots. Regardless, some kind of traceability exists. If purchased subassemblies are used, the supplying vendors usually have test data available. The result is a set of measurements that not only indicates pass/fail but also denotes the health or quality of the product and the processes used to build it, as the report generated with an XML style sheet shows in Figure 2.

Since most manufacturing and test environments produce similar kinds of data, there is a good chance that a data management system might already be wholly or partially available and only need customization. Two steps can help identify and evaluate these systems while reducing risk and cost, speeding up the results, and keeping projects on schedule and under budget.

First, search for a way to view a working data management system. It could be within the company, in a colleague’s company, or as a demo from a vendor or consultant. Second, look for products that are mostly off-the-shelf and match the needs as closely as possible. Assume any purchased product will need customization and verify that customization is possible (ask for a demonstration). Avoid the temptation to build anything from scratch, which can eliminate time for fixing mistakes, debugging mistakes, and learning tough lessons.

A new test data appliance

Cyth Systems implemented the aforementioned data management strategies when designing the Omnimetriq product family (professional and enterprise models shown in Figure 3). Packaged as a plug-and-play data appliance, users can easily buy, install, and manage the device without having to buy a server or install databases. Key features of the data appliance include:

• Ethernet connectivity for easy integration into a test station network
• Predefined test station interfaces for common applications like Unit Testing, Batch Processing, or Quality and Life Testing
• Powerful, off-the-shelf tools from Microsoft and Business Objects for storing data and creating reports
• A standard database design requiring minimal management
• Automated tools to import data from several standardized formats
• Prewritten reports so users can immediately see common information like process yields, failure pareto, and measurement statistics
• Filtering so results can be narrowed for all products, selected products, or any other chosen parameter
• Customized, modifiable reports that can be created easily

Omnimetriq allows companies to match some or all the characteristics of a common manufacturing operation as described earlier.

Tapping the mine

If managing test data is a new item on the agenda, don’t hesitate to search for advice and tools to help get started. If the project is already started, plan and hone the efforts for best results.

Someone interested in mining for gold might want to read a book or find a tutorial, buy a pan or shovel, and learn from an expert who knows where and how to dig. Mining test data isn’t all that different. That which looks like a muddy pool of disorganized data can bring a company a hidden fortune of golden information.

Joe Spinozzi is a founder and senior director of operations for Cyth Systems, a custom automated test systems reseller and integrator based in San Diego, California. With a bachelor’s degree in Physics from Northwest Nazarene University combined with training in quality and Six Sigma, Joe has gathered experience with optics, lasers, RF, and wireless technologies. He recently presented on the topic of real-time data processing and management from moving experimental vehicles at the American Controls Conference in Minneapolis.

Cyth Systems
888-508-7355
[email protected]
www.cyth.com