Responsible QA engineers know the importance of test data in the development towards an efficient test automation framework. Well managed test data in a framework is built around the strategy that there is high test coverage as a result. But, what if there are many test data inputs considered? What if you do not have time to test the several combinations of test data inputs in the process of manual or Automation testing? How can you test fewer test cases and still test “right”? 

Well, this is when combinatorial test design comes to the rescue. It helps design an optimized set of test cases (based on the set of test data variables) that ensure 100% test coverage magically!  The high-quality test design CTD result, in turn, can be fed as input to the automation testing tools, thus speeding up the delivery in the Quality Analysis stage.

What is a combinatorial test design - The mathematical background behind CTD

Combinatorial Test Design is a technique that helps plan test data as input to the manual or automation test process, ensuring 100% test coverage based on the several combinations of test data inputs for a test suite. The mathematical concept of “Orthogonal Arrays” sits at the base of the combinatorial test design. A mathematician professor called Taguchi coined the mathematical algorithm term, Orthogonal Arrays.

How CTD can strengthen existing test data-driven test techniques

Consider the scenario where you have test data stored in an excel file. You may be testing the application manually or using test automation tools.  In this case, you can feed the test data inputs to the CTD tool to help you generate a combination of the test data input interactions. This improved set of test data goes as an input to the manual test or automation test processes. Many researchers use the OA concept for their industrial experiments in many domains all over the world.

Examples of CTD scenarios

For example, let us say there is a simple test to validate the login functionality of an application on several types of browsers and devices. Here are the variables: Username, password, and browser. Please note the values of the variables in the following table:

In the above case, in case you plan to test all the combinations of tests, in a multi-dimensional pattern, it would take 27 tests to design a complete coverage of the tests!

3 X 3X 3 = 27 tests

However, In the case of CTD, if you use a tool that is based on the OA concept, it helps you derive tests based on N-pair interaction of variables, at the same time ensuring full coverage, with a few numbers of tests! The N-wise dimension of testing aims to test all the possibilities of all random combination of N variables’ values.

Following displayed table is the result of a two-pair interaction of variable values. There are only nine test cases in the outcome!

The CTD result lists the final test cases, eventually finding a small test plan that covers complete coverage. Note that, two-pair means that every variable found will be paired with another one variable in a two-pair set. 

The above is an Orthogonal array built from a two-pair interaction of variables. Here, you will notice that every variable is paired with all the other variables at least once! 

Example, “No Password” has been covered in a pair at least in one test case with the variables of -

•  “Username variable” values which are – correct username, wrong username, No username. 
• “Browser variable” values which are – Mozilla, Safari, Firefox.

There are only nine tests that have come out of the Orthogonal Array calculation Versus that all-pair combination value of 27 tests still ensuring full coverage in test design planning! Even then, there is a 100% coverage of tests, with the two-pair interaction levels!

Advantages of CTD

There are several advantages to CTD. Namely –

• An optimized subset of tests ensuring 100% coverage
• A lower, subset of test cases generated still ensuring 100% coverage.
• Higher chances of high-priority defects being found – Eventually defects can be fixed early, before reaching the customer. An early and high-quality product, after all, makes the customer happy!
• Saves time in test design and test execution eventually when the CTD result goes as an input to the test automation tool
• This concept needn’t only be applied in several domains of testing. It can be used in several fields, Software, hardware, etc.

How CTD aids Agile and DevOps

DevOps encourages Automation in all forms, and this is what a CTD tool precisely delivers by helping the QA engineer. It enables the team by using an automated fashion of producing an efficient test design. Using the CTD tool, you can quickly build a set of an optimized set of test cases, which can be served as an input to the test automation scenarios.

Automated tests can provide quick results, and fast feedback compared to a manually planned test design method. It eventually also saves time and avoids long feedback cycles. All of this encourages the Agile way of working. As a result, there is early and continuous delivery of quality products.

 Automated tools that help build CTD scenarios.

• IBM FOCUS is a licensed CTD tool used for test planning. It can read the test data inputs, analyze the test data coverage, select a subset of the tests, and generate a new test plan ensuring full range.
• The pairwise testing tool also offers the all-pairs solution for test design solutions, presenting a visualized test coverage view to the QA engineers.
• And more!

 The test scenarios result in that comes as a result of the above-mentioned tools can be inturn fed as an input to the test automation tools, thus speeding up the test design process.

Wrap Up

The field of combinatorics has come a long way, and CTD is one of the branches that has helped the QA/QC teams in many ways by assisting them in building efficient test designs for the product. No coding skills are required to construct a CTD design. The tabular result is easy to understand. Hence, this helps in aiding the communication between the QA and Business analyst teams while discussing the Business requirements. A mathematical method as such should be heavily relied on considering that it has a high ROI by help saving time and help to find high-quality defects. Especially in the case of large-scale projects involving a large number of variables to be depended on for testing, CTD can help the organization a lot.

About the Author

Dan Martin has been building teams and coaching others to foster innovation and solve real-time problems. Dan also enjoys photography and traveling.