As you can see from previous posts, as Soldevelo testers we had an inspiring journey to Romania. We took part in a test conference and now we want to introduce you to its more technical side.

During the SEETEST conference, we had several interesting tutorials to choose from. One of our choices was ‘Practical Test Design Techniques’ presented by Michaël Pilaeten.

The speaker is from Belgium. He has 15 years of experience in test consultancy in a variety of environments. He has been CTG’s primary ISTQB and certification trainer since 2008. Over the last 5 years, he has helped more than 400 testers in improving their skills set.

Currently, he is responsible for guiding the company’s consultants, partners, and customers on their personal and professional path towards excellence.

His workshop provided hands-on exercises regarding techniques for designing qualitative test cases.

During the lecture, we discussed a couple of test design techniques like:

• Equivalence Partitioning
• Boundary Value Analysis
• Decision Table
• State Transition Testing
• Algorithm Testing
• Classification Tree Testing
• All Pairs

Six of the above techniques were known to us and, I think, for every tester with the ISTQB certificate. That is why we decided not to summarize them in this entry.

While discussing the last technique, we raised the problem of reconciling budget, quality and time requirements. Then the ‘all-pairs’ tool was presented.

To be honest, this tool was something new for us. How simple it is to use this test design technique made us all very interested, so we delved into the subject and tried to use it in the project we’re working on.

Pairwise Testing

ISTQB Glossary explains pairwise testing as ‘A black box test design technique in which test cases are designed to execute all possible discrete combinations of each pair of input parameters.’

It is interesting that 99.8% of all reported defects are single or double-mode defects.

This method allows testing the minimum subset of all pair combinations.

It is able to significantly reduce the number of test cases when risk increases slightly. It ensures that each parameter is tested because it covers all pairs combinations.

This method, for each pair of input parameters, tests all possible discrete combinations of those parameters.

Michaël Pilaeten showed us a list of popular all pairs tool for this method

• Allpairs – James Bach – Satisfice
• CTE (Classification Tree Editor) – Rozorcat
• PICT (Pairwise Independent Combinatorial Testing) – Microsoft
• FoCuS (Functional Coverage Unified Solution) – IBM
• ACTS (Advanced Combinatorial Testing System) – NIST, US Government agency
• Hexawise – Hexawise
• Jenny – Jenkins
• Pairwise – Inductive AS

Manual testing has a few disadvantages. It consumes time and is expensive. Manual testers need to test the same set of test cases multiple times changing one variable and may not be sure that they cover all available test cases in complex applications.

As people who care about the quality of products, we know how important it is to properly prepare test data to cover all test cases.

Using the tool seemed great because it is able to compensate for these problems.

But this tool is not perfect. It has also disadvantages or points that we should keep in mind:

• Pairwise testing does not verify ALL combinations
• Pairwise testing does not give the most commonly used combinations
• Produced combinations are deprived of prioritization
• Some combinations might be even infeasible in the real world, for the final users.

To work with one of the generation tools we need to follow some procedures.

First of all, we need to create a model that will contain identified relevant parameters and possible values for each parameter.

When our model is complete we can use an ‘all pairs’ tool to create the set of test combinations.

Let’s check this procedure in examples.

Real scenario

Two of us are working on the ODK Collect project. It is an Android app for filling out forms and sending them to different types of servers. We are actively involved in manual and automation testing of this application.

As Android OS develops very quickly we support a lot of Android versions. Currently, we verify each case on 7 different Android versions.

For example, ODK Collect is able to connect with a bunch of servers, where data are collected, like:

• ODK Aggregate v2.0
• ODK Aggregate v1.7
• Central
• Google Sheets
• KOBO
• ONA

The aforementioned application is multilingual so testing contains one more requirement, i.e. languages. We are checking two of the most popular languages: English and Spanish. Additionally, we are verifying some language where writing starts from the right of the page and continues to the left like Arabic.

Let’s consider a simple test where sending collected data to servers is verified.

Our variables are:

• 7 Android version
• 6 servers
• 3 languages

Now let’s calculate the number of possible combinations:

= 7 x 6 x 3

= 126

126 test cases only for one, primary functionality such as form sending!

Testing this number of test cases takes a while for sure so it is nearly impossible to repeat them in each regression cycle.

‘All pairs’ tool helps to reduce this number using combinatorial method and orthogonal arrays.

As a first step, we need to create a sheet with all of our data and save it as a .txt file.

Then we need to run the ‘All pair’ tool. The used tool comes from the recommended side https://www.satisfice.com. It works only on the Windows systems.

Run the program from a command-line like this:

allpairs.exe data.txt > testcases.txt

The tool reduces the number of test cases to 42 test cases. It is almost 67% less than initially.

Some values are marked by a tilde. As we can read in the documentation of the used tool, it is the value that doesn’t matter because all of its pairings have already been achieved. In that case, the value chosen is the one that has been paired the fewest times relative to the other cells in the test case. It’s an alert allowing to choose the value that either maximizes the probability of failure or maximizes the impact of the failure, should one occur.

This list of combinations of parameters can reduce human error and can be tested consuming less time and money than our original list and after all, it seems to be doable.

Now, let’s check one more scenario.

ODK Collect has two color modes: light and dark. Recently, a very popular functionality 🙂

Adding this parameter to our scenario causes that the number of test cases is doubled.

= 7 (Android version) x 6 (servers) x 3 (languages) x 2 (modes)

= 256

But if we run the tool again we see that it generated exactly the same number of test cases.

Adding parameter does not always cause an increasing amount of test cases because multiplying the two largest values (7 and 6) indicates the number of pairwise tests.

Math doesn’t lie

The pairwise testing method can dramatically reduce the number of combinations to be covered but remains highly effective in terms of fault detection. It is recommended for cases with a limited number of parameters with multiple possible values.

It is a smart technique that guarantees a win-win situation for test effort and test effectiveness.

We hope you will try to use this technique in your projects because it really reduces the time that one would have to spend on testing, but also eliminates repetitive tests.

We also encourage you to take part in test conferences that are organized in Europe, but also in Poland, maybe someday we will meet there! 🙂