With the proliferation of operating systems and web browsers, plus the increasing need to support more languages, software testing is a vast, complicated proposition. Yet, localization managers must push for faster turnaround and lower costs without any impact on quality.
The traditional approach, if you wanted to be on the safe side, has been to test "everything against everything". But that costs an arm and a leg and takes a lot of time.
This method was just fine in the past when most applications were stand-alone and had no interaction with third-party software. But the new era of technology introduced a different way of development, making software dependent on web servers, database engines, web browsers, etc. What’s more, apps are less and less tied to one operating system, allowing users to use them on any system they prefer. Then deliver several language versions, and your testing scope becomes considerably more complex (and costly).
Imagine that you need to test a web-based application and you want to test it in the following:
- Operating systems—three of them: Windows 10, Fedora 27, and macOS 10.13
- Web browsers—five of them: Microsoft Edge, Google Chrome, Safari, Firefox, and Opera
- Languages—let’s start with eight: French, Italian, German, Spanish, Simplified Chinese, Traditional Chinese, Japanese, and Korean
If we test every language on every browser in every OS, we have to test 120 unique environments! I’m guessing that no global enterprise is willing to pay for that. But there’s hope: pairwise testing.
Here is the description of this technique from Wikipedia:
All-pairs testing or pairwise testing is a combinatorial method of software testing that, for each pair of input parameters to a system (typically, a software algorithm), tests all possible discrete combinations of those parameters. Using carefully chosen test vectors, this can be done much faster than an exhaustive search of all combinations of all parameters, by "parallelizing" the tests of parameter pairs.
If the previous paragraph makes no sense to you, let me explain: you set up a testing matrix (which consists of an operating system, a web browser, and a language) in a way that each test condition in any category of the matrix will pair at least once with one of the items in the remaining two categories. This means that while not every single combination is tested, each option is combined with all other options.
We in Moravia have our own tool named CaseCreator which allows you to specify as many inputs as you want—with all the exceptions you need to consider as well. For example, the Edge browser is only available in Windows, and Safari is only on macOS, so testing your product with these browsers on other systems is not even possible.
You can see the data produced by this tool below—it processed the test conditions listed above. Just pick any item and you will see that it is paired with all the items in other categories. Take a look at German, for instance. We are testing the German product with Firefox on Windows 10, Opera on Fedora 27, Firefox on macOS 10.13, Edge on Windows 10, Chrome on Fedora 27, and Safari on macOS 10.13. We have each operating system condition at least once, and each web browser condition once. That covers all OS + browser combinations for that language.
Our pairwise testing results from CaseCreator.
This testing approach reduces the number of combinations from 120 to 43—a 64% reduction from the 120 scenarios.
Don’t have a pairwise testing tool? There are a handful of them available on the internet—some free, some at a price, and some are code you can build into your own environment. Or, you can go old-school with an Excel spreadsheet. Using our scenario once more, put OSes, browsers, and languages into three columns. With 3 OSes and 5 browsers, that’s 15 possible combinations per language. Copy your language set 15 times down that column, then create your 15 unique combinations of OS and browser for one language. Replicate those combinations for each language to get a total of 120 unique scenario rows. Next, delete the exceptions and identify which combinations you absolutely must test—such as every language with macOS 10.13 + Safari, and Windows 10 + Edge—which are easy to do with filters. Now comes the hard part: choose the other test combinations while avoiding the same combination of OS + browser over and over again.
An example of figuring out pairwise testing scenarios using Excel.
Regardless of how you get your matrix, it’s important to identify in advance any constraints in the testing scenarios, since it’s very likely when mixing and matching technology that not all combinations are always possible. We still advocate using a good matrix generator over homemade methods, since it can be instructed with forbidden combinations to produce a much more precise matrix.
No matter the size of your project, successfully planning pairwise testing can help you achieve optimal test coverage within your budget without majorly impacting the final quality of the tested product.