Maximizing customer coverage

Customer coverage testing means discovering how your software is actually being used and testing it that way.
(Page 1 of 2)

In my April 1999 column, " The pain of platform possibilities," I discussed the impossibility of testing all the conceivable combinations of supported platforms. I also introduced the concept of certified vs. supported platforms as a strategy for identifying what can reasonably be tested internally and what should be supplemented with beta programs or risk mitigation strategies.

Customer coverage testing means discovering how your software is actually being used and testing it that way.
The next issue you are confronted with is what you can really test within the subset of certified platforms. Or, even if you support only one platform, what is realistic to expect in terms of test coverage? The word realistic is the operative word, by the way, because objectives like 100% test coverage are laughable in most cases.

Don't believe me? Let's do the math again.

Believe it

Let's say you are testing a file transfer system that supports six platforms, either sending or receiving, and on each platform you support three operating systems and two versions of each. Within each platform you have four protocols, five file formats, minimum and maximum file sizes, encrypted or nonencrypted, plus two different flavors of receipt confirmation.

Furthermore, let's assume you have reduced the scope of the test effort by agreeing to certify only four platforms and one version of each operating system per platform, and it takes you 45 minutes to configure a single source-target platform combination and 15 minutes to actually set-up, send, and receive a particular file, then check the results.

How many test cases and how much time do you need to test 100% of the possible combinations and only the valid boundaries?

I get 23,040 individual tests, which would require about 168 person-weeks if you get seven productive hours per day--and that's a stretch. You could probably automate a lot of this, but even that takes time and effort to develop and maintain. If you start adding in equivalence classes, including positive and negative cases, the number goes off the charts.

This is a fairly straightforward application. I don't know of that many companies--in fact, let's say none--that can afford this level of effort on a routine basis, although I sincerely hope those who are testing the software controlling weapons of mass destruction can and do.

Face it

My point is not to spread defeatism, but to simply expose the truth. Testing every possible combination of every factor is just not realistic. If you set this as your goal, or allow others to, you are doomed to fail and suffer.

Understand, I'm not against full coverage or high quality, but I am against setting unrealistic expectations that lead to disappointment, demoralization, and--ultimately--turmoil and turnover in the test department.

Page 1 of 2

1 2
Next Page

0 Comments (click to add your comment)
Comment and Contribute


(Maximum characters: 1200). You have characters left.