Application development is changing from a process of crafting individual lines of code to one of assembling pre-built components, which poses interesting questions about its potential impact on testing. The hope, of course, is that if each component is individually tested, then the final aggregation requires less testing. In fact, one reputable testing expert maintains that eventually no testing at all will be required, so long as the components are each tested individually.
Is this true? I agree that using ready-made components can reduce testing, but I disagree vociferously that they can obsolete it altogether.
I have the unfortunate distinction of being able to address the testing of third-party components from personal experience. Unfortunate, because my developers and I were tortured for months by a faulty component we had purchased.
Here’s what happened. We bought a grid control and incorporated it into our software, only to have repeated exception errors that crashed the whole application. The vendor insisted the control was widely used without the type of problem we were reporting and the problem had to be elsewhere.
Puzzled, we spent endless hours, then weeks, even months looking for the root cause, all the while experiencing the annoyance and inconvenience of apparently random but repeated crashes. Finally, the developer decided to simply replace the control. But since it would require a significant effort, he decided to research his options carefully by visiting news and discussion groups frequented by developers. Guess what he found out? The problem we were having was, in fact, known and reported by others. Other developers knew of and had reported the same problem we were having.
When confronted, the vendor hastened to explain that all of the object’s methods and properties were thoroughly tested and worked, but the problem arose because of the particular combination in which we were using them.
The point here is that no matter how well an object is tested, it is impossible for its developer to predict all of the possible permutations in which it may be used. Even a fairly simple object with only a dozen methods could support almost half a million different sequences. Now, this does not mean that you should exhaustively test purchased objects individually, but it does mean that you still must test it within the specific implementation.
Between and Around Objects
A related issue is the interaction between objects. Again, I can speak from experience. Disgusted with the vendor of the grid control, we replaced it with another, only to find that we still experienced crashes. Granted, they were less frequent and did not seem to involve the grid, but they occurred nevertheless. By now I was getting grouchy, and the developer was becoming defensive.
There was only one other third party control in the errant application: a tree view. However, we used this tree view in other applications without incident, so it seemed immune from suspicion except that the crashes seemed to occur when focus was moved to the tree. Through yet more research, the developer finally discovered that the problem was related to a shift in focus from one object to the other.
In this case, it was a competition between two objects for system resources, a problem that would never be manifested when either object was tested individually.
As we have already seen, objects exist as part of a larger system, and problems can arise either from the way the system interacts with the object or the way the objects interact with each other. But there is yet another area of risk: the glue that holds the objects together. Objects aren’t like Lego blocks that simply snap together, they are more like bricks that need mortar between them.
This mortar includes both application and system code that interacts with the object. Simple things can creep in, such as incompatibilities between versions of shared system components. I have seen one object refer to a dynamic library that another also needed, but they each expected a different version and neither was compatible nor able to peacefully co-exist.
Where to Draw the Line
Testing is all about assessing risk and setting your priorities accordingly. It is highly unlikely you will have the time or resources to test your application as thoroughly as you will want or need to, so it makes sense to look for ways to reduce your efforts. However, the idea that you don’t need to test components just because they were developed by others is simply not true.
Does this mean you must exercise each and every method and property for each and every component you use? No. What it does mean is that you need to test your particular implementation of it to be sure it does what you need and in the way you expect.
Whether your system was specially handcrafted or cobbled together from off-the-shelf parts, the fact remains that each implementation is unique and the requirements of each application and customer pose new challenges. //
Linda Hayes is CEO of WorkSoft Inc. She was one of the founders of AutoTester. She can be reached at firstname.lastname@example.org.