The Testing Pyramid

One of the most widely recognized models for conducting software testing is the V-model, which tracks the development life cycle and associated testing tasks at each phase. Unfortunately, this model does not serve independent test groups that are not part of development, especially those groups that represent the customer.

Said another way, the V-model is a test approach that is based on the way the software is built. Another model is needed, however, for testing the software the way it is going to be used.This is where the inverted V-model, or testing pyramid, comes into play.

How it Works

Unlike the V-model, which moves from top to bottom, the testing pyramid moves from the bottom up. The first step is to establish the test environment. This is the foundation–literally–of the entire process, and includes the hardware, software, and data that comprise the test environment. This environment needs to be stable and controllable, or you will not be able to reproduce or automate your tests.

The next level, Build Verification, was covered in last month’s column, An Admission Ticket to Testing. Build Verification is a broad but brief test that assures that the software build has sufficient integrity to be accepted into the test process. It can also be used for emergency or “hot fixes” that need to bypass the regular release and test cycles.

The third step, Critical Process Verification, assures that each critical process functions as expected. Critical processes are those that pose a high risk to the user, in potential loss of time or money. Though not as broad as Build Verification, this level tests each critical process at a deeper level, exercising key paths and data values.

At the top of the pyramid, Functional Verification focuses on specific areas of functionality that have been modified, added, or fixed. Narrower in scope than the previous levels, it goes deeper by exercising special cases or unusual circumstances.

What It Means

Following this pyramid will help prioritize risk, provide for logical breakpoints, and focus management on release readiness.

Prioritizing risk means that the highest risk tasks are attacked first, and lower risk ones are not addressed until later. In this context, risk is measured by the potential loss of money or time. Because testing is, on a practical level, infinite, almost any application area or function can be tested endlessly, using all manners of combinations and permutations. Usually, there is not enough time to test everything, so you must be prepared to ship without completing all of your planned tests. This way, if you have to ship without completing all your planned tests, you will at least know that the untested areas will have the least impact should they fail.

Notice, for example, that the testing pyramid does not test modifications, fixes, or enhancements until after everything else is checked out. Many test teams attack this first. The incontrovertible fact, however, is that it won’t matter that the changes themselves were properly made if they accidentally broke something basic and critical. New functionality has an inherently lower risk because, by definition, the customer or user is not currently using it. Breaking something they are already using has a much higher price tag.

To get the highest risks out of the way first, you need to define the software’s logical breakpoints and stick to them. This way, you complete each level before moving to the next one, allowing you to achieve useful milestones earlier. Otherwise, testing tends to smear into one long, continuous session that is difficult to measure or predict. By completing each layer in the pyramid before moving to the next one, you have shorter interval schedules to manage to.

It is also easier to evaluate release readiness using this model. At each level of the pyramid, you have a clear definition of what is included and what is at risk. We have already given the “hot fix” example; another might be that if you want to provide early releases to beta customers, you and they might agree that at least the critical processes have been verified. There are now multiple decision points along the way, not a single monolithic event–at each point in the pyramid you have a known level of coverage completed and still outstanding.

Practical Application

Like any model, the key to making the pyramid work for you is to get agreement on what is included in each level. Involve your users or customers in helping you to assess risk and determine which tests belong at which level. Then, stick to it. Do not be pressured into an unstructured, seat-of-the-pants test approach just because you are in a hurry. The point of the pyramid is to make sure that when the crisis hits, you can identify a level of risk that you are willing to accept.

As a practical example, which do you think it a more compelling argument: “If we ship now, we will only have completed 58% of our test cases,” or “If we ship now, we don’t know if our critical processes are still working?”

Remember that both testing and development ultimately serve the customer, and it is the risk or reward to them that measure the quality and value of a system. By following the testing pyramid, you can organize your test process around the customer instead of the development team. //

Linda Hayes is CEO of WorkSoft Inc. She was one of the founders of AutoTester. She can be reached at linda@worksoft.com.