Quality Quest: Automated vs. Automatic

No matter how hard you try, you can't avoid the first and most important step in the software testing process; the gathering and defining of requirements that will be tested. This step requires intelligence and skill that no automated testing tool can provide.
Posted November 14, 2000

Linda M. Hayes

(Page 1 of 2)

Linda M. Hayes

Often when people contemplate the idea of automated software testing, they make the mental leap to what I consider software testing's "Holy Grail": automatic testing. In other words, it's not just the automated execution of predefined tests, but the automatic generation of the tests themselves. Not only hands-free, but brain-free.

Automatic software testing takes one of two forms--generating test cases from requirements or generating requirements from the software itself. The first one is possible, though not effortless. The second one is not possible, and even if it were it would be pointless.

Automated Test Cases

What a great idea, a tool that can generate all the necessary test cases from the application's requirements. Notice the word "necessary" as opposed to the word "possible." If you generate all possible test cases, you are likely to generate an unmanageable and infinite number of possibilities in even straightforward applications. Generating only the necessary test cases means you have selectively identified those test cases that are unique. This is the only way to achieve complete coverage with manageable volumes. Caliber RBT, a testing system developed by Atlanta's Technology Builders Inc., can generate all the necessary test cases to assure complete coverage when given the application requirements. It employs a proven testing model known as cause-effect node graphing, which has its roots in the testing of hardware.

For this approach to work, however, you need to do two things. First, you must commit to specifying your application's requirements with enough mathematical precision and internal integrity to support the testing model. Second, you must develop the test scenarios, or pathways through the application, to execute the test cases. For example, you must create a test scenario that has the steps needed to navigate through the application as well as provide the inputs and verify the outputs that comprise the test case.

Although this is a worthy and achievable goal, it is not "automatic" software testing.

Automated Requirements

What an even better idea--have self-testing software that creates its own requirements and then generates the test cases accordingly. That tool would be a tool that examines the software source code, divines what it is doing, then generates the necessary test cases. These test cases are needed to exercise every pathway and every input or output, including boundaries, ranges, types, and even random values. If this idea sounds too good to be true, that's because it probably is.

At the most basic level, requirements define what the application should do, not what it does. So, if you derive the requirements from the software itself, then you have a completely self-referencing model. Obviously, this means that if the software is missing features or has implemented them improperly, then any self-generated requirements will likewise be incomplete or incorrect. Said another way, if the students write the test, they will certainly pass it. But that doesn't mean they have mastered the subject.

Even if you dismiss these testing inconsistencies by saying all you want to do is make sure the software does what it's designed to do without causing errors, you then have a different problem. The vast majority of an application's functionality does not exist in the source code as static information. Given that most modern applications rely heavily on multiple interfaces, components, and objects, and may take advantage of concepts like inheritance (where objects are reused and also extended or modified), it is impossible to determine what the actual behavior will be until execution occurs. And, although there are ways of tracing through the source code during execution, to effect execution you have to drive the system. This means you must supply some form of input actions or data. So you are back to the original testing problem--you must know what test cases you want to execute before you can trace the code.

Page 1 of 2

1 2
Next Page

0 Comments (click to add your comment)
Comment and Contribute


(Maximum characters: 1200). You have characters left.