The next wave: Testing end-to-end

If you aren’t dealing with end-to-end testing now, you will be soon. As software developers assemble systems from components and integrate those systems across platforms and even other companies, they are becoming increasingly aware that traditional, application-centric testing no longer suffices. The boundaries between applications are blurring, and changing components at any level can have far-reaching effects that are both high risk and difficult to predict. A single application programming interface (API) layer, for example, that services requests to a central database can support–and affect–applications across an entire organization.

In response, companies are beginning to make substantial investments in building enterprise-level test organizations and environments. Like the “model office” of old, this is a “model enterprise,” in which companies stage and test critical systems together before promoting those systems to production. This wave of investment is creating a sea change in software quality as it graduates from a technical discipline to a business discipline. The cumulative effect is a tidal wave poised to break.

What is end-to-end testing, anyway? What does it mean? How do you do it?

Horizontal and vertical

There are two ways to do end-to-end testing. The most commonly understood meaning of end-to-end testing is that testing occurs horizontally, across multiple applications. For example, a Web-based order-entry system might interface to the back-office accounts receivable, inventory, and shipping systems. A horizontal end-to-end test includes verifying transactions through each application, start to finish, assuring that all related processes occur correctly. For example, an order item list is properly removed from inventory, shipped to the customer, and invoiced. This level of integration might exist within a single enterprise resource planning (ERP) application, but most companies have a mixture of applications acquired from different vendors or internally developed at different times.

Vertical end-to-end testing refers to testing each of the layers of a single application’s architecture from top to bottom. For example, the order-entry system might use HTML to access a Web server that calls an API on the transaction server, which in turn generates SQL transactions against the mainframe database. Other applications may share the API and SQL components, so those interfaces must be tested both individually and vertically in an end-to-end environment. This is a special challenge, because intermediate layers such as API or SQL middleware are “headless” in the sense that they have no user interface that lends itself to the usual manual testing fallback. As a result, you must either test these layers through the various applications that they support–a daunting task because there can be so many of them–or you must write an artificial front end, thus creating yet another development project.

Strategy

Out of necessity, testing across different applications and layers must involve many different areas of the enterprise. Yet, it only takes a single group to establish and control the test environment and process itself.

The first step is to establish and fund the model enterprise team. This group will provide the planning, scheduling, and entry/exit criteria for the end-to-end test environment. The team must also garner enough funding to acquire and maintain the hardware and software infrastructure necessary to model the enterprise delivery systems closely enough to simulate real-world conditions. Granted, there need not be 100% duplication, but there must be a meaningful subset that can properly exercise critical functions.

The second step is to build the test environments. Your company will most likely require more than one test environment to accommodate the set of normal transitions applications make from the development through the test cycles and the final staging area before production. Then you need to define the entry criteria for each application or layer in each environment. For instance, establish when a system or application is ready to be installed in one stage, what qualifies it to be migrated to the next, and, most importantly, how you know it is ready to graduate into production.

The scale of this effort is so vast that the only way to start is to prioritize your applications. You must start with the company’s core systems, such as inventory and manufacturing. Then work your way up the chain. Because you will not be able to bring up all systems at once, develop a plan for simulating some interfaces. For example, if you use a barcode reader to add goods into inventory, you will likely need to substitute the actual activity with a file of items. Also, identify the systems that must be tested cooperatively with a third-party provider or vendor, and make arrangements to share the effort whenever possible so that you can approximate real-world conditions.

Finally, come to grips with what your priorities are: Find out which systems can be tested in the model enterprise environment, and which systems can bypass it. I have seen too many efforts fail because of an all-or-nothing approach to end-to-end testing. The best question to ask yourself is this: Which systems can be pulled out of production without crippling critical, ongoing operations? Any system that cannot be retracted should be part of the model enterprise.

The most important strategy of all is to enlist the support of top-level management, and educate them on the essential interconnections across all of the applications and layers that comprise the enterprise. Make sure someone in authority understands that it doesn’t matter whether all the pieces work if all the pieces don’t work as one. //

Linda Hayes is CEO of WorkSoft Inc. She was one of the founders of AutoTester. She can be reached at linda@worksoft.com.