After Katrina... Remapping Disaster Plans

Hurricanes Katrina and Rita changed the landscape of disaster recovery. What did we learn from these savage storms? What will we do better to prepare for next time?
In IT, there was an accepted rule of thumb that stated primary and secondary data storage facilities should be 25 kilometers apart. That was the rule until a few weeks ago, anyway.

The multi-state path of destruction of hurricanes Rita and Katrina has turned that rule on its head, forcing many IT managers to rethink their disaster recovery plans.

''We used to assume that disasters would not have encompassed 300 miles. Katrina was one of the broadest national disasters to hit the U.S. and it blew that right out of the water,'' says Andreas Antonopoulos, senior vice president at Nemertes Research in New York City.

Antonopoulos says a new rule of thumb is emerging that says IT managers should plan their off-site storage to be in a different geographic region than their primary data center.

''What we learned from Katrina is that in a disaster that big, you have failures across many facilities. You have to take into consideration that you have no water, no electricity, structural damage and no personnel. Normally, you take into account some of these conditions, but not all of them at once,'' he says.

For instance, he says companies located wholly in New Orleans were completely out of commission.

To avoid this scenario, experts say companies should redraw their plans to address remote storage and recovery facilities, as well as emergency staff to manage the process.

The first step to remapping disaster recovery plans is to decide what data has priority, according to Dave Kershen, practice manager for the data management group at Sun Microsystems, Inc. in Dallas.

''If you're a customer-facing organization, you're going to want to make sure your customer data is ready to go,'' says Kershen. ''It would cover the gamut from customer support, which is your most visible area, to backing up pieces of customer care, such as billing, crediting and invoicing. If you're a Web portal, then you need to make sure your Web hosting system is online and running.''

Taking the time to prioritize is key because of the high costs for remote backups. ''It's more expensive than doing it on your own locally,'' Kershen admits.

What drives up the cost are the service costs for storing the data outside your own data center -- whether you outsource to a remote backup service or collocate at a data center -- and the expense of frequently sending gigabytes of data across the wide area.

But Antonopolous says to look at remote backup costs as an insurance policy. ''You have to look at the risk you're willing to assume,'' he says.

He also recommends using information life-cycle management tools as a guide for determining what data to back up remotely.

''Your storage strategy should match your disaster recovery strategy,'' Antonopolous explains. ''Look at each piece of data and decide what level of freshness you need for it. Can it be on tape or does it have to be online and immediately available? Also, there are different rates of frequencies for backup that you can consider.

''Data can be stored in real time, daily, weekly or even monthly,'' he adds. ''The less frequent, the lower your cost. You need to make sure that the right data is matched to the right method.''

Allen Gwinn, senior IT director at Southern Methodist University in Dallas, says, ''I only back up what I can't afford to lose. That's my litmus test.''

Gwinn, who weathered the most recent hurricanes, as well as the great flood of 1995 that hit Dallas, is a big believer in off-site storage. ''You do have to have your data in at least two locations,'' he says.

He makes sure that staff and faculty email and file server files are backed up regularly and stored in several locations. He not only mirrors the files at a data center more than 10 miles away, but he also keeps a current copy of the data on 400G-byte external hard drives wrapped in weatherproof casing with an IT staff member in case of a forced evacuation.

But Gwinn stops short of sending his data out of state.

''Are you going to spend $15,000 to $20,000 to move data to an area that you'll probably never have to go to recover the data? That's a knee-jerk reaction. You look at the resources available to you and what money you have available to mitigate your risk. For me, it's the value verses the risk assessment to geographically diversify my storage,'' he says.

People Power

Experts warn that many IT groups fail to address a critical part of disaster recovery: loss of personnel. ''You may be able to recover all your data, but not all your people. For companies with a base that is geographically narrow, that is a tremendous risk,'' Antonopoulos says.

He adds that a flu outbreak could wreak just as much havoc as a hurricane if personnel issues aren't addressed in disaster recovery plans.

Antonopoulos recommends that companies with a geographically distributed base cross-train their staffs on disaster recovery so they can get the network up and running from an alternate location.

''For companies that have adopted encryption, the key escrow is a key component to this strategy. Your key management system should have widely distributed access -- it should not be dependent on a single person or a single region. There should be multiple copies of the keys,'' he says.

Kershen agrees. ''In New Orleans, there were companies that lost their data and their encryption keys and they were doing backflips to get their data restored.''

He encourages his clients during implementation and integration to ''ensure there are a number of people who have authority and access passwords. We highly recommend they be from alternate data centers in alternate geographic locations.''

''I think some of the best disaster recovery plans involve duplicate job training exercises across geographically diverse locations,'' says Kershen. ''The staff has the ability to cross-train and while it may not be their primary function, they could limp along in case of a disaster.''

Kershen says companies need only look at the financial impact of a disaster to see the viability in remapping their off-site storage and recovery plans.

''Every hour of downtime in the energy industry, for instance, could equate a million dollars in lost revenue opportunity,'' says Kershen. ''Those numbers are scaring people into action. And the scope of disasters is much larger. We're seeing whole cities being taken out of service with prolonged outages and that changes things. A stand-by generator isn't going to cut it.

0 Comments (click to add your comment)
Comment and Contribute


(Maximum characters: 1200). You have characters left.