Datamation content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.
When Hurricane Sandy struck the Atlantic coast in October 2012, Allied Building Products’ data center in New Jersey was submerged in four feet of water in a matter of minutes. The facility was completely wiped out, and it was three months until a new one was up and running.
But Allied had a well-tested and effective disaster recovery/business continuity plan in place. The company’s operations were rapidly switched to a SunGard AS disaster recovery facility in Philadelphia. Servers and applications were brought back online, and the company’s IT infrastructure continued to operate from there for another three months until they could be switched to the new Allied data center.
Despite exhaustive planning and preparation, the recovery operation wasn’t quite perfect, Scott Fisher, Allied’s director of IT operations, admits. “There were some things that we hadn’t thought seriously enough about, because nobody ever really thinks that a disaster will happen to them. We thought that we would never lose some applications or that we could live without them if they did fail, but it turned out that we really couldn’t,” he says.
Fortunately SunGard had sufficient extra hardware at its facility to get these applications up and running anyway, so this turned out to be a non-issue for Allied. But the episode illustrates a simple fact: disaster recovery plans are just that – plans. Which means you can never be completely certain that they will be sufficient to prevent your firm from becoming one of the 43% of companies that, according to Gartner, never reopen following a disaster.
Roberta Witty, a business continuity expert at Gartner, says that simple things that were overlooked before Sandy undermined many companies’ disaster recovery efforts after the disaster struck. “Some plans called for people to work from home, but in the event they couldn’t because the power outages were so widespread,” she says.
Other plans failed because companies failed to anticipate the duration of power outages caused by the hurricane. “Companies were prepared for a three day outage, rather than a five day or more one, so they needed more equipment, data backup, power – more of everything,” she says.
But it often takes simple steps to help ensure disaster recovery plans are effective, she adds. “Proactive companies had declared a disaster by that Friday and got staff in other locations in advance,” says Witty. (The storm surge from Sandy hit New York City the following Monday.)
To maximize the chances of your disaster recovery plan being successful, here are ten things you should be considering:
1. Focus on your backups
It doesn’t matter how good the rest of your disaster recovery plan is if you don’t have the right data to restore. Potential problems include having backup data that is not fresh enough, data that is corrupted, or discovering that certain applications’ data haven’t been backed up.
Don’t forget that in the event of a major disaster like Sandy your backup data may not be available if it’s stored nearby. In planning for a disaster, make sure your backup data is copied to a geographically remote location
2. Ensure that staff members are involved in planning
General Eisenhower once said “In preparing for battle I have always found that plans are useless, but planning is indispensable.”
The same is true with disaster recovery planning: key staff need to be involved in the creation of a plan, as it is the planning itself that is the basis of disaster recovery.
3. Test as often as practical
Testing a disaster recovery plan is vital to see whether it works in controlled conditions. (But don’t forget that testing a plan doesn’t show you that it will actually work at 4 AM during a hurricane.)
Testing frequently works as a drill or practice session, ensuring that staff gets familiar with procedures and what they are expected to do. And testing as often as possible is your best chance of discovering things that have been overlooked – such as new hardware, applications or dependencies that haven’t yet been incorporated into your plan.
The drawback is that testing takes up staff time and costs money – especially when disaster recovery plans involve third party facilities. If that’s the case, then you may need to book testing sessions in advance. Expect to pay more if you plan to carry out more than one test a year.
4. Ensure the plan is well documented
Disaster recovery plans can easily fail if the plan’s procedures are not described in sufficient detail. That’s because during tests, plans may be executed successfully even if they are not sufficiently detailed. Staffers are able to draw on their own knowledge of systems, applications and infrastructure. But in the event of a real disaster these staff may not be available.
The bottom line is that successful tests may give you the illusion that a plan can be implemented in a disaster, when in reality it may not be possible because it contains insufficient detail.
5. Revisit your disaster recovery plans as often as possible.
From the moment you finish a plan, your production and recovery configurations begin to diverge as equipment, applications and staff change.
That means that your recovery plan is out of date as soon as it’s completed, and that’s why it’s vital to review and update it on a regular and frequent basis. Once a week may be excessive, but once a year is certainly not frequent enough.
6. Consult widely about what is important
Business continuity is all about ensuring that you have the systems, applications and data in place to carry on business after a disaster, with the most important things back online the soonest.
A common mistake is to underestimate your dependency on certain applications or data and then to discover that they are not available quickly enough after a disaster. That’s why it’s important to remember that business can only resume when you have everything that you need.
7. Ensure you have enough power
Disasters can have effects – like power blackouts – that last much longer than the disaster itself. If your plans involve generating your own power then it’s vital to ensure you have enough fuel to do so for several weeks, not just days or even hours, so you can keep going for as long as the interruption lasts.
8. Don’t forget communications
If a disaster strikes then your plan may require staff to call remote locations – either your own facilities or those of third parties – to put the disaster recovery plan into operation.
But in a real disaster, staff may not be able to access fixed line or mobile phone systems as easily as they can in a test scenario. That means your plan should detail how communication can be carried out in the event of mass outages in as many ways as possible. Many companies issue key staff with satellite phones for this purpose and also plan to use social media to get messages to staff.
9. Don’t get locked out because passwords are unavailable
Your disaster recovery procedures may work perfectly, but don’t forget that the systems have to be usable once they are up and running. That means that staff needs access to passwords and other authentication credentials at disaster recovery sites.
Failure to ensure systems passwords are available at these sites will make your business continuity plans unworkable.
10.Make sure the plan is available
The final thing to remember is that staff needs the plan to execute it. Be sure that an up-to date-copy of all relevant documents is available at multiple locations so that when disaster strikes it’s easily and immediately available to those who need it.
Photo courtesy of Shutterstock.
