The terrorist attacks of Sept. 11 have taught many CIOs and CSOs one thing -- prepare for the unexpected.
Dave Johnson, director of technology at Grant Thornton, a global accounting, tax and business advisory firm based in Chicago, can testify to that. Johnson and his IT team had spent years transforming the company's network architecture from a disarray of decentralized offices running a hodgepodge of servers, PCs, notebooks and switches into a unified national network focused on four regional hubs and a major centralized data center.
His mirrored, redundant system was prepared to handle the loss of a hub. But Johnson always thought a hub would go down because of a callous worker with a backhoe or maybe even an earthquake. He never counted on a catastrophic terrorist attack taking out data and voice connection with his New York hub that sat only four blocks from Ground Zero.
"We designed it to address an outage," says Johnson, whose Chicago hub picked up the New York-bound network traffic within six minutes of the hub's outage. "We had been living day-to-day with a redundant environment...I just never thought it would be an outage caused by this type of event."
But with that experience behind him, Johnson says the one thing he has learned is that he built his network the right way. He set up mirrored systems. He set up redundancies. He set up automatic hub backups. And he trained his IT workers to be able to be able to switch roles and work together, no matter what part of the country they're working in.
And industry security experts say Johnson definitely is on the right track. They just wish more IT managers would follow his lead, charging that the IT security fervor that erupted last fall was soon replaced with complacency born of too few IT workers to handle big change-over projects, budgets without enough money to fund new systems and rebuilds and a lack of expertise to put plans into motion.
"You've got to approach data recovery in a multi-tier manner," says Tom Hickman, engineering operations and quality assurance manager of Connected Corp., a Framingham, Mass.-based PC data protection company. "If you can't get your critical systems back online in 24 hours, there's a good chance you'll no longer be a company."
Hickman, Johnson and Dan Woolley, a vice president at Reston, Va.-based SilentRunner, a network security firm under the umbrella of Raytheon, say there are basic steps that IT managers should be taking to prepare their company for any kind of outage -- whether it be a terrorist attack or a wayward worker with a backhoe.
Here are a few of their recommendations:
Analyze your business and rank your business needs in terms of priorities;
-- What business needs have to be back up and running within minutes or hours?
-- What can be down for 24 hours?
-- What services or functions could be down for a week or two weeks?
Identify and prioritize risks, such as natural disasters, insider threats and physical, as well as cyber, terrorist attacks. Develop plans and policies to lesson those risks;
Put together recovery teams with defined personnel, roles, functions and hierarchy;
Do you have the backup systems in place to keep the business running?
Are backups automatic and done in periodic intervals, such as every minute, every hour or every day? Figure out your backup priorities. Not everything needs to be backed up every hour but some things should;
Don't have all your IT people working in the same place. If something happened to that building, you would lose all of your talent and it would cripple your ability to get the network up and running again;
Figure out your biggest over potential failures. Is it power, Internet access, a building or phone lines?
If you're a software company, is your source code spread around so if something happens to one server or one location, all isn't lost?
Map out your network. Know what's on it and where it's located.
Know the systems running on your network;
Know where workers are supposed to be located and have contact information for them and their families. Set up notification procedures, such as calling trees;
Have contact information for business partners, contractors, consultants and vendors at hand;
Set up generators to keep your electronics functioning;
Keep backups off site;
Have mirrored data centers and servers;
Do you have a backup ISP or teleco?
Set up escape routes for your buildings and make sure employees are familiar with them;
Establish quick-ship programs with vendors to get the equipment you might need to replace to get back in operation;
Plan for remote access needs in case you suddenly can't work onsite.