Creating a Resilient IT System

If something knocks your system off line, will it immediately come back up and work as planned? If not, you've got some planning to do.
When we look at how important IT systems have become to organizations and society as a whole, we need to factor in resiliency when designing them.

Resiliency pertains to the system's ability to return to its original state after encountering trouble. In other words, if a risk event knocks a system offline, a highly resilient system will return back to work and function as planned as soon as possible.

While many may take this process for granted, not all systems recover cleanly. Sometimes IT staff isn't even involved. But other times a lot of staff members are involved and they're dealing with a great deal of stress getting that system back online. And whether all the data is there or not is another story altogether.

From now on, we cannot afford fragile systems or systems that require an unmanageable amount of time and effort to recover. We must take resiliency into account.

How many systems do you have that will come back online if the power is cut and the UPS runs to the point of exhaustion, causing a hard crash? Stand-alone PCs and network devices are usually pretty good about coming back. However, as the level of complexity and interdependency increases, simply coming back online after a hard crash may mean corrupted storage, split clusters and the failure of dependent services.

That means you shouldn't bet on highly complex systems simply coming back up after whatever negative event you experience -- be it hardware or software failure or some form of security incident.

From an organizational perspective, if power is lost at a plant for two days, can it recover? If a key service is lost because a database becomes corrupt, can the business recover? Organizations that can bounce back are resilient and the ones that can't may have some troubled times.

Making your system resilient takes a lot of planning.

To build resilient systems, you need a holistic mentality. Prioritize every foreseeable risk and then determine not only how to reduce the risk in the first place, but determine how to minimize its impact on the system and the organization. Those are two different issues.

Granted, recovery controls, also known as corrective controls, are risk mitigating controls. However, we need to make sure that teams managing systems take into account not just controls that reduce the probability of a risk event, but also reduce the impact of the event. They must plan for failure, not optimism.

Resiliency directly targets minimizing the impact by bringing people, processes and technology either back to their original state or a modified state until the risk has been reasonably addressed.

Any system has three dimensions that must be considered -- people, processes and technology. To build in resiliency, all must be taken into account because if one of them fails, then the likelihood of poor resiliency and overall system failure increases.

In addressing the 'people' dimension, there must be identified backups and cross-training to ensure that if anyone is sick, on vacation or incapacitated, there is another person, if not entire other teams, who can fill in. For example, if a data center is damaged due to a natural disaster and the staff there is trying to address their legitimate family crises, is there another group that can do the work from another site and take some of the pressure off?

When addressing process issues, IT administrators must spend some time assessing.

Are the current processes so rigid that they break under any variation? Or are there logical emergency processes that can be triggered in the event of a problem?

Technology is interesting. If you have the right people and the right processes supporting it, then magic happens. If either element is weak, the technology is standing on a bad foundation.

With that said, resiliency can apply directly to the technology. To illustrate, some systems are very sensitive to power or temperature fluctuations. If there is a high risk that neither of those elements can be reasonable safeguarded, then you have a fragile system -- one that will be prone to break during regular operations, let alone in an emergency.

Be sure to consider environmental and other risks when evaluating systems and subsystems. Be sure to factor in resiliency during the evaluation. What if the power spikes? What if there is a brown out? What if the room temperature goes to 100 degrees Fahrenheit for 48 hours? How will any of these risks affect the system's ability to recover following?

Again, the key is to identify risks, prioritize them, and then figure out how to mitigate the most likely ones in an effective and cost-efficient manner.

0 Comments (click to add your comment)
Comment and Contribute


(Maximum characters: 1200). You have characters left.