Shrink-Wrapped Testing: A False Sense of Security?

I was chatting with a customer the other day when the topic of testing their firewall solution came up. During the conversation I asked who he was going to use to do the testing. When he pointed toward the systems administrator, a capable individual, but one who has had nothing to do with firewalls, I was a little taken aback. Much as I had faith in this individual to create a user account or even install an operating system, the thought of him testing the corporate firewall was as scary as my next date with the dentist.

What, I asked in the steadiest voice I could muster, made the client think that its junior administrator was capable of testing the firewall? Instead of a justification of the administrator’s technical skills, the client simply reached into a cardboard box and produced a shrink-wrapped package from within it. The package contained firewall testing software, and the sticker on the box proclaimed among other things, that it was ‘usable by those with only a basic understanding of firewalls’. It would seem the claim was about to be put to the test. The client informed me the software was the same as that used by an outside security consultant during his last visit. In effect, the client resented paying the fees charged by the security consultant for using the same software that he could buy and operate himself.

As the threats from outside sources have increased in their complexity, the ease with which our security systems, such as firewalls, can be tested has increased also. Packaged software now enables us to test security solutions and determine their effectiveness with ease. But can using a packaged software solution really offer the protection that’s available from a dedicated and specialized security provider? To put it another way, does this kind of software literally lure us into a false sense of security?

Competence Cannot Always Be Shrink-Wrapped
In a sense, there is no reason why testing a security solution should not be as simple as point and click. Most of the other things we do on a daily basis are done the same way. Perhaps the bigger issue is that while the software to test our security solutions may be simple and easy to use, are those doing the pointing and clicking able to effectively test, and , just as important, interpret the information produced from such a test? In addition, are they able to act on the information produced from the test to correct the problem? Given that in many cases the person conducting the test is the network or server administrator, you have to wonder whether the task is not more suited to someone who does it for a living.

In IT, as in any other field, but particularly in security, real world experience makes such a difference. In a given year, a system administrator may have to deal with one or two security incidents. A security consultant will most likely deal with more than this on a single day. The knowledge and experiences gained from this intensive exposure allows security consultants to develop finely honed skills in both risk assessment and identification. They are far more able to thoroughly test a security solution than someone who has just read the instruction manual for a software package.

That is not to say that using a security consultant is foolproof. Not all security consultants, or consultancies, are created equal. As much time should be invested in choosing a security consultant as choosing the security solution in the first place. The introduction of certifications programs by a number of the leading security software vendors can lead you to believe that holders of these certifications are competent and knowledgeable, but it is not a guarantee. In the same way that there are inexperienced and ‘paper’ holders of other certifications, security certifications are no different. The exact skills which are so important in the work environment — up to date knowledge and hands on experience — are the two hardest things to incorporate into a certification test.

As mentioned earlier, that is not to say that certified individuals are not competent, but the value of the certification can only be estimated when it is backed up by practical on-the-job experience.

Understanding The Risks
Through this experience, security consultants are able to understand business issues as they relate to security. Understanding the risks is actually a step that comes before any kind of remedial actions, as Allen Vance, Vice President of offer management for Internet Security Systems, a leading provider of security testing software, points out.

“First, customers must understand what kind of business level risks they have,” says Vance. “A bank will have different associated risks than, for example, a baker. Next, you have to determine whether you have the appropriate skills to manage the solution in house. In each case, not determining your needs or understanding the requirements fully will most likely prove to be a false economy.” Vance has the luxury of providing an impartial view on the subject, as customers of ISS fall into both camps.

If, at the end of the day, you have the skills in-house and understand the risks that you are protecting against, using testing software and performing your own checks may be a valid approach. But if there is any doubt, use a suitably qualified and experienced security consultant. As one veteran security consultant puts it, you could save a few bucks and it could cost you your business.

Compelling argument, isn’t it?

Drew Bird writes for CrossNodes, an internet.com site where this story first appeared.