But remedying this has not proven easy.
After spending as much as $170 million on custom software to address this issue, the FBI announced on Jan. 13 that it might have to scrap the software and start over.
''The FBI's long-anticipated Virtual Case File has been a train wreck in slow motion,'' Sen. Patrick Leahy, (D-Vt.) said in a statement released that day.
On a smaller scale, IT departments are having trouble managing their own security data.
''Everything was a mess,'' says Jim Patterson, security analyst for the State of Illinois Legislative Information System (ILIS). ''Even from a big vendor like Cisco, each device had its own reporting console, and there was no way to have a central point to manage them.''
To make things worse, each device didn't just require its own software, but a dedicated PC in order to avoid running into conflicts.
''We would try to gather data from all these different devices, but there was no way to correlate the information,'' Patterson adds. ''The Cisco Pix firewalls were generating so much traffic -- millions of messages a day -- that the little built-in SQL server couldn't even handle it.''
At Cisco's recommendation, ILIS turned to a new type of management software called Security Information Management (SIM), installing nFX software from NetForensics, Inc., a company focused on the SIM market and based in Edison, N.J. This allowed Patterson to aggregate all the security information into a single database for analysis and alerting.
''The netForensics has a realtime event console with a scrolling display of what is happening,'' says Patterson. ''This reduces the number of people you need to have monitoring security since everything is on one central location.''
SIM is an outgrowth of network log management software adapted for use with security devices and software, including firewalls, Intrusion Detection Systems (IDS), Intrusion Prevention Systems (IPS), authorization software and anti-virus. One of the main drivers has been simply being able to make sense out of the huge amount of data that these devices spit out on a regular basis. It is impossible to manually go through the millions of messages and gain a clear concept of what is happening from a security standpoint.
''Many times, this is driven by a failed IDS project that dumps out too much data to effectively interpret,'' says Paul Proctor, vice president of Security and Risk Strategies for META Group, an analyst firm based in Stamford, Conn. ''IDS implementations fail because organizations do not tune them properly, not because they inherently produce too much data.''
Of course, when you have devices generating that much information, it can be hard to properly tune the devices, thereby reducing the number of messages.
Patterson says having a SIM has enabled the state to fine tune its firewalls and sensors.
META Group's Proctor advises that organizations shouldn't start out with the goal of cutting down on what they have. Instead, they should start by determining what they need.
''A more effective approach is to start with a detection requirements list tied to business needs, and then determine which events need to be collected to support those requirements,'' he says. ''If you take this approach, SIMs can have value.''
A Global View
Installing and configuring a SIM can be a major undertaking. Patterson says the Illinois system started out running its SIM on a single server, but found that as the number of security devices increased, he had to split it up. He is now monitoring 30 devices at three sites.
The netForensics software resides on three low-end Dell dual-processor servers -- one for collecting the data, one for the Oracle database, and a third for reporting and analysis. Larger installations require much more.
''When they embark on a large SIM project (with more than 300 audit sources/nodes) they should put aside at least $50,000 in their services budget for the vendor or a competent third-party to come in and install, and tune for appropriate business requirements,'' says Proctor. Deployments with more than 1,000 nodes are usually multi-year efforts, so set realistic expectations and project goals.''
Unisys Global Infrastructure Services of Blue Bell, Penn., for example, has three security operations centers. One is in Blue Bell, Amsterdam, another is in the Netherlands, and the third is in Wellington, New Zealand. The centers provide the company, which has 200 managed security clients, services around the globe.
Unisys Global began deploying a SIM from ArcSight, Inc. of Cupertino, Calif. in June f 2003. The final roll out will be completed this year. Once this is completed, the security analysis will be performed at three levels -- customer, regional and global. Having the global system in place lets them spot a problem in one area of the world and take action to harden security in others areas before they are hit.
''It has proven useful in helping to detect the zero-day threats out there before there is a signature available for it,'' says John Summers, Global Director for Managed Security Services. ''Our European operations center, for example, found a particular threat, what the network traffic looked like, what ports it was talking on, and we wrote a specific correlation rule to monitor data on those ports.''
Summers says that having a SIM has two main values.
To begin with, it enables them to do complex pattern detection across a heterogeneous infrastructure. This has been useful in spotting blended threats which seek to exploit multiple vulnerabilities.
The other benefit is that it is able to reduce the number of false positives, allowing them to accurately spot the true threats.
''With IDS or any security device, you get way too many messages coming in, so to handle it, people turn down the gain on their sensors so they put out less noise, but also put out less signal,'' says Summers. ''But an event correlation platform allows you to turn the gain up again and gives you a more accurate ability to detect suspicious or bad activity.''