The challenge of Big Data is a daunting one. We all know that data is exploding, but just how much data is out there? No one is certain, but former Google CEO Eric Schmidt has argued that we now create an entire human history's worth of data every two days. "There was 5 exabytes of information created between the dawn of civilization through 2003," Schmidt said a couple of years ago, "but that much information is now created every two days, and the pace is increasing."
Those numbers may be exaggerated. RJMetrics CEO Robert J. Moore said in a TEDx talk recently that "23 exabytes of information was recorded and replicated in 2002. We now record and transfer that much information every seven days."
Gartner believes that enterprise data will grow 650 percent in the next five years, while IDC argues that the world’s information now doubles about every year and a half. IDC says that in 2011 we created 1.8 zettabytes (or 1.8 trillion GBs) of information, which is enough data to fill 57.5 billion 32GB Apple iPads, enough iPads to build a Great iPad Wall of China twice as tall as the original.
According to IBM, every single day we create 2.5 quintillion bytes of data. IBM argues that the exponential growth of data means that 90 percent of the data that exists in the world today has been created in the last two years. "This data comes from everywhere: sensors used to gather climate information, posts to social media sites, digital pictures and videos, e-commerce transaction records, and cell phone GPS coordinates, to name a few."
Of course, it's important to remember that in early human history, anything as ephemeral as a tweet just would not have been recorded, so these comparisons can only be taken so far.
To put the data explosion in context, consider this. Every minute of every day we create
Another challenge facing Big Data analysts is the fact that data is stored all over the place, in different systems. Breaking down data siloes is a major challenge. Another is creating Big Data platforms that can pull in unstructured data as easily as structured data.
When you get into the Big Data weeds, though, more arcane challenges emerge. For instance, traditional databases were not designed to take advantage of multicore processors. Thus, they are much slower at processing data than they could be, which has led to the concept of "Fast Data," with startups such as ParStream attempting to overcome various legacy issues associated with databases.
Whatever the exact number, we have a lot of data to contend with. Accumulating data is one thing. Doing something with it is another. You wouldn't refer to a hoarder who accumulates old newspapers, empty tuna fish cans and live kittens as a "discerning collector," after all. You wouldn't visit a hoarder's house to learn about history, the way you conceivably could from, say, an antiques collector. The signal-to-noise ratio is just too low.
With data, though, the world is full of hoarders. Digital storage is so cheap that people store everything—or, more accurately, don't bother to delete anything. The same is true online, where online storage vendors now routinely give away GBs of data storage before charging a thin dime.
Today, businesses are struggling to contend with this out-of-control data sprawl—because if they don't, they won't stay competitive.
According to IBM, exponential data growth is leaving most organizations with serious blind spots. IBM found that one in three business leaders admit to frequently making decisions with no data to back them up. Their decisions are either based on information they don’t have or don’t really trust. Even more surprising, one in two business leaders admit that they don’t have working access to the information they need to effectively do their jobs.
Most business leaders and knowledge workers know that relevant data is out there, but they don't know where. Even if they have a rough idea, they're not sure how to extract it in any meaningful way. Finally, once they manage to find relevant data, they often aren't sure how current or accurate it is.
This is where Big Data analytics comes in. What we're after isn't just raw data. We want the knowledge that comes from analyzing that data.