The Surprising Truth about Big Data

Datamation content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More .

Big Data gets a lot of headlines. If any technology can be called heavily hyped, Big Data earns the prize for most breathless predictions of enterprise influence.

Typical of the rosy predictions is this from IDC: spending on Big Data-related infrastructure, software and services will grow at a torrid compound annual rate of 23.1 percent between 2014 and 2019, reaching a hefty $48.6 billion in 2019.

For companies, it would seem, boosting revenue as easy as implementing a Big Data solution and hiring an accountant who can track your windfall profits.

However, a research report from Dresner Advisory Services adds a sober note to the Big Data hype. Dresner is helmed by Howard Dresner, who understands the market: he’s widely known as “the father of business intelligence” (having coined the term). He led Gartner’s business intelligence research practice for 13 years.

Dresner’s research report, from November 2015, includes input from roughly 3,000 organizations, as well as crowdsourcing and vendors’ customer communities. The survey reports that:

* A mere 17 percent actively use Big Data in their organization today.

* A lukewarm 47 percent “may use” Big Data in the future.

* A remarkably large 36 percent have no plans for Big Data. In essence, fully a third of companies say: Big Data, who cares?

The Dresner survey includes this downbeat note: “despite an extended period of awareness building and hype, actual deployment of big data analytics is not broadly applicable to most organizations at the present time.” (Italics mine.)

big data adoption

Source: Dresner Advisory

And yet the results reveal a conflicted attitude. Despite lackluster adoption, a full 59 percent say Big Data is “critically important.” Huh? That’s contradictory: A majority says it’s critically important, yet a skinny 17 percent actually use it?

Clearly, executive’s attitudes toward Big Data contain big ambiguities.

Big Data adoption is greatest among large businesses and institutions. Big firms have the deep pockets required to deploy competitive technology early in the game. Given a size of more than 5,000 employees, 36 percent now use Big Data.

Yet that still begs the question: what about the 19 percent of large companies with no plans to use Big Data? Can they keep up with competitors that are leveraging insights from Big Data?

Defining the Market

The Dresner survey acknowledges that it contradicts other studies that tout Big Data adoption. “We recognize that this finding contrasts with other market studies and agree that there are powerful use cases for big data that may not be immediately suitable for supplanting existing resources and technologies such as SQL.”

Other Big Data experts echoed Dresner’s assessment of the market.

Fern Halper, Research Director for Advanced Analytics at TDWI, and a co-author of Big Data for Dummies, said the Dresner report “is probably directionally correct.”

She noted that a TDWI survey that focused on readiness for Big Data found that less than 10 percent of companies felt they could manage petabytes of data. A much higher percent, of course, have experience managing terabytes. Further complicating the issue, many companies are handling structured data, “but we know that most data is disparate data,” she said.

James Kobielus, IBM’s Big Data evangelist, concurred that Dresner’s numbers are probably in the ballpark. However, the Big Data market is not as stuck in infancy as the survey suggests, he said.

Assessing Big Data market adoption rates, “depends on where you set the threshold,” he said. How you define the Big Data market defines how you measure adoption.

If you set it down at one terabyte, you already have broad adoption, Kobielus said. “Very large data sets for analytic applications, within enterprises, have been in adoption for quite some time.”

“Now if you scope it more narrowly, to simply Hadoop – one platform – then that’s another matter altogether,” he said. “If you’re talking about some specific, new technology that does low latency data integration, like Spark streaming, that’s a fairly immature market.”

Why the Foot Dragging?

I asked Howard Dresner why companies have been slow to adopt Big Data. The low adoption is not an indictment of Big Data technologies per se, he responded. Rather, companies face an array of obstacles, including:

* Inertia: Existing investments in technology and skills make moving forward difficult.

* The status quo is working: Generally-speaking, existing technologies are getting the job done.

* Learning curve: Big data skills and knowledge are lacking; external resources are costly.

* Brave new world: Big data ecosystem dynamics and open source philosophy are intimidating to some establishment IT folks.

Dresner’s comment about the lack of skills was echoed by Stephanie McReynolds, VP of Marketing for Big Data start-up Alation. Low adoption rates reflect “the challenge of having enough tooling around Hadoop,” she said. “Right now you kind of have to be an expert to use Hadoop.” Using Hadoop for actual analysis – not just ETL (extract, transform and load) – requires expertise. A company needs a data scientist or someone comfortable with coding to access that information. “We’re on the early edge of making it easier for the non-technical specialist to work with Hadoop data, and integrate with the database and the BI tools that they’re already familiar with.”

Halper said that, in TDWI’s surveys, the biggest impediment to Big Data adoption is always “‘We don’t have the skills’ and ‘we don’t have the staffing.’” She added: “Hand in hand with that, they can’t get the budget, and they can’t get executive support.” The executives don’t always feel the business case is strong enough to justify budget.

“Maybe they’re not seeing the actual business case for it, which is what [Big Data] is – as opposed to build it and they will come.”

The Other Side of the Coin

Despite the lagging adoption of cutting edge Big Data, one thing’s sure: any technology that offers a big competitive advantage will – with time – be deployed by companies that can afford it.

A chart from the Dresner suggests that 2016 may be a major growth year for Big Data. Remember, 47 percent said in November 2015 that they had not yet adopted Big Data, but had plans to. The chart below shows that, of this 47 percent, 4 percent expected to adopt by year-end 2015, and an additional 27 percent plan to adopt in 2016, for a total of 31 percent.

To summarize the math (roughly): about 15 percent of all companies are on the cusp of adoption this year. Given that only 17 percent have currently adopted, this additional 15 percent represents a near doubling of current Big Data adopters.

Simply put, a big move is now happening in Big Data.

vmworld

Source: Dresner Advisory

Halper has seen the shift toward Big Data firsthand. “A couple of years ago, people would say ‘What’s Hadoop?’” and that has really changed, she said.

She has also seen the growing interest in Big Data, with plenty of optimistic plans. So much excitement, in fact, that maybe not all of it will actually materialize into true adoption, she said. “We’re on the cusp [of greater adoption] – if people stick to their plans.”

In this related video, I moderate a discussion of Big Data with James Kobielus, Stephanie McReynolds and Fern Halper: