Datamation content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.
As we run our businesses, what do we trust more: Big Data or our own gut instinct?
Big Data is awesome in its capacity to digest reams of complex information. But it is weak in its ability to weigh nuanced variables.
Human instinct is frail and biased in computational power. Yet it is amazing in its ability to combine nuanced variables and make an exponential leap.
It’s a tough challenge facing businesses today: do we trust the data analytics trend lines, or rely on the hard-earned wisdom of human managers?
The answer, of course, is usually “some mix of both.” But it’s not that easy. What’s the mix? And how to combine them?
Untold revenue gains rest on answering those questions correctly. Maximizing profit requires making the best decisions (and executing on them, of course).
To provide insight on this thorny issue, I spoke with six Big Data experts at the Strata Data Conference in San Francisco:
- Ben Lorica, Chief Data Scientist, O’Reilly Media
- Nong Li, CTO, Okera
- Jon Bock, VP of Marketing, Streamlio
- Jeff Curie, Principal, The Curie Point
- Jon Rooney, VP of Marketing, Domino Data Lab
- James Kotecki, Director of Marketing, Infinia ML
SEE VIDEO WITH EXPERT ADVICE BELOW
The Difficulty of Mixing Big Data and Human Wisdom
If your business struggles to make the most from Big Data, you’re not alone.
On one hand, businesses hear that Big Data offers decision-making magic. Plug in your metrics and glean insight for huge competitive advantage. Big Data equals big dollars.
Yet even as users lean on Big Data, they know (even if they won’t admit it) that the results are confusing, problematic, even worthless. Sometimes they’re a fast path into deep weeds.
In a 2019 survey from New Vantage, a remarkable 77 percent of respondents admitted that “business adoption of Big Data and AI plans is creating a challenge for their organization.”
Translated: it’s hard to figure this stuff out.
Gartner recently opined that, “Through 2022, only 20% of analytic insights will deliver business outcomes.”
Translated: We constantly stare at the metrics but it’s not making us much money.
Despite the challenges, harnessing Big Data is absolutely essential. The idea that any business can compete without using analytics is woefully outdated. It’s like a ship without a compass. In the infamous Target predictive analytics example, the retailer predicted customer pregnancy even before other family members knew – that’s the shocking competitive power of analytics.
If you’re a manager, you feel the pressure: you know at this very minute your competitors are staring at the numbers. Weighing them, juggling them. Using hyper-specific reporting to guide plans that will grab market share from you. If you’re not doing likewise, you’re falling behind quickly.
Yet as you’ve surely seen, some of the most critical variables can’t be precisely quantified – which in the eyes of your software means they can’t be calculated at all.
Oh sure, sometimes your metrics are trustable. Your analytics app shows sales of 1,200 widgets this quarter, up from last quarter’s 1,000. So your multi-colored dashboard reports that sales are up 20 percent – that’s a rock solid number.
But straight reporting isn’t the magic of Big Data. Truly leveraging that expensive analytics tool means using it to make complex business decisions. Making those insightful decisions that blast you past your competitors.
The problem is this: the more complex the decision, the more likely it includes variables that can’t be quantified. Let’s look at a typical way that data analytics offers no help.
Number Crunching Meets the Real World
In this hypothetical example, let’s say your company has a Western Division and an Eastern Division. It’s time to invest and grow the company: do you invest more in Western or Eastern?
Western has always been the legacy cash cow, a steady revenue machine. But now the bright trend lines in your Big Data app show that Eastern is growing quicker. Drilling down, you see that in Eastern, “revenue per sales rep, per quarter” is up 17.8 percent over Western.
That’s a golden metric. Human wisdom alone would never have discovered that info nugget.
Based on the metrics, the budget spigot opens wide and cash pours into Eastern. We can’t wait for the boosted revenue numbers.
Oh goodness. What the Big Data doesn’t reveal – because it can’t be quantified – is that an HR pro in Eastern, Jessica Roberts, is a genius at recruiting top salespeople. Hence that higher revenue per sales rep metric.
However, Jessica, being talented and ambitious, moves to a competitor. Over the next 24 months, Eastern’s revenue trends drearily downward as Jessica’s hires leave and are replaced by an HR department that’s asleep at the wheel.
The question of where to invest was a core issue for the business – but the Big Data software offered no help.
Then again, you might say: it’s not the software that went wrong, it was the people who looked at the software. Ultimately that’s true. Which brings us to one of the thorniest problems in using Big Data.
Then It Gets Really Tricky
Let’s accept as a given that most businesses are combining Big Data and human instinct when they make decisions. But to what extent – and how – to combine these two inputs is a constant question.
Ironically, the decision of how to weigh the data and gut instinct is itself a human decision. The software can’t tell you how seriously to take its results. So modern business management has become a “meta” process, with decision making at two different levels:
- The pre-decision making process, which involves a human decision about how to weigh the data versus human input.
- The final decision itself.
Bottom line, since all even the best analytics software can do is churn out gorgeous pie charts, the challenge remains. You decide on the balance.
Will you rely largely on the data, with only the merest factor of human wisdom?
This approach was favored by the captain of the Titanic. Chart the most direct approach, raise the speed to the max, arrive as soon as possible. Alas, there was a lurking variable that no thoughtful navigator would have ignored.
Or, will you rely completely on our own hunch, hardly glancing at the data?
This approach can bedevil startup founders, whose businesses are their babies. Sometimes an outside venture capital funder forces them to look at a spreadsheet to keep the business growing.
Hopefully you’ll find a more careful mix then either of these extremes. One thing is sure: your revenue depends on how you balance these factors.
As I spoke with the experts in the video below, it was clear: combining instinct and data is more art than science. At the least, it requires really knowing your business and really understanding the software.
What’s your opinion? Please use the Comments section below to add your experience/insight about combining Big Data and human wisdom. What’s a Big Data best practice?
