This problem simply highlights one of the most pressing obstacles facing predictive analytics: data lock. Its all well and good to say youll leverage social media, but when it comes time to pull out the data and actually do something with it, todays BI suites arent quite up to the task.
Today, the big chore is getting data out of data warehouses and BI suites and into forecasting software. The big BI vendors are all working on integrating predictive features into their core BI suites, but itll take longer before they can pull in information from wherever an organization deems valuable.
The wisdom-of-crowds insights gleaned from the real-time web will have to wait.
Another serious obstacle is making sense of the data itself. Most knowledge workers arent statisticians. The trick here is to represent data as something other than raw numbers and formulas.
Visualizing complex data is critical if predictive analytics is to ever catch on. BI vendors all claim to be working on better representations, but the jury is still out on what will effectively engage average knowledge workers.
One of the dangers of any sort of forecasting is getting too enamored with models that truncate reality, limit inputs and outputs and then claim to accurately predict the future.
Wall Street quants, those doing predictive analysis for financial firms, were blindsided by the economic collapse. Few saw it coming, and few of their models showed even hints of problems to come.
Financial models underestimated risk, overestimated growth, and greatly overestimated the rational behavior of investors. Will organizations using predictive analytics learn from epic failures like this one? It remains to be seen.
What Ill predict is that as predictive analytics catches on, smart organizations will work hard to verify the inputs to their formulas, constantly questioning basic assumptions. These organizations may seem cautious and conservative, and they may catch some flak from Wall Street. But in the long run, theyll be a much safer bet than those predicting nothing but smooth sailing ahead.