“Hadoop is taking on some very strong footholds in the enterprise,” he said. “Lots of people test out free stuff. Where Hadoop is right now is moving away from experimenters to guys saying ‘we’re going to use this, we’re going to use it in an effective manner and it will deliver value to the business.’”
He compares Hadoop with RDBMS, which came out in the 1970s but didn’t really take off until the 1980s when Oracle made a massive push to support developers. “When RDBMSes came out, there was a huge push to get people to become DBAs. They learned SQL. Well, we’re in the early days of Hadoop. Right now it’s high pressure but in a few years, the ecosystem has to train people. The majority of efforts from Cloudera and Hortonworks are spending a lot of time on education. Will it be overnight? No.”
And he agrees with Adrian’s contention that for Hadoop to grow in use it has to be accessible to more than just Java programmers, since Hadoop is written in Java. “You need the skillsets, the tooling, the value-added apps. The only way you can make this accessible to the mere mortals is to hide the complexity,” said Nanduri. “If they are well managed we will see continued adoption.”
Finding the Hadoop and Big Data Talent
The biggest challenge, though, is the oft-documented problem of finding the talent to get the job done. There is a huge shortage of people with Big Data skills, and they can command top dollar. “The top challenges we hear from organizations using Hadoop today, number one is obtaining skills and capabilities, number two is figuring out how to get value out of it,” said Adrian.
In fact, he argues that the technology is way ahead of the skills. “The truth is today, the Hadoop stack is very competent and capable of doing things people want to do with it. The tech is ready for the kinds of things most people are thinking of doing. Some researchers say skills are not a problem. They are dead wrong. Skills are definitely a problem,” said Adrian.
Enterprise companies with large services divisions like IBM and HP and pure services firms like CSC and InfoSys are training people as fast as they can, but you don’t create a data scientist overnight. “People can’t write the algorithms, never mind implement them. Some large systems integrators are turning down jobs because they can't provide the resources,” said Adrian.
That’s been 2nd Watch’s experience, which has had to turn down some projects, according to Barnes, due to a shortage of talent. For most of the Hadoop or Big Data-type projects, 2nd Watch has a longer lead time because those few individuals with the appropriate skillsets are booked longer than a typical project. So the company is hiring bright talent and training them.
“[Training them is] not easy at all. We’re talking to bright people and training them but finding people with those skillsets [already] is almost impossible. I speak with architects and engineers all the time, there’s a great deal of interest in training candidates but I’m not seeing that translate into a net surplus of candidate available. So it could be demand is absorbing them faster than folks are being trained.”
Hadoop Going Forward
Adrian believes Hadoop will eventually live up to its hype. “It's a more cost-effective approach hitherto unexploited information than is available. It represents an opportunity we haven't exploited. As it becomes more available and more manageable for the enterprise class it will be adopted,” he said.
But he added Hadoop will change, too. “Two to three years from now, people could be talking about their Spark stack, that's entirely possible. These things continue to change. The answer to what is Hadoop is different today than it was two years ago and will be different two years from now. I think it has legs and will be here for quite some time,” said Adrian.
Barnes said interest is growing, slow but steady. “It has got the mindshare. A lot of the conversations we have are with key decision makers all the way up to executive level are very aware. They’re familiar with it and the promise. We’re having interesting conversations for the possibilities of it. It’s not just mining usage logs or parsing for fraud detection anymore,” he said.
Photo courtesy of Shutterstock.