Expert Advice: Improving Your Big Data Strategy

hi I'm James McGuire editor of data mation and our topic today is improving your big data strategy to talk about that we've got uh two and possibly three thought Leaders with us is Fern helper research director for advanced analytics at tdwi hello to you Fern hi how you doing today good how are you good good and you know I I I have your book I I've I've actually bought this a while ago probably more than a year ago which you you the big day guide the big data for dummies which you wrote with Judith herwitz Alan Nan and Marca Kaufman hope you're still selling some copies on that yep you know gets outdated so quick I I know it's amazing to put that down a print is almost an act of Courage you know uh also with us is uh James kilis a big data evangelist for big blue for IBM hello to you Jim hi James hi Fern hello everybody out there it's a lovely day that we're expecting a blizzard to start in the next 24 hours uh it's a lovely day before storm because because you're there in the east coast right J where you at Washington DC area so yes big data we're snowed in with big data here too J Jim what what what are like a couple of the projects you're right in the middle of at this point in terms of your big data life sure well I'll tell you what I do for IBM in a nutshell I'm of course a subject matter expert I blog and speak on it constantly everybody can see that I tweet you know but uh fundamentally what I do for IBM is that I'm um I do thought leadership marketing I manage the technical marketing sector of the IBM big data analytics Hub which is our one of our primary content marketing channels um we publish I'm a publisher ENT I'm like an almost like an Editor in Chief essentially you're you're a Content expert in in essence yeah among other things those are some of the hats that I wear for for big blue uh also with us is Stephanie McReynolds uh VP of marketing for silicon valy startup eltion hello to you Stephanie hi James thanks for having me yeah so in in a nutshell what does Eli do I don't think that everyone knows about Elation quite yet not not quite yet we're working on that but maybe not quite yet so Elation is a as a data catalog and um you know I think that we're differentiated from the perspective of we're a data catalog that's really built for collaboration both uh human machine collaboration as well as collaboration between individuals um you might be asking now what is a data catalog well a data catalog I've heard it described as Yelp for Big Data is you know as one of the the tag terms yeah that's a great a great analogy think Yelp for Big Data or even like an Amazon for Big Data you know on Yelp you have descriptions of places and users are able to tag those those places um provide um a rating for places they they visit and share places with other people and if you're a cons consumer and you're trying to figure out how to where to go to dinner you search for a place and find it we're basically doing the same thing for for data so inhalation um individuals can um tag data sets communicate and collaborate around um data sets we also give analysts insight into what's the lineage of that data where did it start from is it actually stored in teradata or Hadoop or vertica or is it in a semantic layer in your your bi tool and help analysts increase their prod it and also trust the data a little bit more because they have all the information great so big data you know it's the interesting thing about the the area of big data is that you know we've all covered it for years and and been involved with it and but I I wonder what the actual adoption rate is and and I i' ask the three of you what you know how the market is really doing I I read this report recently from Howard Dresner advisory it talked about the actual adoption rate among companies and the figure that he came up with was was 177% which says to me that actual involvement in Big Data continues to be closer to its infancy than really really being all the way mainstream I which kind of surprise me fer what's your what's your sense of where the market really is today yeah I I think he's probably in directionally correct I mean we do a lot of surveys obviously at tdwi and we actually also have assessments so one of them was actually a big data maturity model where users and could still do this you can go to our website and click on it maturity models go in answer a bunch of questions and find out how mature you are in terms of big data and so people answered a whole bunch of questions and it looks like in terms of you know volume I'd say and and we have a Hadoop Readiness assessment out there now to see if you're actually ready for Hadoop and in it we actually ask how much data do you collect how much data are you used to managing you know that sort of thing and and less than 10% of organizations are actually of at least the respondents you know to our survey were actually feel like they can manage pedabytes of data you know much higher percent like 30 or 40% are managing or have experienced managing terabytes of data so if you look at you know volume Big Data 10 terabytes you know so there's there's enough organizations out there that can do that in terms of you know the kinds of data that they're managing mostly it's structured data but of course you know we know big data is disparate data so I'd say in surveys that we do about 30% of organizations actually are looking at capturing and analyzing text Data you know and then you look at sort of streaming data right in terms of velocity High Velocity sorts of data and you know that's that's low that's under 20% although there's huge interest in it you know when you look at these charts it's sort of like you know 30% are interested 30% are doing it it and another 40 50% claim that they're going to be doing it in the next couple years so we're definitely on the cusp if these people actually stick to their plans you know it's sometimes people get exuberant when they answer surveys so you know it's I think Howard is probably directionally correct with that ask the same thing but one one caveat to the question I guess if if it is still you know minor like that what why the why the foot drag considering we've been told these many years that big data is this huge competitive Advantage so why why aren't companies just flocking to it you know as quickly as they can Jim what what do what do you think well first of all maybe say that Fern's numbers and twi's numbers are probably pretty much within the ballpark in terms of overall adoption but big data analytics Solutions and we're Diversified provider of the products and services for big data analytics Big Data in all of its solutionized glory is remains one of our core growth areas growth segments of IBM analytics across the board so it's growing for us and now in terms of um your question can you repeat your question again the question is um it seems like big data adoption is slower than you might think based on the many headlines and a lot of coverage that has gotten over the years 177% based on my reading of a Howard jary advisory research report is it is does the market really seen close as close to infancy as that report would suggest not really I mean if you look at the fact that very large databases that was an old term for Big Data been around for quite a long time petascale or I should say terabyte scale uh data warehouses have been um you know in a significant number of I used to be at Forester of course significant number of user companies uh um for for quite for quite a while now the data warehouses of the worlds that AC cross on average cross the terabyte barrier um you know at least five six seven years ago back during my analyst days on average so in a sense you know depends on where you set the threshold of Big Data if it's purely this the amount of story am of data you have in storage for analytical uh applications if you started down at one terabyte for say a data warehouse and so forth you have a a broad adoption for big data now if if you Scope it more narrowly big data to refer to Simply Hado one form um then that's another matter all together but I think very large data sets for analytic applications um within Enterprises have been in adoption for for quite some time um so in a sense you know what do you call Big Data where do you say well I guess I guess what I mean and I'm GNA I'm going to sort of guesstimate in terms of what the research report meant is is not the total amount of Big Data that's being stored you know but it's like what they're actually doing with it are they really effectively mining it for insight are are they constantly sifting through it is there some way to actually get information from all that data well you know really if you take the three VES of Big Data the volume I hit on volume you know if you if you start at terabytes one terabyte has big data you have a large Market or large adoption if you start at one petabyte you have a small Market likewise the velocity if you start if it's if it's it's purely focused on stream Computing environments in adoption if you broaden it to complex event processing that's been around for a long long time so I mean in in particular Niche markets like C of course in the financial Market from long back you could say that's an old established Market that's held its own um or if you're talking about some specific new uh technology that does say high uh low latency data integration and analytics um like spark streaming no it's a fairly immature market so it really depends on which vs you're using like you know variety you have unstructured data stores from long back for ECM to some degree there have been um you know those kinds of platforms used for analytics for quite some time in various markets like in publishing and so forth so once again where do you which V's do you select when you're defining Big Data what thresholds of those V's in terms of magnitude of the amount of data in storage whatever where do you set that bar you can make you can Define it as a large Market or as a small Market if you just find it as one platform like a head dupe or if you just narrow it down even further to say spark on top of a dup right step Stephanie what's your what's your tick are you seeing Big D is closer to its um infancy or is it is it closer to to mainstream well I do I do agree with um with James that it depends how you define big data and how are you classifying this market so you know you look at um big data from a database perspective and Tera dat has had a peted by a club for I think close to 20 years right so if you defined it broadly as folks that are using large volumes of of data or that are approaching Big Data from a structure data perspective um there's lots of adoption out there um I think you know Gartner what's the percentage in your in your what what do you hear oh I it's hard for me to put a percentage on it from a from a vendor perspective a sells only to companies that are doing it big data right right so all your customers are already on board yeah cers right but you're not encountering a lot of people who go well we want to get on board but we're not yet you know call us in 2017 we don't we actually don't talk to a lot of folks who say call us in 2017 so that's you know I know for a fact that we're talking databases there's a large Market out there I think Gartner pegged um back in may they published a report that was a little controversial at the time time uh from a vendor perspective but they pegged um Hado adoption at 26% which is quite a bit larger than than Howard's number and I think if you ask claer they'll give you even a higher number so it's CL claer is very big on Hadoop yeah yeah I mean I think yeah we we had less than it was around a little bit less than 30% adoption of Hadoop if you know with the people that are respondents so that's actually closer to uh I mean if you just look at Hadoop you know because we we see a lot of people at a lot of our events and conferences you know and and they have what you're saying I mean they have terabytes worth of data stuck in you know their databas not stuck in their databases you know but in their warehouses but they're talking about you know now extending the ecosystem to Hadoop and about a little less than 30% of them are you know claim that they actually are doing something with Hadoop it's probably proof of concept it's probably not you know may not who knows you know it's actually becoming more mainstream now a couple years ago people would say what's Hadoop you know and obviously they're all you know jumping on the Hadoop using Hadoop you know for for mainstream in terms of adoption bi you mentioned Howard Dresner he's been called the father of bi bi in his current status as a mark distinct Market it's been around for 25 years or so and I remember when I was at Forest or Boris Eon did some research that found that bi adoption this is like five years ago was no more than like 10% operational bi in terms of actual knowledge workers using operational bi bi by almost anybody's Reckoning is a mainstream technology in the Enterprise world but it doesn't have a huge adoption in terms of not everybody in the business world is using bi every day likewise not everybody's using uh uh big data but there are obviously strong niches in verticals and also within the horizontals within organizations that are using big data constantly Chief marketing officers and their organization use Big Data constantly for social sentiment analysis we certainly do at IBM so it's like so for example the the whole big data Revolution really got going five years ago when the CMOS of the world discovered it they discovered something called Hadoop and map ruce that can help them you know filter the fire hose of Twitter to identify you you know you know sentiment that they can drive into their marketing campaign so in many ways it's already mainstream in the CMO World Big Data it's just not mainstream down at the at you know at the desktop of everybody's desktop Stephanie did you get your did you finish your thought there I want to make sure you finish your thought about the adoption in the market well I was just gonna say that my guess is some of Howard's numbers since you know since since he is involved in the bi Community are more reflective of what does bi on Hadoop today look like and we do know that majority of the Hadoop implementations I think no matter what study you look at majority of the Hadoop implementations are still for transforming data and what looks like a more like an ETL workload and so I think that that's where we have a little bit of a barrier to adoption is how do we make it easier for business people who want to use big data for analytics and want to use Hadoop how do we help them find access the information and actually get some of these self-service bi experiences on top of Hadoop as as infrastructure is is that one of the biggest barriers to adoption I I would think that I me I guess my question would be you know what is holding people back and that obviously is a competitive advantage to using big data and many different levels um you why why Stephanie if you take a step at this what why are people not flocking faster to it given that it's the advantage that it could be yeah I think it's a reflection of just um the challenge of having enough tooling around Hadoop and enough you know right now you kind of have to be an expert to be able to to use had doop right so organizations um I I would say more organizations than maybe Fernwood or are um moving past the PO pocc concept with Hado and they actually have Hado in production but it's for ETL workloads and it still requires if you're going to do analysis someone who is a data scientist or at least likes to to code or do do some coding to access that information and I think we're just on the early edge of um really making it easier for the non technical Specialists to work with uh Hado Hado data and integrate that with the databases they're working with and the bi tools they're already familiar with quick one I I I talked to these two analysts last summer and they're talking about Hadoop and the skill set and one very was very firm and he said to me oh no somebody that's holding back Hadoop is actually the lack of Hadoop specialist and that's really causing a brick wall and then like two days later talking to another analyst either forest or a Gartner I'm not sure which saying oh well people in house could could learn this Fair easily they got some developers in house they could learn this so there really is no skill set uh backlog but Jim go ahead what's what's holding people back made an important Point uh for example if we focus on head dup which is of course in many ways the the the foundation and and fundamental platform everybody focuses on for Big Data it's not the only platform and it's never it was never intended to be for all use cases Hadoop in the last five years became a substantial Market and has migrated in terms of the use cases we at IBM our customers use it for ETL information Refinery for queriable archive for data Lake for information unstructured information governance all these very critical important Niche use cases Hadoop that's hadoop's core function but other big data Technologies Apache spark Has Come Along on gang busters for stream Computing for graph analysis machine learning and then you got of course r as a language you've got lots of other things like misos and everything else in the whole open source analytic ecosystem under Apache and so forth that fulfill various niches within you know in other words everybody's seems a lot of people in this industry seem to think that Hadoop was was somehow faded or destined to become the Uber platform for all use cases and it was never intended to and it's not going to be that and that's not a bad thing it's it's proven fundamental to many of our customers uh analytics and and and uh and application strategies as the core information Refinery nothing else can replace Hadoop for that core multi-structured information Refinery function Fern could would you point to anything in particular a couple factors that you think are are are slowing the process down of adoption or is it just a question of a of a new technology is going to be slow by definition well okay I just want to say one thing first is that please big data does not equal Hadoop you know as points out I mean there's lots of platforms there's lots of aspects you know to it you know it's to me what good is collecting all of the data if you're not going to analyze it you know that's that's my thing about it but in terms of challenges that we see people talking about in terms of analytics and Hadoop and big data and you know any sort of analytics and data management sort of challenges it always falls into the top one is always we don't have the skills you know and we or we don't have the Staffing for it so that's always sort of that one comes out on top you know very often so they're worried about you know rightly or wrongly they're worried about the skills and and whether they can get the staff and then hand inand with that you they're worried about they can't get the budget and they can't get executive support you know those are sort of some of the Big Top the big top three you know the business case isn't strong enough we can't get Executives on board they don't want this you know because there maybe maybe they're not seeing the business case forward and what the actual business need you know is which should be driving this as opposed to build it and they will come you know that there's a need for this so those are the top usually the top three the skills the budget the executive support well I guess if I if I'm a big data uh developer or a big data Big Data vendor and I'm hearing that I'm GNA say to myself oh I've got to go in and give a sales pitch it says my platform is actually super easy to use you're not going to need an advanced skill set to to make this platform happen is is is that not going on Jim is is that don't the salespeople say that when they when they make their pitch well see salespeople in every company try to make the best case for their product or solution you know um essentially um what it comes down to is that you know IBM um stresses to customers that they need the right platforms for the right applications but they and they need the right skill sets within their organizations and there are specific skill set that are critically important for leveraging the value of big data on whatever platform you use data science is critically important data scientists are in fact the app developers of this new generation so what you're talking about here is that who builds the machine learning models that are the fundamental IP inside of cognitive Computing applications for unstructured natural language proess it's the data scientists of the world so what we stress to customers now one of the key things that you need to build build skill sets within your development organization bring more data science skills bring more of the right kinds of tooling like spark with the libraries of algorithms and models and so forth and you need to educate your staff your developers on things like spark um and then send them way to boot camps to learn how to do more Agile development of you know Big Data applications inside of tools like spark um and so a lot of it really focus like we did at data Palo it's an ongoing series of boot camps really three-day community events where developers come together to get uh upgraded in their skills on the new generation of Big Data tools and to learn to build you know in an agile fashion applications rapidly to iterate um so it's really a lot of it is just a lot of our customers are saying yeah we need to just relearn um how to develop applications in the era of cognitive Computing so that's which which which brings a study of you know best practice um Stephanie if you're if you're advising people if someone comes to you and and says well we you know you're in big data what advice do you give us in terms of like improving our strategy or getting getting the most of our deployment I me what do you hear or what what do you tell people instead of you know some best practices for Big Data yeah I I actually I actually tell folks that the the most important part of this equation is is not to think about the technology but really think about how big data is going to change the workstyle of your organization and how we analyze information and one of the things that Hado has done for us is introduce a new type of workflow around schema on read which is more Discovery oriented analytics it's wait give me the term again schema on read SCH schema and read so so that's a technical that's a a technical concept basically we don't have to model the data in advance to be able to apply it to answering a business question okay and what's great about schem on read is it introduces a more exploratory experience with the data where I as a business person I don't have to in advance of trying to apply data to my business problem I don't have to communicate requirements to build a data model in advance and try to guess all the questions I want to ask of the data I can really sit down with Hado and be able to just start to explore data and and um be able to um take kind of a a more hypothesis test ener and I think that's the biggest impact that big data can have on business organizations is giving them that freedom to use data as a as an asset to actually solve problems but to get there you got to change your mindset and um I think a lot of organizations get get stuck and their their big data or their Publications get stuck technology um they're not able to um think about how their analysts might be able to work differently and so one of the things we're trying to do as the vendor is get organizations and analysts to think differently about how they find and access their data and and collaborate around it and that's a pretty exciting place to to be with organizations F what's what's your advice when people say to you you know we have big data but it's it may not be working for us so how can we how can we get more out of this is there is there any like best practices that you throw out to people well there's a lot of best practices I guess the way you said it I mean the first best practice to me would be you know if they're not not getting enough out of it you know maybe they're not thinking about how they need to use it I said this before you know what's what are they trying to get out of it if they're just sort of collecting it and not doing anything with it you know what's the business problem that they're trying to solve with it and then you know when they know the business problem if they claim that they can't get executive support or the budget I mean they have to evangelize it you know it's sometimes it's not if you have an executive on board who's interested that's great but a lot of organizations we talk to you know they actually go door to door to door and and you know they sit down with the executives and you know they explain the value they explain you know what they can be doing and they look for the executive that can help them and then they build it up from there it's sort of like not a silver bullet it's just a lot of hard work a lot of people say you know a lot of it or developers they say you know we become salespeople you know so they're actually trying to sell the concepts so that's what they're doing and you know they get the executives on board they get the budget you know they build their prototypes and proof of Concepts that show value and then you know it sort of takes off it it does take off from there but you know in between that there's lots of sort of best practices about how you evangelize how you talk you know building centers of excellence to you know get skills how you hire people you know how you get the data scientist skills in your organization and that sort of thing so it's it's a long it's a longer list of best practices I guess I'd say what about looking to the Future and and Jim you had mentioned You' got some predictions for 2016 which I'm curious get so I I guess I I would ask where is where is Big Data going in the next say three to five years but also you know when when are the robots finally going to take over do you you know the answer to that J because I I I want to go on record right now is I am I am in favor of robots yeah I at the end of the hour I'll just quickly sum up the fact that um more and more of the actual um distillation of insights from Big Data is getting automated through actually going to interrupt you I think we're having a lot of feedback it might be question of muting your is is it can you mute your speakers as you speak maybe that's it I'm not sure because we're getting a lot of feedback that we weren't getting before I don't know no okay well go ahead tell tell me tell me where we're going three to five years out yeah I know we're at the top and I need to run but I'll let you all right basically um Big Data distillation of the insights from Big Data is getting automated through um Technologies specifically machine learning deep learning and whatnot that's basic in other words it's robotic distillation continuous distillation of insights from massive data sets that are getting ever more you know diverse ever larger and coming in faster so it's you know it's unsupervised learning it's data science it's artificial neural networks it's you know automated recognition of of faces and and voices and sentiment and all that and then that's driving more and more business application a things like you know uh marketing and customer service and and ter you know you know these these chat Bots that you deal with when you go to like Staples or whatever it might be whoever's website that's big data under the covers and more of that is is getting a those kind of engagements with your stakeholders your customers getting automated through the magic of Big Data under the covers machine learning models driving you know automated scripts and S you know so really where big data is going is a lot of it lot of the data science workflow is getting automated to the hillt uh and it has to because companies live and die by these models and these models are can't be you can't have you can't hire millions of data scientists to be building these and tweaking these models night and day a lot of that's that factory pipeline for machine learning has to get automated and is and that's that's where it's going F Fern what do you see looking out three to five years like where is where is Big Data going the future I totally agree with what Jim saying I was going to say operationalized and embedding you know that analytics are going to be operationalized they're going to be comparted processes automated you know in systems um decisions that are you know too complex for humans to do sort of over and over again you know using these techniques so it's going to be operationalized it's going to be embedded you know to bring the decision to the point of the decision maker or automate the decision and I also think you know that's going to also be driven a lot by The Internet of Things you know in terms of data being produced in the use cases sort of out of that we're going to see a lot more of that around preventive maintenance and customer engagement and you know all all sorts of use cases around um internet of things which is also going to feed right into that Automation and embedding and operationalizing so I think that's where a lot of this is going Stephanie your take in the future and and do you have a plan for when the robots take over are you ready for that well we're actually big on robots here at Elian we have we have a machine learning component in our product that we've named Ali the robot which you may not know but robots all right you're ready you're ready then okay we're ready but I I think um in all in all seriousness um I think we have a little bit of a a different take on the role of robots and and machine learning in this B equation and what we're really excited about is the opportunity for um the robots and the humans to uh collaborate and I think that we can actually take down the um the skill set needed to access and interact with with big data um and let a common analyst or a business user do that if we can figure out how to get um machine learning and human interaction to work together um to really um speed up that process of finding the data cleaning it figuring out how to apply it there's a lot of opportunity um for human machine collaboration in optimizing that process and you know there's lots of great computer scientists uh it's not just us there's um you know folks like Joe helstein at Berkeley and Mike Stonebreaker at MIT and many more leaders in the in the space um that I could name who have a similar belief and I I think it's up to uh us as a a vendor Community to figure out what are the right um user interfaces and interactions and applications and models for really making that a reality to help uh help everyone see the the business value from big big data n no I you know human and and machine learning combination seems like it's going to be a big for 2016 uh I I appreciate the three of you expertise fantastic I will I'll send you the link we can all tweet it and promote it and I I thank you very much for sharing your expertise thank you very much Al righty bye bye bye bye bye

This transcript was generated automatically from the video's captions and may contain errors.