dcsimg

10 Top Big Data Trends for 2018

  • 10 Top Big Data Trends for 2018

    10 Top Big Data Trends for 2018
    Organizations are moving their big data to the cloud — slowly — but they still haven't overcome challenges related to siloes, staff skill shortages and data governance.
  • 1. Most enterprises are overconfident in their big data capabilities.

    Overconfidence

    Enterprises think they are better at big data analytics than they really are. Among those surveyed in 2018, 78 percent said that they would rate their organizations' big data maturity as "medium" or "high." But when AtScale independently assessed enterprises, only 12 percent met the criteria for high maturity. While that's an improvement from the 8 percent who were at high maturity in 2016, it indicates that organizations still have a lot of room for improvement.

    Perhaps those overconfident big data professionals should pay closer attention to Gartner's predictions on big data project failure rates. In 2016, the analyst firm predicted that 60 percent of big data projects would fail. But in late 2017 Gartner analyst Nick Heudecker tweeted, "We were too conservative. The failure rate is closer to 85%. And the problem isn't technology."

    Image Source: Pixabay
  • 2. 55% of enterprises still have siloed analytics teams.

    Siloes

    If, as Heudecker claims, the problem with big data projects isn't technology, what is the issue?

    Part of the reason for big data failures may stem from organizational challenges. The AtScale survey found that 55 percent of enterprises still had a siloed, decentralized analytics team. Only 34 percent of companies had followed the best practice of setting up a center of excellence or other centralized data science team.

    Some industries were better in this regard than others. Utilities and online firms were the most likely to have centralized teams, while telecommunication and financial services companies were lagging behind with multiple siloed groups.

    Image Source: Pixabay
  • 3. 77% of enterprises expect to use the cloud for big data in the future.

    Clouds

    The AtScale survey also revealed a clear belief that most big data analytics will one day take place in the cloud. In 2018, 77 percent of those surveyed said that some or all of their big data will eventually be deployed in the cloud, up from 72 percent who said the same thing in 2017.

    Analysts seem to agree that big data is moving to the cloud. In a blog post, Brian Hopkins, Forrester vice president and principal analyst, wrote, "In five years you’ll be using Insight PaaS for big data in the public cloud." He added that cloud big data spending is growing nearly 7.5 times faster than on-premise big data software spending. The reason? Cloud big data costs are going down exponentially while AI technology is making these services better all the time.

    Image Source: Pixabay
  • 4. 41% of enterprises aren't using the cloud for big data today.

    Cloud

    Cloud may be the future, but it isn't the reality today. In the AtScale survey, 41 percent of respondents said their organization had no big data in the cloud today. In addition, 28 percent had less than half of their big data in the cloud, while 15 percent had more than half in the cloud and 16 percent were 100 percent in the cloud.

    In fact, many big data professionals haven't yet been convinced that the cloud is the right deployment approach for big data. When asked which deployment type they would choose for their big data projects, nearly half (47 percent) chose non-cloud, compared to 40 percent who chose cloud deployment and 13 percent who were undecided.

    Image Source: Pixabay
  • 5. PowerBI is becoming more popular.

    PowerBI logo

    The AtScale survey also uncovered a shift in the popularity of various business intelligence (BI) solutions. The top two tools — Tableau and Microsoft Excel, in that order — have remained the same since 2016. Among 2018 respondents, 63 percent use Tableau, and 50 percent use Excel.

    However, the number three spot on the list saw a dramatic shift. In 2016 and 2017, Business Objects was third on the list. But in 2018, Business Objects dropped down to seventh place. Microsoft PowerBI, which was seventh in 2017 and not on the list in 2016, came in third for the most recent survey. Among respondents, 25 percent said they use PowerBI, putting it ahead of solutions like Qlik (23 percent), IBM Cognos (19 percent), Microstrategy (19 percent) and SAP/Business Objects (18 percent). That led the report authors to conclude, "Microsoft Business Intelligence (PowerBI and Excel) are having the most impact in this market."

    Image Source: Microsoft
  • 6. 95% of enterprises will do as much or more with big data in the future.

    Graphs

    It's hardly surprising, but still noteworthy, that 95 percent of those surveyed said their firms planned to do as much or more with big data over the next three months, while 5 percent said their companies would be scaling back their big data use.

    That's in keeping with advice from Gartner, which last year advised, "As data and analytics become more widely adopted than ever before, the potential for business growth is truly exponential rather than just cumulative. Those who fail to act today will suffer not just in 2017, but also hugely limit their potential for growth in 2018 and beyond, as the returns from increased insight, responsiveness and efficiency snowball."

    Image Source: Pixabay
  • 7. Organizations still can't find professionals with big data skills.

    Big data skills

    For three years in a row, finding employees with the right big data skill set has been the number one challenge cited by respondents to the AtScale survey. Many different organizations now offer courses in data science and big data, but they haven't been able to train new workers fast enough to keep up with demand. As a result, salaries for big data jobs are sky-high.

    For a while, it seemed as though this lack of qualified professionals was driving enterprises to choose big data solutions with self-service capabilities that allow other businesspeople to use analytics tools. However, although self-service big data climbed in to 47 percent in 2017, in 2018 it actually dropped down to 41 percent.

    Image Source: Pixabay
  • 8. Data governance is becoming a big issue.

    Data governance

    The other big data issue that is now on everyone's radar is governance, which was the second most common concern for 2018. This represents a major shift because back in 2016, governance was fifth on the list, and even in 2017 it was third. Management and security, which were the number two and number three concerns, respectively, in 2016, fell to fifth and fourth for 2018.

    Part of the reason for growing concern about governance may be the scandals at Facebook and elsewhere related to user privacy, as well as increasing governmental regulation. The EU's General Data Protection Regulation (GDPR) takes effect next month and has many organizations scrambling to ensure that they are in compliance.

    Image Source: Pixabay
  • 9. Only 19% of enterprises are using big data systems to replace data warehouses.

    Warehouse

    Even though enterprises are embracing big data analytics, they aren't necessarily getting rid of their legacy data warehouses and other technologies. In fact, many are developing hybrid architecture that includes on-premises data warehouses, data lakes built on Hadoop and related technologies, and cloud-based big data services. In the AtScale survey, only 19 percent of respondents said they planned to completely replace their older data solutions with new big data tools. Forty-two percent said new big data tools were augmenting their older investments, and 39 percent said they were replacing some tools but augmenting others.

    Image Source: Pixabay
  • 10. 66% of enterprises believe big data is strategic or game-changing.

    Big data analytics

    When asked about the role that big data plays in their organization, about two-thirds (66 percent) of respondents said they considered it "strategic" or "game-changing." Only 17 percent said big data was still "experimental" at their companies, and another 17 percent described their efforts as "tactical."

    That perspective on big data makes it clear that this is one trend that won't be going away anytime soon. Gartner's Ted Friedman, vice president and distinguished analyst at Gartner, offers these three predictions about what will happen as more enterprises begin to view big data as strategic:

    1. "Data and analytics will drive modern business operations, and not simply reflect their performance."

    2. "Organizations will take a holistic approach to data and analytics. Businesses will create end-to-end architectures that will allow for data management and analytics from the core to the edge of the organization."

    3. "Executives will make data and analytics part of the business strategy, which will allow data and analytics professionals to assume new roles and create business growth."

    Image Source: Pixabay
  • 1 of

10 Top Big Data Trends for 2018

  • 1 of
  • 10 Top Big Data Trends for 2018

    10 Top Big Data Trends for 2018

    Organizations are moving their big data to the cloud — slowly — but they still haven't overcome challenges related to siloes, staff skill shortages and data governance.
  • Overconfidence

    1. Most enterprises are overconfident in their big data capabilities.

    Enterprises think they are better at big data analytics than they really are. Among those surveyed in 2018, 78 percent said that they would rate their organizations' big data maturity as "medium" or "high." But when AtScale independently assessed enterprises, only 12 percent met the criteria for high maturity. While that's an improvement from the 8 percent who were at high maturity in 2016, it indicates that organizations still have a lot of room for improvement.

    Perhaps those overconfident big data professionals should pay closer attention to Gartner's predictions on big data project failure rates. In 2016, the analyst firm predicted that 60 percent of big data projects would fail. But in late 2017 Gartner analyst Nick Heudecker tweeted, "We were too conservative. The failure rate is closer to 85%. And the problem isn't technology."

    Image Source: Pixabay
  • Siloes

    2. 55% of enterprises still have siloed analytics teams.

    If, as Heudecker claims, the problem with big data projects isn't technology, what is the issue?

    Part of the reason for big data failures may stem from organizational challenges. The AtScale survey found that 55 percent of enterprises still had a siloed, decentralized analytics team. Only 34 percent of companies had followed the best practice of setting up a center of excellence or other centralized data science team.

    Some industries were better in this regard than others. Utilities and online firms were the most likely to have centralized teams, while telecommunication and financial services companies were lagging behind with multiple siloed groups.

    Image Source: Pixabay
  • Clouds

    3. 77% of enterprises expect to use the cloud for big data in the future.

    The AtScale survey also revealed a clear belief that most big data analytics will one day take place in the cloud. In 2018, 77 percent of those surveyed said that some or all of their big data will eventually be deployed in the cloud, up from 72 percent who said the same thing in 2017.

    Analysts seem to agree that big data is moving to the cloud. In a blog post, Brian Hopkins, Forrester vice president and principal analyst, wrote, "In five years you’ll be using Insight PaaS for big data in the public cloud." He added that cloud big data spending is growing nearly 7.5 times faster than on-premise big data software spending. The reason? Cloud big data costs are going down exponentially while AI technology is making these services better all the time.

    Image Source: Pixabay
  • Cloud

    4. 41% of enterprises aren't using the cloud for big data today.

    Cloud may be the future, but it isn't the reality today. In the AtScale survey, 41 percent of respondents said their organization had no big data in the cloud today. In addition, 28 percent had less than half of their big data in the cloud, while 15 percent had more than half in the cloud and 16 percent were 100 percent in the cloud.

    In fact, many big data professionals haven't yet been convinced that the cloud is the right deployment approach for big data. When asked which deployment type they would choose for their big data projects, nearly half (47 percent) chose non-cloud, compared to 40 percent who chose cloud deployment and 13 percent who were undecided.

    Image Source: Pixabay
  • PowerBI logo

    5. PowerBI is becoming more popular.

    The AtScale survey also uncovered a shift in the popularity of various business intelligence (BI) solutions. The top two tools — Tableau and Microsoft Excel, in that order — have remained the same since 2016. Among 2018 respondents, 63 percent use Tableau, and 50 percent use Excel.

    However, the number three spot on the list saw a dramatic shift. In 2016 and 2017, Business Objects was third on the list. But in 2018, Business Objects dropped down to seventh place. Microsoft PowerBI, which was seventh in 2017 and not on the list in 2016, came in third for the most recent survey. Among respondents, 25 percent said they use PowerBI, putting it ahead of solutions like Qlik (23 percent), IBM Cognos (19 percent), Microstrategy (19 percent) and SAP/Business Objects (18 percent). That led the report authors to conclude, "Microsoft Business Intelligence (PowerBI and Excel) are having the most impact in this market."

    Image Source: Microsoft
  • Graphs

    6. 95% of enterprises will do as much or more with big data in the future.

    It's hardly surprising, but still noteworthy, that 95 percent of those surveyed said their firms planned to do as much or more with big data over the next three months, while 5 percent said their companies would be scaling back their big data use.

    That's in keeping with advice from Gartner, which last year advised, "As data and analytics become more widely adopted than ever before, the potential for business growth is truly exponential rather than just cumulative. Those who fail to act today will suffer not just in 2017, but also hugely limit their potential for growth in 2018 and beyond, as the returns from increased insight, responsiveness and efficiency snowball."

    Image Source: Pixabay
  • Big data skills

    7. Organizations still can't find professionals with big data skills.

    For three years in a row, finding employees with the right big data skill set has been the number one challenge cited by respondents to the AtScale survey. Many different organizations now offer courses in data science and big data, but they haven't been able to train new workers fast enough to keep up with demand. As a result, salaries for big data jobs are sky-high.

    For a while, it seemed as though this lack of qualified professionals was driving enterprises to choose big data solutions with self-service capabilities that allow other businesspeople to use analytics tools. However, although self-service big data climbed in to 47 percent in 2017, in 2018 it actually dropped down to 41 percent.

    Image Source: Pixabay
  • Data governance

    8. Data governance is becoming a big issue.

    The other big data issue that is now on everyone's radar is governance, which was the second most common concern for 2018. This represents a major shift because back in 2016, governance was fifth on the list, and even in 2017 it was third. Management and security, which were the number two and number three concerns, respectively, in 2016, fell to fifth and fourth for 2018.

    Part of the reason for growing concern about governance may be the scandals at Facebook and elsewhere related to user privacy, as well as increasing governmental regulation. The EU's General Data Protection Regulation (GDPR) takes effect next month and has many organizations scrambling to ensure that they are in compliance.

    Image Source: Pixabay
  • Warehouse

    9. Only 19% of enterprises are using big data systems to replace data warehouses.

    Even though enterprises are embracing big data analytics, they aren't necessarily getting rid of their legacy data warehouses and other technologies. In fact, many are developing hybrid architecture that includes on-premises data warehouses, data lakes built on Hadoop and related technologies, and cloud-based big data services. In the AtScale survey, only 19 percent of respondents said they planned to completely replace their older data solutions with new big data tools. Forty-two percent said new big data tools were augmenting their older investments, and 39 percent said they were replacing some tools but augmenting others.

    Image Source: Pixabay
  • Big data analytics

    10. 66% of enterprises believe big data is strategic or game-changing.

    When asked about the role that big data plays in their organization, about two-thirds (66 percent) of respondents said they considered it "strategic" or "game-changing." Only 17 percent said big data was still "experimental" at their companies, and another 17 percent described their efforts as "tactical."

    That perspective on big data makes it clear that this is one trend that won't be going away anytime soon. Gartner's Ted Friedman, vice president and distinguished analyst at Gartner, offers these three predictions about what will happen as more enterprises begin to view big data as strategic:

    1. "Data and analytics will drive modern business operations, and not simply reflect their performance."

    2. "Organizations will take a holistic approach to data and analytics. Businesses will create end-to-end architectures that will allow for data management and analytics from the core to the edge of the organization."

    3. "Executives will make data and analytics part of the business strategy, which will allow data and analytics professionals to assume new roles and create business growth."

    Image Source: Pixabay

Big data analytics (BDA) is no longer the latest hot IT trend. Instead, today, it's an established part of doing business for most companies.

In the words of Dan Vesset, group vice president of analytics and information management at IDC, "After years of traversing the adoption S-curve, big data and business analytics solutions have finally hit mainstream. BDA as an enabler of decision support and decision automation is now firmly on the radar of top executives. This category of solutions is also one of the key pillars of enabling digital transformation efforts across industries and business processes globally."

His firm predicts that enterprise BDA spending will grow at a compound annual growth rate (CAGR) of 11.9 percent through 2020, when revenues will exceed $210 billion.

However, as the field of big data analytics matures, enterprise experiences with the technology are evolving. They are looking to new tools and deployment models, and while they have overcome some of the challenges involved in working with big data, others continue to plague analytics projects.

To gauge exactly how big data practices are changing —and how they are not — business intelligence vendor AtScale recently conducted a worldwide survey in cooperation with Cloudera, Hortonworks, MapR, Tableau, Apache Software Foundation and ODPi (a Linux Foundation Project). Over a three-year period, they collected responses from 5,593 enterprise architects, big data leaders and other IT professionals from all industries. The resulting 2018 Big Data Maturity Survey Report included 10 key takeaways that highlight key big data trends.

Image Source: Pixabay

Submit a Comment

Loading Comments...