Big data is no longer just a buzzword. Researchers at Forrester have "found that, in 2016, almost 40 percent of firms are implementing and expanding big data technology adoption. Another 30 percent are planning to adopt big data in the next 12 months." Similarly, the Big Data Executive Survey 2016 from NewVantage Partners found that 62.5 percent of firms now have at least one big data project in production, and only 5.4 percent of organizations have no big data initiatives planned or underway.
Researchers say the adoption of big data technologies is unlikely to slow anytime soon. IDC predicts that the big data and business analytics market will increase from $130.1 billion this year to more than $203 billion in 2020. "The availability of data, a new generation of technology, and a cultural shift toward data-driven decision making continue to drive demand for big data and analytics technology and services," said Dan Vesset, group vice president, analytics and information management. "This market is forecast to grow 11.3 percent in 2016 after revenues reached $122 billion worldwide in 2015 and is expected to continue at a compound annual growth rate (CAGR) of 11.7 percent through 2020."
While it's clear that the big data market will grow, how organizations will be using their big data is a little less clear. New big data technologies are entering the market, while use of some older technologies continues to grow.
To evaluate big data solutions for your enterprise, see our list of top big data companies
Big Data Trends
Truly keeping track of Big Data trends is like trying to monitor the daily shifts in the wind – the minute you sense a direction, it changes. Yet the following trends are clearly shaping Big Data going forward.
1. Big Data and Open Source
Open source applications like Apache Hadoop, Spark and others have come to dominate the big data space, and that trend looks likely to continue. One survey found that nearly 60 percent of enterprises expect to have Hadoop clusters running in production by the end of this year. And according to Forrester, Hadoop usage is increasing 32.9 percent per year.
Experts say that in 2017, many enterprises will expand their use of Hadoop and NoSQL technologies, as well as looking for ways to speed up their big data processing. Many will be seeking technologies that allow them to access and respond to data in real time.
Hadoop is a high profile example of an open source Big Data project.
2. In-Memory Technology
One of the technologies that companies are investigating in an attempt to speed their big data processing is in-memory technology. In a traditional database, the data is stored in storage systems equipped with hard drives or solid state drives (SSDs). In-memory technology stores the data in RAM instead, which is many, many times faster. A report from Forrester Research forecasts that in-memory data fabric will grow 29.2 percent per year.
3. Machine Learning
As big data analytics capabilities have progressed, some enterprises have begun investing in machine learning (ML). Machine learning is a branch of artificial intelligence that focuses on allowing computers to learn new things without being explicitly programmed. In other words, it analyzes existing big data stores to come to conclusions which change how the application behaves.
According to Gartner machine learning is one of the top 10 strategic technology trends for 2017. It noted that today's most advanced machine learning and artificial intelligence systems are moving "beyond traditional rule-based algorithms to create systems that understand, learn, predict, adapt and potentially operate autonomously."
Machine learning plays a profound role in current Big Data technology. (Image source: MapR.)
4. Predictive Analytics
Predictive analytics is closely related to machine learning; in fact, ML systems often provide the engines for predictive analytics software. In the early days of big data analytics, organizations were looking back at their data to see what happened and then later they started using their analytics tools to investigate why those things happened. Predictive analytics goes one step further, using the big data analysis to predict what will happen in the future.
The number of organizations using predictive analytics today is surprisingly low—only 29 percent according to a 2016 survey from PwC. However, numerous vendors have recently come out with predictive analytics tools, so that number could skyrocket in the coming years as businesses become more aware of this powerful tool.
5. Big Data Intelligent Apps
Another way that enterprises are using machine learning and AI technologies is to create intelligent apps. These applications often incorporate big data analytics, analyzing users' previous behaviors in order to provide personalization and better service. One example that has become very familiar is the recommendation engines that now power many ecommerce and entertainment apps.
In its list of Top 10 Strategic Technology Trends for 2017, Gartner listed intelligent apps second. "Over the next 10 years, virtually every app, application and service will incorporate some level of AI," said David Cearley, vice president and Gartner Fellow. "This will form a long-term trend that will continually evolve and expand the application of AI and machine learning for apps and services."
6. Intelligent Security
Many enterprises are also incorporating big data analytics into their security strategy. Organizations' security log data provides a treasure trove of information about past cyberattack attempts that organizations can use to predict, prevent and mitigate future attempts. As a result, some organizations are integrating their security information and event management (SIEM) software with big data platforms like Hadoop. Others are turning to security vendors whose products incorporate big data analytics capabilities.
The Internet of Things is also likely to have a sizable impact on big data. According to a September 2016 report from IDC, "31.4 percent of organizations surveyed have launched IoT solutions, with an additional 43 percent looking to deploy in the next 12 months."
With all those new devices and applications coming online, organizations are going to experience even faster data growth than they have experienced in the past. Many will need new technologies and systems in order to be able to handle and make sense of the flood of big data coming from their IoT deployments.
The torrent of data harvested by IoT will certainly increase the demand for Big Data applications. (Image Source: Verizon State of the Market: Internet of Things 2016)
8. Edge Computing
One new technology that could help companies deal with their IoT big data is edge computing. In edge computing, the big data analysis happens very close to the IoT devices and sensors instead of in a data center or the cloud. For enterprises, this offers some significant benefits. They have less data flowing over their networks, which can improve performance and save on cloud computing costs. It allows organizations to delete IoT data that is only valuable for a limited amount of time, reducing storage and infrastructure costs. Edge computing can also speed up the analysis process, allowing decision makers to take action on insights faster than before.
9. High Salaries
For IT workers, the increase in big data analytics will likely mean high demand and high salaries for those with big data skills. According to IDC, "In the U.S. alone there will be 181,000 deep analytics roles in 2018 and five times that many positions requiring related skills in data management and interpretation."
As a result of that scarcity, Robert Half Technology predicts that average compensation for data scientists will increase 6.5 percent in 2017 and range from $116,000 to $163,500. Similarly, big data engineers should see pay increases of 5.8 percent with salaries ranging from $135,000 to $196,000 for next year.
As the cost of hiring big experts rises, many organizations are likely to be looking for tools that allow regular business professionals to meet their own big data analytics needs. IDC has previously predicted "Visual data discovery tools will be growing 2.5 times faster than rest of the business intelligence (BI) market. By 2018, investing in this enabler of end-user self service will become a requirement for all enterprises."
Several vendors have already launched big data analytics tools with "self-service" capabilities, and experts expect that trend to continue into 2017 and beyond. IT is likely to become less involved in the process as big data analytics becomes more integrated into the ways that people in all parts of the business do their jobs.