Big Data Pros and Cons

NewVantage Partners’ Big Data Executive Survey 2018 found that 97.2 percent of the firms surveyed were investing in big data and AI initiatives.

All that investment is leading to a booming market. According to IDC, worldwide spending on big data and analytics is growing at a compound annual growth rate (GAGR) of 11.9 percent, and revenues will likely total more than $210 billion by 2020.

If you’re in the market for enterprise big data tools, see our list of top big data companies

But spending a lot of money on big data analytics doesn’t guarantee that organizations are getting the results that they want — or even that they know what they are doing. In its 2018 Big Data Maturity Survey, vendor AtScale noted that while 78 percent of companies believe they are at a “medium” or “high” level of big data maturity, in reality, only 12 percent meet the criteria a high level of maturity. And organizations at lower levels of maturity continue to struggle with multiple challenges in regards to big data analytics.

Before embarking on any new analytics project, experts recommend that enterprises carefully weigh the pros and cons of big data to see whether the initiative will be worth the risk and cost.

Learn more about big data with our courses on TechRepublic Academy!

Advantages of Big Data

The vast majority of companies say that the benefits of big data are substantial. In the NewVantage Partners survey, 73.2 percent of executives said that they had seen measurable business results from their efforts. In addition, “Those executives who responded ‘no’ believe that it is too early to tell what impact these investments will have on their firm.”

So what are those benefits?

Enterprises report multiple advantages of big data, including the following:

• Better decision-making: In the NewVantage Partners survey, 36.2 percent of respondents said that better decision-making was the number one goal of their big data analytics efforts. In addition, 84.1 percent had started working toward that goal, and 59.0 percent had experienced some measurable success, for an overall success rate of 69.0 percent. Analytics can give business decision-makers the data-driven insights they need to help their companies compete and grow.

• Increased productivity: A separate survey from vendor Syncsort found that 59.9 percent of respondents were using big data tools like Hadoop and Spark to increase business user productivity. Modern big data tools are allowing analysts to analyze more data, more quickly, which increases their personal productivity. In addition, the insights gained from those analytics often allow organizations to increase productivity more broadly throughout the company.

• Reduce costs: Both the Syncsort and the NewVantage surveys found that big data analytics were helping companies decrease their expenses. Nearly six out of ten (59.4 percent) respondents told Syncsort big data tools had helped them increase operational efficiency and reduce costs, and about two thirds (66.7 percent) of respondents to the NewVantage survey said they had started using big data to decrease expenses. Interestingly, however, only 13.0 percent of respondents selected cost reduction as their primary goal for big data analytics, suggesting that for many this is merely a very welcome side benefit.

• Improved customer service: Among respondents to the NewVantage survey, improving customer service was the second most common primary goal for big data analytics projects, and 53.4 percent of companies had experienced some success in this regard. Social media, customer relationship management (CRM) systems and other points of customer contact give today’s enterprises a wealth of information about their customers, and it is only natural that they would use this data to better serve those customers.

• Fraud detection: Another common use for big data analytics — particularly in the financial services industry — is fraud detection. One of the big advantages of big data analytics systems that rely on machine learning is that they are excellent at detecting patterns and anomalies. These abilities can give banks and credit card companies the ability to spot stolen credit cards or fraudulent purchases, often before the cardholder even knows that something is wrong.

• Increased revenue: When organizations use big data to improve their decision-making and improve their customer service, increased revenue is often the natural result. In the Syncsort survey, more than half of respondents (54.7 percent) said they were using big data tools to increase revenue and accelerate growth based on better insights.

• Increased agility: Again, from the Syncsort report, 41.7 percent of respondents said that one of the benefits of big data was the ability to increase business/IT agility. Many organizations are using their big data to better align their IT and business efforts, and they are using their analytics to support faster and more frequent changes to their business strategies and tactics.

• Greater innovation: Innovation is another common benefit of big data, and the NewVantage survey found that 11.6 percent of executives are investing in analytics primarily as a means to innovate and disrupt their markets. They reason that if they can glean insights that their competitors don’t have, they may be able to get out ahead of the rest of the market with new products and services.

• Faster speed to market: Along those same lines, executives also told NewVantage that they were using big data to achieve faster time-to-market. Only 8.8 percent said that this was their number one goal for big data, but 53.6 percent have started working toward that goal, and of those, 54.1 percent had achieved some success. This advantage of big data is also likely to result in additional benefits, such as faster growth and higher revenue.

Disadvantages of Big Data

On the other side of the equation, many companies have also reported significant challenges when implementing big data analytics initiatives. Reported disadvantages of big data include the following:

• Need for talent: Data scientists and big data experts are among the most highly coveted —and highly paid — workers in the IT field. The AtScale survey found that the lack of a big data skill set has been the number one big data challenge for the past three years. And in the Syncsort survey, respondents ranked skills and staff as the second biggest challenge when creating a data lake. Hiring or training staff can increase costs considerably, and the process of acquiring big data skills can take considerable time.

• Data quality:In the Syncsort survey, the number one disadvantage to working with big data was the need to address data quality issues. Before they can use big data for analytics efforts, data scientists and analysts need to ensure that the information they are using is accurate, relevant and in the proper format for analysis. That slows the reporting process considerably, but if enterprises don’t address data quality issues, they may find that the insights generated by their analytics are worthless — or even harmful if acted upon.

• Need for cultural change: Many of the organizations that are utilizing big data analytics don’t just want to get a little bit better at reporting, they want to use analytics to create a data-driven culture throughout the company. In fact, in the NewVantage survey, a full 98.6 percent of executives said that their firms were in the process of creating this new type of corporate culture. However, changing culture is a tall order. So far, only 32.4 percent were reporting success on this front.

• Compliance: Another thorny issue for big analytics efforts is complying with government regulations. Much of the information included in companies’ big data stores is sensitive or personal, and that means the firm may need to ensure that they are meeting industry standards or government requirements when handling and storing the data. In the Syncsort survey, data governance, including compliance, was the third most significant barrier to working with big data. In fact, when respondents were asked to rank big data challenges on a scale from 1 (most significant) to 5 (least significant), this disadvantage of big data got more 1s than another other challenge.

• Cybersecurity risks: Storing big data, particularly sensitive data, can make companies a more attractive target for cyberattackers. In the AtScale survey, respondents have consistently listed security as one of the top challenges of big data, and in the NewVantage report, executives ranked cybersecurity breaches as the single greatest data threat their companies face.

• Rapid change: Another potential drawback to big data analytics is that the technology is changing rapidly. Organizations face the very real possibility that they will invest in a particular technology only to have something much better come along a few months later. Syncsort respondents ranked this disadvantage of big data fourth among all the potential challenges they faced.

• Hardware needs: Another significant issue for organizations is the IT infrastructure necessary to support big data analytics initiatives. Storage space to house the data, networking bandwidth to transfer it to and from analytics systems, and compute resources to perform those analytics are all expensive to purchase and maintain. Some organizations can offset this problem by using cloud-based analytics, but that usually doesn’t eliminate the infrastructure problems entirely.

• Costs: Many of today’s big data tools rely on open source technology, which dramatically reduces software costs, but enterprises still face significant expenses related to staffing, hardware, maintenance and related services. It’s not uncommon for big data analytics initiatives to run significantly over budget and to take more time to deploy than IT managers had originally anticipated.

• Difficulty integrating legacy systems: Most enterprises that have been around for very many years have siloed data in a variety of different applications and systems throughout their environments. Integrating all those disparate data sources and moving data where it needs to be also adds to the time and expense of working with big data.

Perceived Challenges of Big Data

Source: AtScale 2018 Big Data Maturity Survey

Pros and Cons of Big Data

In the end, when weighing big data pros and cons, most organizations decide that the advantages outweigh the disadvantages. However, the relative drawbacks and benefits of big data are always worth careful consideration before launching a new big data project.

More Information about Big Data

• Big data trends
• Big data best practices
• Big data use cases