Cloud and Big Data: Gaining a Competitive Edge

Cloud computing and Big Data are, in 2018, essentially the perfect marriage.

So it’s no surprise that analysts and enterprise IT decision makers agree: 2018 will see a flood of big data projects moving into the public cloud.

In a report titled, Move Your Big Data Into the Public Cloud: You Won’t be Able to Keep Up With Customers If You Don’t, Forrester analyst Brian Hopkins wrote, “Public cloud adoption is the No. 1 priority for technology decision makers investing in big data.”

A separate Forrester survey, commissioned by Oracle and Intel, found that 80 percent of the 431 organizations surveyed want to move their big data and analytics initiatives to the cloud.

And IDC has predicted, “By 2018, new cloud pricing models will emerge for specific analytics workloads, contributing to five times higher growth in spending on cloud versus on-premises analytics solutions.”

Why the sudden rush to the cloud?

First, most enterprises have experienced success with initial on-premises big data projects. 2017 data from NewVantage Partners found that 95 percent of the executives surveyed had undertaken a big data project, and 80 percent described those initiatives as successful. As a result of that early success, many want to expand their use of big data tools. And that means more big data software spending. IDC has forecasted that the big data analytics software market will achieve a 10.6 percent compound annual growth rate over the next five years as companies grow their investments.

However, enterprises haven’t been completely happy with their on-premises big data options. Most big data projects involve Hadoop, which is notoriously tricky to use. According to Gartner, “The complexity and questionable usefulness of the entire Hadoop stack is causing many organizations to reconsider its role in their information infrastructure. Instead, organizations are looking at increasingly competitive and convenient cloud-based options with on-demand pricing and fit-for-purpose data processing options.”

Analysts say those companies that have been among the first to migrate big data to the public cloud have been experiencing advantages. Hopkins warned, “While most firms think they have time to migrate, disruptors steal customers by leveraging big data cloud innovation such as serverless analytics and artificial intelligence (AI).” He advised enterprises, “You must immediately shift your big data investment from on-premises or hybrid toward public cloud.”

Cloud and Big Data: Benefits of Migrating

Cloud-based big data services offer all the same benefits associated with other public cloud services. They promise greater agility and flexibility, the ability to access data from anywhere, easy provisioning and management, and cost benefits.

In addition, public clouds are particularly well suited for big data and analytics workloads because of their scalability. According to cloud-based big data vendor Qubole, “The cloud and big data are really a perfect match as the dynamic scalability and affordability of the cloud match well with the massive scale and ‘bursty’ nature of resources that are required for big data analytics.”

Similarly, Gartner has noted “Cloud-based big data services offer impressive capabilities like rapid provisioning, massive scalability and simplified management.” And in the Oracle- and Intel-sponsored report, Forrester wrote, “companies that move more into the cloud for big data analytics achieve greater innovation, increased integration, and higher levels of security.”

The trend toward increasing use of software as a service (SaaS) also makes cloud-based big data services more attractive. Progress Software’s 2017 Data Connectivity Outlook found that “from 2016 to 2017, SaaS adoption increased from 62% to 79%. Out of all the SaaS adopters, more than 50% use two or more SaaS data sources, while more than 35% use three or more.” With so much of their data already residing in the cloud, many enterprises find that it makes sense to leave the data in the cloud for integration and analytics rather than transferring it back on premises.

As previously mentioned, the cloud also makes it easier to take advantage of technological innovations, such as artificial intelligence and machine learning. Hopkins wrote, “By 2020, firms that are not fully leveraging the public cloud for big data analytics will be hard-pressed to keep the pace set by digital leaders that innovate with emerging technology faster than their competitors.”

And the cloud may also offer price advantages over the long term. “Many firms see the five- or 10-year TCO of storing large data sets in the public cloud as too high compared with on-premises,” explained Hopkins. “You should expect prices to be cut in half, while analytic processing and capability double, every few years. Exponentially falling costs and rising power will make the public economic incentives irresistible.”

He concluded, “Enterprise architects must recognize that the combination of big data and public cloud is not just a trend; it is an extinction-level event for digital dinosaurs. Digital predators who get there first will exploit the accelerating cycle of big data innovation in the public cloud, becoming more customer obsessed. Digital dinosaurs will recognize they are too late, will scramble to win back customers, and eventually die off.”

Challenges of Migrating Big Data to the Cloud

Of course, migrating multiple terabytes or petabytes to a public cloud service also has significant challenges.

In the Progress survey, 49 percent of respondents said that having their data spread across an increasing number of data sources was presenting integration challenges. And 41 percent that integrating cloud data with on-premises data was tough.

The NewVantage survey highlighted the role that culture has to play. Among those surveyed 52.5 percent point to organizational issues or resistance as the biggest impediment to their big data projects, and 29.5 percent said their organizations lacked a coherent data strategy.

Companies have to overcome technological hurdles as well. Forrester Research found that integration, security and data management continue to be key concerns.

Source: Forrester Research, “Going Big Data? You Need a Cloud Strategy,” January 2017

Tips for Big Data in the Cloud

So how can organizations overcome these challenges? Experts offer several pieces of advice:

• Develop a roadmap. Hopkins wrote, “Stop investing in on-premises big data capabilities right now.” He recommended that enterprises make a plan for eventually moving all their big data initiatives to the public cloud, starting with the projects that offer the greatest potential value.
• Don’t just “lift and shift.” Analysts and vendors alike said that simply cloning your existing on-premise big data infrastructure in a cloud service is unlikely to achieve all the benefits organizations are hoping to achieve. Instead, they need to rearchitect their solutions so that they are able to take advantage of the unique benefits and automation capabilities offered by the public cloud.
• Decide whether SaaS, PaaS or IaaS will be best for your needs. Enterprises have a lot of different options for big data cloud services, including SaaS, platform as a service (PaaS) and infrastructure as a service (IaaS). Your organization’s existing infrastructure, big data applications, compliance needs and level of expertise may make one more attractive than the others.
• Try out your options. Most public cloud vendors allow you to try out their services at little or no cost for a limited period of time. Experts suggest availing yourself of these trial opportunities before committing yourself to a large-scale big data project.
• Consider a migration partner. Numerous vendors, including the leading public cloud providers, offer services designed to help ease the process of shifting your big data from your data centers to the cloud. Using one of these services could help speed the cloud migration process.

Cloud-Based Big Data Services

Literally dozens of different companies offer cloud-based big data services. Well-known options include the following:

• Amazon Web Services
• Microsoft Azure
• Google Cloud Platform
• IBM Cloud
• Oracle
• Tableau
• Teradata
• Datameer
• TIBCO Jaspersoft
• Hortonworks
• Cloudera Altus
• Databricks
• Qubole

Big Data Services from the Leading Public Cloud Vendors

In the highly competitive cloud and big data market, the offerings from vendor change constantly. To stay current, you’ll want to visit these pages frequently.