HPC Storage Grows Cloudier, Flashier

Organizations running high performance computing (HPC) workloads are increasingly seeking out cloud-based storage solutions and speedy flash-enabled systems to help them cope with growing complexity and the sheer amounts of data they are managing nowadays, according to new research from DataDirect Networks (DDN).

For starters, organizations are making use of more data, the company found in its survey of over 100 HPC professionals. Eighty-five percent of reported that they are using or managing more than one petabyte (PB) of storage, a 12-percent increase compared to last year’s results. Nearly 30 percent said they in charge of over 10PB of storage.

Cloud Storage and Backup Benefits

Protecting your company’s data is critical. Cloud storage with automated backup is scalable, flexible and provides peace of mind. Cobalt Iron’s enterprise-grade backup and recovery solution is known for its hands-free automation and reliability, at a lower cost. Cloud backup that just works.

SCHEDULE FREE CONSULT/DEMO

To help them cope, HPC specialists are turning to the cloud, albeit cautiously.

Nearly half (48 percent) of all respondents said they planned to stash at least some of their data on a public or private cloud, an 11 percent jump compared to 2016. Yet, only five percent of those polled said they expect to place more than 30 percent of their data in the cloud.

Regardless, cloud usage is on the upswing. Forty percent of respondents said they planned to place at least some of their data on the cloud, compared to only 20 percent a year ago.

Flash storage is another technology organizations are using in their complex and resource-intensive HPC workloads.

While flash usage in HPC environments remains steady at 90 percent in basic data center deployments, the amount of data being placed on solid-state drives (SSDs) and other forms of flash storage is on the rise. Currently, most organizations (76 percent) have less than 20 percent of their data on flash but many more organizations expect more of their data to get soaked up by flash-enabled storage systems.

Twenty-five percent of respondents said they expect to pump 20 to 30 percent of their data into flash storage in 2018, while another 10 percent are estimating that they will place 20 to 40 percent of their data on flash storage. Accelerating file system metadata (54 percent) and application-specific data (54 percent) are the top flash use cases.

In terms of challenges, I/O (input/output) bottlenecks remain a big worry among 76 percent of organizations using running complex analytics downloads. Despite flash’s ability to transfer data at a brisk pace, at least compared to traditional hard disk drives, it’s not the cure-all for HPC environments, according to DDN.

“I/O performance is a huge bottleneck to unlocking the power of HPC applications in use today, and customers are beginning to realize that simply adding flash to the storage infrastructure isn’t delivering the anticipated application level improvements,” said Kurt Kuckein, director of marketing for DDN, in a statement. “Suppliers are starting to offer architectures that include flash tiers optimized specifically for NVM [non-volatile memory] and customers are actively pursuing implementations utilizing these technologies.”

Pedro Hernandez is a contributing editor at Enterprise Storage Forum. Follow him on Twitter @ecoINSITE.

RELATED NEWS AND ANALYSIS

• Huawei’s AI Update: Things Are Moving Faster Than We Think

  FEATURE |  By Rob Enderle,
  December 04, 2020

• Keeping Machine Learning Algorithms Honest in the ‘Ethics-First’ Era

  ARTIFICIAL INTELLIGENCE |  By Guest Author,
  November 18, 2020

• Key Trends in Chatbots and RPA

  FEATURE |  By Guest Author,
  November 10, 2020

• Top 10 AIOps Companies

  FEATURE |  By Samuel Greengard,
  November 05, 2020

• What is Text Analysis?

  ARTIFICIAL INTELLIGENCE |  By Guest Author,
  November 02, 2020

• How Intel’s Work With Autonomous Cars Could Redefine General Purpose AI

  ARTIFICIAL INTELLIGENCE |  By Rob Enderle,
  October 29, 2020

• Dell Technologies World: Weaving Together Human And Machine Interaction For AI And Robotics

  ARTIFICIAL INTELLIGENCE |  By Rob Enderle,
  October 23, 2020

• The Super Moderator, or How IBM Project Debater Could Save Social Media

  FEATURE |  By Rob Enderle,
  October 16, 2020

• Top 10 Chatbot Platforms

  FEATURE |  By Cynthia Harvey,
  October 07, 2020

• Finding a Career Path in AI

  ARTIFICIAL INTELLIGENCE |  By Guest Author,
  October 05, 2020

• CIOs Discuss the Promise of AI and Data Science

  FEATURE |  By Guest Author,
  September 25, 2020

• Microsoft Is Building An AI Product That Could Predict The Future

  FEATURE |  By Rob Enderle,
  September 25, 2020

• Top 10 Machine Learning Companies 2020

  FEATURE |  By Cynthia Harvey,
  September 22, 2020

• NVIDIA and ARM: Massively Changing The AI Landscape

  ARTIFICIAL INTELLIGENCE |  By Rob Enderle,
  September 18, 2020

• Continuous Intelligence: Expert Discussion [Video and Podcast]

  ARTIFICIAL INTELLIGENCE |  By James Maguire,
  September 14, 2020

• Artificial Intelligence: Governance and Ethics [Video]

  ARTIFICIAL INTELLIGENCE |  By James Maguire,
  September 13, 2020

• IBM Watson At The US Open: Showcasing The Power Of A Mature Enterprise-Class AI

  FEATURE |  By Rob Enderle,
  September 11, 2020

• Artificial Intelligence: Perception vs. Reality

  FEATURE |  By James Maguire,
  September 09, 2020

• Anticipating The Coming Wave Of AI Enhanced PCs

  FEATURE |  By Rob Enderle,
  September 05, 2020

• The Critical Nature Of IBM’s NLP (Natural Language Processing) Effort

  ARTIFICIAL INTELLIGENCE |  By Rob Enderle,
  August 14, 2020