Data management trends are coalescing around the need to create a holistic framework of data that can be tapped into remotely or on-premises in the cloud or in the data center. Whether structured or unstructured, this data must move easily and securely between cloud, on-premises, and remote platforms, and it must be readily available to everyone with a need to know and unavailable to anyone else.
Experts predict 175 zettabytes of data worldwide within two years, much of it coming from IoT (Internet of Things) devices. Companies of all sizes should expect significant troves of data, most of it unstructured and not necessarily compatible with system of record (SOR) databases that have long driven mission-critical enterprise systems like enterprise resource planning (ERP).
Even unstructured data should be subject to many of the same rules that govern structured SOR data. For example, unstructured data must be secured with the highest levels of data integrity and reliability if the business is to depend on it. It must also meet regulatory and internal governance standards, and it must be able to move freely among systems and applications on clouds, internal data repositories, and mobile storage.
To keep pace with the enormous demands of managing voluminous high velocity and variegated data day-in and day-out, software-based tools and automation must be incorporated into data management practices. Newer automation technologies like data observability will only grow in importance, especially as user citizen development and localized data use expand.
All of these forces require careful consideration as enterprise IT builds its data management roadmap. Accordingly, here are seven emergent data management trends in 2023.
Hybrid End-to-End Data Management Frameworks
Enterprises can expect huge amounts of structured and unstructured data coming in from a wide range of sources, including outside cloud providers; IoT devices, robots, drones, RF readers, and MRI or CNC machines; internal SOR systems; and remote users working on smart phones and notepads. All of this data might be committed to long- or short- term storage in the on-premise data center, in a cloud, or on a mobile or distributed server platform. In some cases, real-time data may need to be monitored and/or accessed as it streams in real time.
In this hybrid environment, the data, its uses, and its users are diverse—data managers will need data management and security software that can span all of these hybrid activities and uses so data can be safely and securely transported and stored point to point.
IBM is a leader in the data management framework space, but SAP, Tibco, Talend, Oracle, and others also offer end-to end data fabric management solutions. A second aspect of data management is being able to secure data, no matter where it is sent from or where it resides—end-to-end security mesh software from vendors such as Fortinet, Palo Alto Networks, and Crowdstrike can meet this need.
The Consolidation of Data Observability Tools
Because many applications now use multiple cloud and on-premises platforms to access and process data, observability—the ability to track data and events across multiple platform and system barriers with software—is a key focus for enterprises looking to monitor end-to-end movements of data and applications. The issue with most organizations that are using observability tools today is that they are using too many different tools to effect end-to-end data and application visibility across platforms.
Vendors like Middleware and Datadog recognize this and are focused on delivering integrated, “single pane of glass” observability tool sets. These tools enable enterprises to reduce the number of different observability tools they use into a single toolset that’s able to monitor data and event movements across multiple cloud and on premises systems and platforms.
Master Data Management for Legacy Systems
As businesses move forward with new technologies, they face the challenge of figuring out what to do with older ones. But some of those continue to provide value as legacy systems—systems that are outdated or that continue to run mission-critical functions vital to the enterprise.
Some of these legacy systems—for example, enterprise resource planning (ERP) systems like SAP or Oracle—offer comprehensive, integrated master data management (MDM) toolsets for managing data on their cloud or on-premises solutions. Increasingly enterprises using these systems are adopting and deploying these MDM toolsets as part of their overall data governance strategies.
MDM tools offer user-friendly ways to manage system data and to import data from outside sources. MDM software provides a single view of the data, no matter where it resides, and IT sets the MDM business rules for data consistency, quality, security, and governance.
Data Management Using AI/ML
While the trend of using artificial intelligence and machine learning (AI/ML) for data management is not new, it continues to grow in popularity driven by big data concerns as the unprecedented volume of data enterprises are faced with managing collides with an ongoing staffing shortage across the tech industry as a whole—especially in data-focused roles.
AI and ML introduce highly valuable automation to manual processes that have been prone to human error. Foundational data management tasks like data identification and classification can be handled more efficiently and accurately by advanced technologies in the AI/ML space, and enterprises are using it to support more advanced data management tasks such as:
- Data cataloging
- Metadata management
- Data mapping
- Anomaly detection
- Metadata auto-discovery
- Data governance control monitoring
As AI/ML continues to evolve, we can expect to see software solutions that offer intelligent, learning-based approaches including search, discovery, and capacity planning.
Prioritizing Data Security
In the first quarter of 2023, over six million data records were breached worldwide. A data breach can destroy a company’s reputation, impact revenue, endanger customer loyalty, and get people fired.This is why security of all IT—especially as more IT moves to the edge and the IoT—is an important priority for CIOs and a major IT investment area.
To meet data security challenges, security solution providers are moving toward more end-to-end security fabric solutions. They are offering training for employees and IT, since increases in user citizen development and poor user security habits can be major causes of breaches.
Although many of these security functions will be performed by the IT and network groups, clean, secure, and reliable data is foremost a database administrator, data analyst, and data storage concern as well.
Automating Data Preparation
The exponential growth of big data volumes and a shrinking pool of data science talent is stressing organizations. In some cases, more than 60 percent of expensive data science time is spent cleaning and preparing data.
Software vendors want to change this corporate pain point with an increase in data preparation and cleaning automation software that can perform these tedious, manual operations. Automated data preparation solutions ingest, store, organize, and maintain data, often using AI and ML, and can handle such manually intensive tasks as data preparation and data cleansing.
Using Blockchain and Distributed Ledger Technology
Distributed ledger systems enable enterprises to maintain more secure transaction records, track assets, and keep audit trails. This technology, along with blockchain technology, stores data in a decentralized form that cannot be altered, improving the authenticity and accuracy of records related to data handling. This includes financial transaction data, sensitive data retrieval activity, and more.
Blockchain technology can be used in data management to improve the security, shareability, and consistency of data. It can also be used to provide automatic verification, offering avenues to improve data governance and security.
Bottom Line: The Future of Data Management
As businesses confront the need to collect and analyze massive volumes of data from a variety of sources, they seek new means of data management that can keep pace with the expanding need. Cutting edge technologies like AI/ML and blockchain can be used to automate and enhance some aspects of data management, and software vendors are incorporating them into their platforms to make them an integral part of the work. As new technologies continue to evolve, data management methods will evolve with them, integrating them into processes driven by increasing demand.