Data management is the IT discipline focused on ingesting, preparing, organizing, processing, storing, maintaining, and securing data throughout the enterprise. Data management is typically the responsibility of a data architect or database administrator, and the goal is ensuring that the organization’s data is consistent, usable, and secure across all enterprise systems and applications. End-to-end data management is aspirational for most enterprises, but all businesses should have an intentional, overarching data management strategy in place to guide their work.
Table of Contents
How does Data Management Work?
Effective data management is done using a host of software-based tools that render data consistent across all systems, ensure it is of highest quality, and ensure that it meets security and governance standards. While data management is generally the role of a data architect, it engages nearly every IT discipline.
For example, if a business contracts with outside cloud vendors, data management often falls to the IT application manager, an IT security unit, the database group, an IT vendor contract management group, or even outside users and auditors. It is their responsibility to ensure that the data being furnished by the vendors meets or exceeds the standards that enterprises set for themselves.
When new applications and systems access data from other systems, the application team generally works with the database team to ensure that all data is accessible and usable across all system boundaries. The IT storage group or network group might make decisions about where data is ultimately stored. In short, virtually the entire IT team is involved in data management at some point, with the data architect or data administrator giving direction.
Types of Data Management
Data must be managed both when it is “at rest” in data repositories and when it is “on the move” between systems and processors. Accordingly, there are several different types of data management—the foundational pieces of data management are data architecture, data modeling, and data cataloging.
The data architect or database administrator orchestrates an architecture that encompasses the totality of data and information throughout the enterprise. Data might pass through clouds, onsite storage, networks, and applications, for example, but the goal is to gain 100 percent visibility of each piece of data and every IT asset that it touches.
Why is Data Management Important?
The use of analytics and artificial intelligence and machine learning (AI/ML) coupled with the increased digitalization and automation of business processes will enable business strategies to move forward—but none of these technologies will succeed if the data they use doesn’t work.
Data doesn’t work for a range of reasons, from inconsistencies or poor quality data to the inability of clouds, systems, and data repositories to work together and exchange data—or even the inability of users in any corner of the enterprise to access all of the data they need for analytics and AI.
High quality and highly available data comes about when data is rigorously managed, curated, maintained, and secured. All of these operations are fundamental building blocks of data management.
Benefits of Data Management
The purpose of data management is to create the very best versions of data throughout the enterprise. When this happens, management and users have confidence in the data that they’re using. Data management results in better data and company benefits in several ways.
Through both automated and manual means, data managers clean data to eliminate redundancy and inconsistencies across all systems. This reduces the possibility of someone in the company making an ill-informed business decision because of poor data and reduces data storage costs, since redundant data can be discarded.
More Diverse Data
Data managers use automated tools like extract, transform, and load (ETL) to easily transport data from system to system and from repository to data repository. This lets users gain access to a wider variety of data from many different systems. ETL software automation also reduces the time it would take IT to manually integrate different systems and databases.
More Secure Data
Since data security is also part of data management, there is reduced risk of security breaches. Data can better be governed for privacy and other concerns of IT governance, reducing risk for the business.
More Reliable Data
Active data management assures that all data repositories are regularly backed up to latest versions. This prepares the business for any disaster or interruption of service, since data can quickly be failed over and recovered.
Challenges in Data Management
The sheer volume of data companies gather, both structured and unstructured—from both internal and external systems—must be managed on a daily basis, which means it must be prepared and monitored to conform to data quality standards, corporate governance, access and authorization rules, and security.
This can be daunting for data managers with limited staff. As more companies move to cloud, there is also an additional layer of data management—corporate data managers must collaborate with and rely on cloud-based business partners for the stewardship and protection of data.
Volume is just one of the challenges enterprises face in the field of data management.
Diversity of Data
Organizations must develop methods that enable data management across a variety of different systems and databases, each with its own data structure. Often, this requires different tools to manage different IT assets.
Edge and IoT
The installation of more edge IT and Internet of Things (IoT) applications poses new challenges for data management. Collectively, these devices and applications come with wide open endpoints that provide easy gateways into networks and data for malware and virus bad actors. The failure to manage data coming in through these endpoints subjects the organization to higher risks of a security breach that can threaten a company reputation and earnings, and even bring down systems.
Many software automation tools can reformat data, making it inconsistent, or remove redundant data. However, there still are many cases where IT and even users must manually go through data in order to find instances of incomplete or incorrect data. These efforts are time-consuming.
The market has seen an exponential growth in the availability and use of ETL tools and third party application programming interfaces (APIs) that can automate large portions of data integrations between systems and data repositories. Unfortunately, not every integration can be addressed this way. In cases where automation and integration tools don’t work, IT must still integrate data exchanges between systems and data repositories “by hand,” repetitively testing until the integration works.
Examples of Data Management
There are both administrative and technical examples of data management. Here are some of the most common:
- Regulatory and compliance—Outside regulators and auditors want to see documentation and logs of policies and procedures for corporate security and data governance. IT data managers or staff must develop and oversee these activities.
- Applications—An application might require data from several cloud and on-premises systems; that incoming data must be vetted for consistency and quality, and may need to be transformed into a format that can be accepted and processed by downstream systems. The data manager must follow each data chain to ensure that data passes through each point of exchange smoothly, and that the data can securely interact and interoperate with other systems and data.
- End of year reviews/data purges—Storing all raw data that pours into an enterprise can be expensive, and not all of that data is useful. Data managers should collaborate with user managers and regulators annually to determine which data must be kept and which can be jettisoned in end-of-year data purges.
Bottom Line: Better Data Management for Better Enterprise Data
Data management is a multifaceted process that must address all stages of the data lifecycle: when data is created and first enters corporate systems, when it is used and must be moved between systems, and when the data has reached end of life and must be retired. Given the rapid accumulation of data companies experience today, data management can no longer be performed without the help of software tools and automation.
As companies understand the significance of high quality data for analytics, AI, and digital initiatives, they must be willing to invest more into data management and the essential role it plays. Done right, data management ensures consistent, secure, higher-quality data that leads to better outcomes and decisions for organizations.
Read The Future of Data Management to see how the experts believe the field of managing secure, consistent, and quality data is evolving and learn what you need to know to be out in front of the curve.