A famous quote often incorrectly attributed to Charles Darwin states, “It is not the strongest of the species that survives, nor the most intelligent that survives. It is the one that is most adaptable to change.”
Not only is this true for living beings, but also companies. Of the original 500 companies on the Fortune 500 list from 1955, only 52 remain on that list today. Many of the best-known brands of the time – including Studebaker, Zenith, and Detroit Steel – no longer exist while others still in operation have consistently lost market share over the years and faded into oblivion.
In a period of a human lifetime, 78 years, the complexion of the largest companies in the U.S. has dramatically changed – shifting from those focused on manufacturing and raw materials to those focused on high technology, healthcare and retail.
The constant state of creative destruction within the corporate world – spurred by ongoing tectonic changes like the growth of the internet, global pandemics or even artificial intelligence – ensures that only those companies able to adapt to constant change will thrive eventually.
If businesses need to constantly adapt to thrive, then the same must be true for master data management (MDM). This means that if there is going to be a future for MDM, it must be one of adaptability in the people, processes and systems supporting MDM programs.
The Current State of MDM
Unfortunately, the world of traditional MDM looks much as it did 20 years ago. Core processes such as matching, stewardship and hierarchy management all rely on largely the same approaches – which make some fundamental assumptions about the very nature of how master data is managed in large organizations that increasingly fail to align to the realities of digital business:
· One such assumption is the idea that master data must be replicated into a central hub before it can be ‘mastered.’
· Another is that master data models, once established, should consistently remain the same and that a single model can support an entire enterprise.
· A third long-held assumption is that transactional data, by definition, cannot be master data and that there is a tradeoff between the scalability of an MDM program and the amount of data it can support.
Assumptions like these have created the operating principles that allowed MDM to establish itself as a unique and necessary data management discipline 20 years ago (given that at the time, the assumptions were all true).
5 Critical Capabilities for Adaptive MDM
Today, these assumptions hinder the ability of MDM programs to adapt to the needs of modern business, and they must be challenged.
We must revisit not only these legacy principles but all aspects of an MDM operating model spanning people, processes and technologies. For each, we must ask the question, “How do we make this MDM process or system more adaptable?”
When we start to think differently about what ‘good’ looks like for modern and adaptive MDM, five key critical capabilities emerge. These include:
1. Adaptive Data Governance, Data Models and Context-Driven Truth
For MDM to be more adaptive, its supporting governance framework must also be adaptive. In practice, what this will mean is that we no longer cling to the misguided notion that there is only one version of truth for any individual business entity or domain.
A more adaptive approach to both data governance and MDM acknowledges that ‘truth’ is contextually bound and that there are multiple operating contexts in any one business. The ‘truth’ for marketing can be different than the ‘truth’ for finance – and they can both be right. The same is true for the C-suite.
An adaptive approach to governance means that an organization must acknowledge that ‘context matters,’ which also means that multiple sets of governance policies could be in play for any one piece of data based on how it is created or consumed.
This also means that multiple master data models could exist at any point in time. The added complexity posed by adaptive data governance and master data modeling will present massive challenges for most governance organizations. Luckily, AI will play an increasingly key role in augmenting many of these policies.
2. Adaptive Entity Resolution (aka ‘In Situ’ Data Matching)
Since the beginning of MDM as a discipline, legacy approaches to entity resolution have long served to severely limit the scalability and adaptability of MDM programs.
Entity resolution processes required master data to be replicated into a single MDM hub, where compute-intensive, ‘fuzzy matching’ algorithms (which have changed truly little over the last decade+) could slowly churn through data, evaluating every potential pair of records in a dataset.
The more data, the longer these processes would take, which for some large companies could represent several weeks of processing time.[1]
The resource intensity and data replication required by these processes imposed several unnatural limitations on MDM programs.
Most notably, the idea that an MDM program should manage the ‘least amount of data for the greatest amount of value’ became a guiding principle in the industry. Data that could/should be included ‘in scope’ for an MDM program became constrained not by the needs of the business, but by the throughput of the system.
Matching ‘In Situ’ to Address Compute and Implementation Challenges
More adaptive approaches to entity resolution and master data persistence for MDM will involve running match processes ‘in situ,’ or directly against the source data sets, wherever that data resides.
Master data will no longer be replicated in MDM hubs, which will separate master data persistence from the business rules managed by MDM software. Master data hubs will become key registries and follow more of what Gartner would call a ‘registry’ style of MDM.
Modern technologies, such as knowledge graphs, will also be utilized in matching processes to help drive new levels of efficiencies and insights from legacy match processes. Rather than match all data indiscriminately, graphs will inform MDM systems of potential relationships between entities/domains, where detailed probabilistic, ML-assisted match processes will be applied only where they need to be applied.
These changes will lessen many of the constraints previously imposed by legacy approaches to entity resolution – and will drastically expand the universe of data that could be considered ‘master.’ These innovative approaches will fuel entirely new levels of adaptability by allowing a change in thinking away from human-only definitions of master data (that were subject to serious technical constraints), to machine-assisted definitions of master data based on insights gleaned from knowledge graphs.
Rather than having the business conform to the rules and limitations of the system, the MDM program will conform to the realities of the business, as reflected through the insights gleaned during match processes or other discovery processes in adjacent software solutions (such as a data catalog).
3. External Data Sharing
The greatest benefit of MDM is that it allows organizations to exploit economies of scale that exist when organizations share data across functional silos. The difference between how companies define or manage ‘customer’ data (or any other widely shared data) can be minimized, and cross functional efficiencies are exploited.
As economies of scale are realized when companies break silos within their four walls, so too can be they be realized when they break silos that naturally exist between companies. More adaptable forms of MDM will leverage external data sharing – which includes sharing of a limited set of data and governance rules – where that sharing optimizes processes in which each party has a vested interest. Any complex supply chain is an example where more widespread sharing of data and governance policies could produce outsized results.
These results would not be limited to just business results, but also the efficiency of data management programs themselves. The creation of data sharing and governance ecosystems would mean that instead of 100 companies all maintaining and managing/governing/stewarding the same data repeatedly, it could be managed as a shared asset.
Everyone in the ecosystem would benefit from the update of a single record, rather than each company updating it 100 times over. The incentives to update or steward a common asset in a shared data ecosystem could potentially be provided via some form of blockchain-supported system, not unlike the incentives that exist for Bitcoin miners to validate transactions in that ecosystem.
4. Multi-dimensional Hierarchies
Multi-dimensional hierarchies. Adaptive MDM approaches in the future will include new ways of managing data about relationships. Instead of relatively ‘flat,’ single-dimensional relationships, adaptive MDM programs will embrace multi-dimensional hierarchies that manage second-, third- and Nth-degree relationships within data.
As witnessed in the COVID-19 pandemic, simply knowing that Acme incorporated is (or is not) your supplier is not enough. In a business world that requires highly adaptive forms of MDM, you will need to know who the suppliers of your suppliers are to sufficiently manage/mitigate any supply chain risk.
5. AI-Augmented Everything
Yes, AI will most certainly impact MDM. Its greatest impacts will not necessarily be the augmentation of data stewardship and data quality (although those will certainly happen, and already are), they will be on the management of master data itself.
This includes the augmentation of the:
- Creation and enforcement of governance policies
- Identification and classification of master data (i.e., what is, what is not master data)
- Definition and creation of master data models
- Identification of master data hierarchies and relationships
- Business rules used for entity resolution
- Business rules used for the creation of gold master records and survivorship
The augmentation and automation of many master data and related governance processes will allow MDM programs to significant scale by removing many of the limitations historically associated with significant levels of human oversight – making them far more adaptable than ever before.
As these capabilities are developed and evolve within organizations, so will the people and processes needed to support them – a subject of a separate future blog post (spoiler alert – adaptive master data management requires change management and product management as critical enabling competencies).
Companies failing to evolve all aspects of their MDM operating model needed to operationalize these new adaptive approaches will fall behind those that can.
The lesson of the 445 companies no longer on the Fortune 500 should be a lesson to all data practitioners responsible for an enterprise-wide MDM program – adapt and evolve, or risk fading away for good.
Profisee Adaptive MDM for Modern Digital Transformation
Profisee understands that modern digital transformation and business goals require a modern, adaptive approach to MDM.
Profisee’s adaptive configuration, stewardship and architecture help hundreds of customers implement quickly and scale across their business needs – while leveraging OpenAI integrations to augment and streamline ongoing stewardship.
SEE THE PROFISEE ADAPTIVE MDM FOR YOURSELF
Watch a demo on-demand to experience the Profisee Adaptive MDM platform first-hand and learn how you can get started quickly, streamline data stewardship with an AI co-pilot and add value in weeks, not months.
Malcolm Hawker
Head of Data Strategy @ Profisee
[1] I once had a conversation with the Chief Architect at a F100 company who told me that running a full match of all their master data records, if they had to start from nothing, would take ‘several months.’
