Gen AI Is How Most Companies Will Operationalize AI at Scale

Earlier this year I had the pleasure of giving a presentation at a Gartner Data and Analytics event with Piethein Strengholt, author of Data Management at Scale, where we shared our perspective on how gen AI-enabled chatbots and smart agents will be the primary way that most organizations operationalize AI in the short term. (He and I shared many of our perspectives in an episode of my CDO Matters Podcast, available here.)

I am confident that most companies — especially those without established data science functions — won’t be in the business of building or training large language models (LLMs). Major concerns for how to govern and manage data used to train AI models will be for the minority of companies willing to make significant investments in a data science function and supporting data management and governance processes.

For the rest, the biggest data management issues surrounding the use of smart agents and chatbots — including those using tools like Azure OpenAI Studio to build their own chatbots — is the use of RAG (retrieval augmented generation) patterns within complex prompts to mitigate hallucinations of out-of-the-box LLMs. Given generative AI solutions are built on and optimized with unstructured data, the use of internal data from structured sources within RAG patterns is limited, but slowly expanding.

Using knowledge graphs in RAG patterns to provide the necessary context about structured data so it can be consumed in a complex prompt holds great potential. This means knowledge graphs should be on any company’s radar that’s looking to bridge the world of structured data and the data chatbots consume in fine tuning processes, or complex prompting.

Focus on the Right Foundations: Unstructured Data and AI Governance

When ChatGPT burst on to the market, many pundits and consultants told data professionals that they needed to “focus on the foundations” of data management to ensure they were best positioned to build or use an AI-based solution. While this is true, exactly what foundations you focus on is extremely important, as are the types of AI you intend to use.

Generative AI solutions consume unstructured data. The reality of most data governance programs today is that most unstructured data is entirely un-governed — a reality I highlighted in an article I wrote for Forbes. If you intend to build or fine tune your own language models — or use unstructured data on complex prompts to “ground” the behavior of LLMs — then bringing your unstructured data into your data governance program is critical.

Another consideration is that it’s typical at companies with established data science functions for data scientists to not use the output of more traditional data management processes, such as data quality and master data management. Instead, data scientists extract data straight from source systems. What this means is that a “focus on foundations” through more traditional methods like data quality, governance and MDM processes will not benefit data scientists.

To realize the economies of scale a governance process provides, it’s important for these companies to find ways to integrate their data science functions into more traditional data management operating models. A great starting point for many companies here is the use of a single integrated cloud data management platform, such as Microsoft Fabric running in the Azure ecosystem.

AI Requires a Different Mindset

Another impact that AI is having on data management is that data leaders need to think differently about how they manage and govern data for AI use cases. AI-based solutions are inherently probabilistic, which means that applying legacy deterministic approaches to how you manage data designed for AI-based solutions is a bad fit.

One example is having the same data quality expectations for both AI and more classic analytics. In many situations, it’s fully expected that data used to build AI models will have a certain degree of variability since the goal of the model is to ensure the data used is representative of the broader dataset. Data used for analytics is held to a different standard, where the expectation is that data represents the most accurate expression of a business process.

In other words, the quality of data used by AI-based systems should model reality (the way things are), while the quality of data used for analytics should model an ideal reality (the way things should be).

The probabilistic nature of AI-based systems also means that applying highly rigid, broad-based quality standards to data used to build or optimize AI models is a bad fit. The quality rules should ultimately be defined not by enterprise-wide standards used by analytical tools but by the specific quality rules needed to support the AI use case.

Context Is King

The ability to use AI to start augmenting traditional data management processes (like data quality rules) requires additional insights into the context of how data is used across the organization. For many, this added context comes from the use of graph technologies to reveal previously unknown relationships in data.

The combination of knowledge graphs and AI to analyze large amounts of transactional data and metadata allows for the creation of AI models and what Gartner calls “recommendation engines” that will eventually enable the automation of many data management processes, including data governance. I wrote about this four years ago at Gartner, and significant progress has been made since then.

This combination of AI and advanced analytics is a foundational component of a data fabric, the capabilities of which go well beyond a data access and virtualization layer. My CDO Matters Podcast episode Demystifying the Data Fabric provides much more insights into the significant benefits of a data fabric, particularly around the automation of data management.

Examples here include:

• Using a combination of graph insights and some form of machine learning to start augmenting the process of entity resolution (matching) used in master data management systems — which vendors are increasingly offering
• Using graph-based insights to help augment or automate the process of modeling data and data hierarchies
• Using AI to recommend specific data quality rules designed to optimize the efficiency of a given business process

Vendors will increasingly enable these capabilities, but data teams with skilled resources in both graph and data science will be able to build their own.

Metadata as a First-Class Citizen: The Growth of Data Catalogs

Another impact of a heightened focus on AI in data management is the increased importance of metadata management. For many, this starts with the implementation of some form of a data catalog, which allows organizations to create robust inventories of their data estates.

With a growing focus on governing unstructured data, data teams are increasingly relying on catalogs to be the primary repository of information around the acceptable use of data within an organization — particularly for use by AI based systems. Rising popularity of data products is also fueling the growth of data catalogs — which, for those with a data science function, includes AI-based data models and products.

Data Management Software Systems Are Converging on the Cloud

The proliferation of data management and governance silos in organizations — including those which today separate data science from traditional analytics — creates the need for vendors to provide more holistic environments that combine multiple data workloads, pipelines and management applications into a single cohesive, interoperable cloud-based infrastructure.

Microsoft Fabric is a great example of this, as it combines data science, data warehousing and classic analytics into a single management platform. Other vendors like Snowflake and Databricks offer similar solutions. The ongoing explosion of data volumes creates even more motivation to move data workloads to the cloud, as does the expansion of computing power needed to build and test AI-based models.

These forces are converging, motivating data leaders to consolidate on a single IaaS service provider.