Hype is Fading, Knowledge Gaps Remain

As of early March 2024, it appears that much of the initial hype around generative AI and large language models (LLMs) is fading. For many, the hype is taking a back seat to all the hard work required to meaningfully and responsibly integrate these new technologies.

Based on my travels and my many ongoing conversations with data leaders across the globe, about half of all companies are using some form of gen AI (with most aspiring to do more) while another half is still trying to figure things out.

For this latter population, many still have significant concerns about their overall AI readiness as well as the quality of their data to support AI use cases. I spoke with a senior leader at a big company recently who specifically said they aspire to create gen AI models using internal data sitting in their data lake, even though most of that data is in a structure not suitable for training gen AI models.

While concerns about the quality of enterprise data are often valid, it’s still surprising that many of us do not recognize that text data is the preferred format for training gen AI models — and more importantly — that text data is usually completely ungoverned in most companies. This means that concerns about the suitability or quality of using existing data for training gen AI models are valid — but not for the reasons many cite.

The Fastest Path to AI Value Realization

Over the last year, it’s become abundantly obvious to me that the fastest path to utilize gen AI technologies in most organizations is through the adoption of chatbot and copilot-enabled smart agents. These agents essentially act as gen AI enabled apps, which mediate exchanges between the user and an LLM.

While it’s still important for users to understand how to optimally engineer prompts, in time these agents will increasingly become more context-aware and context-specific, not unlike using a steak knife to cut meat. You can use any knife you want to try to cut meat, but one is particularly well suited for the job.

The same will be true with copilots and smart agents, where agents will increasingly be built specifically to support specific use cases. This is already happening. As an example, look no further than the OpenAI copilots being deployed within Microsoft applications, such as the Power BI copilot. In many ways, it’s not incorrect to view these agents as rather powerful query writing bots, which will enable just about anyone to become a data analyst in your organization.

Operationalizing Chatbots and the Air Canada Debacle

Using copilots and smart agents will not negate the need to ensure the data they are consuming is accurate, consistent and trustworthy. (Although I do suspect that they will increasingly leverage capabilities — like using RAG patterns or intermediary queries to graph or vector databases — to reduce hallucinations and improve response accuracy.) But again, the reliability of what the copilots tell users will very much depend on the data they consume.

Defining internal policies for the proper use and configuration of these copilots, be they internally or externally facing, will be a critical dependency for their successful deployment.

Look no further than the recent news story citing an Air Canada chatbot that gave incorrect information about the company’s bereavement policy to a grieving customer. In a rather shocking twist, rather than admit their mistake and simply credit the customer for a ticket, Air Canada took the issue to court, attempting to claim that their AI smart agent wasn’t an agent of the company. Huh?

Thankfully, the grieving customer won their claim and Canada’s worst airline was forced to do the right thing.

When I first read about this issue, I naturally concluded that the chatbot had hallucinated, but based on the news stories I read, it’s unclear if that’s the case — even though that’s exactly how most media outlets framed the narrative. It’s equally possible that the information the chatbot cited was incorrect at the source, or it was incorrect when it was consumed during a training or fine-tuning exercise.

Put another way, the root of the problem highlighted in the Air Canada story could very well have been a software engineering or data governance problem and not a problem with the underlying LLM. My previous point about text data being “ungoverned” (or very poorly managed) could very well help explain what many have been prompted to believe is a problem with the AI.

The same is equally true with the behavior of the chatbot, which may have been a custom bot built on top of an opensource LLM that was incorrectly engineered.

The Air Canada chatbot has since been decommissioned, and the company left with a major PR headache. As nebulous as the cause of the issue may be, it certainly highlights the many issues that companies must navigate on their road to AI value realization.

This navigation must necessarily include the creation of acceptable use policies for AI models within companies (at a use case level), and it must also include policies that define how software engineers must interact with data scientists in the creation and deployment of any custom built or configured/fined tuned gen AI solution.

Data scientists building or configuring gen AI models can no longer operate in a vacuum and must work closely with their peers who are building applications that interact with their models. This is especially the case during both model development and model testing.

The same is true with people using gen AI internally, such as a software engineer using a copilot to automate some portions of code writing or testing. Changes to existing software development lifecycle policies must be made which ensure that any code generated by a copilot goes through the same level of rigor (or maybe even more) as a human would.

Are these policies “data governance” polices as we’ve historically defined them? I don’t think they are, although many data practitioners seem to be suggesting they are and that any use of AI requires “AI governance.”