Why Every AI Decision Still Needs a Human’s Name with Kelle O'Neal

Good morning, good afternoon, good evening, good whatever time it is, wherever you are in our amazing planet Earth. I’m Malcolm. I’m the host of the CDMatters podcast. Thanks for joining today.

It’s almost a miracle that we’re here because Kelly and I were just wrestling with technology related issues. We couldn’t find mics, couldn’t make sound work. Just it’s just all the things.

So I’m glad you’re joining today. We’re gonna have an abridged version of the CDO matters podcast. We’re we’re only gonna talk for about twenty, twenty five minutes, but what we’re gonna talk about, what what happens in a world when we give agents autonomy to make decisions. We’re gonna talk about decision making.

Yep. I’m thrilled today to be joined by Kelly O’Neil, who’s the CEO and founder of First San Francisco Partners. If you’re not familiar with them, you should be. They know data.

They know our space. They get what you do for a living. So, Kelly, thank you for joining today.

Thank you for having me.

We made it. We’re here.

Yay. Yay. Yay. Yay. The good news is now we know that it can work.

Oh my gosh. Oh my gosh. We we like, for for those of you who are listening at home, you’re like, what are they talking about? We we wrestled for half an hour to make our our audio settings work, which is kind of crazy.

But, Kelly, I I first decided that I wanted to talk to you about two and a half years ago, maybe even three years ago, when we were at a DGIQ event, and you were giving a presentation, and you were talking about the changes related to unstructured data, the changes that we need to do to our governance programs to better support GenAI, and you were saying things that nobody else at the time was saying. And I remember I was in the front row, I’m screaming like, yes. Yes. Yes.

Yes. And at that very moment, I think it was about two years ago, I decided that is somebody whose brain I need to pick. I’m so glad you could join us today.

Absolutely. And now I’m like, what was I saying?

Oh, it was no. It was you were you were making complete sense. It had something to do with with unstructured data changes to governance foundations to support Gen AI Yeah. Acting differently, thinking differently.

And anytime I hear anybody talking about doing things differently, then, like, that’s it. Birds of a feather. Let’s chat. Yes.

So well, tell me tell me what you’re working on with when you’re with with your clients in relation to decision making. And, you know, the whole topic of today is what happens when we give decisions over to software, and and are we ready to do that? What what are some of the cooler things you’re working on?

Yeah, so we kind of house this conversation under the AI governance umbrella. Now, that is obviously a category that has you know, unclear boundaries right now because, I mean, I was just reading another article this morning where the definition of AI governance doesn’t really align with kind of what we think about with AI governance, but I think it from the realm of decision making. So kind of my background is kind of in the data world as opposed to in the tech AI world, specifically data governance. And so when we thought about data governance, we always thought about good governance was good decision making.

But of course, now that is no longer a valuable construct because the decision making is now what needs to be governed. So we’ve had to completely flip this on our heads and everything else, because then the question becomes, how do we govern a decision where there’s some traceability and understanding, but not entire traceability and understanding. And that’s where it becomes really complicated. So we can think about governing the decision at an atomic level within that autonomous AI thing.

But then what happens is, and I know I’m not answering a question right now, Malcolm, but I’m just kind of talking about like how the, why it’s complicated. So we can think about governing an agent and trying to create transparency and understanding and therefore a level of control within an agent. But companies are looking at deploying thousands of agents, tens of thousands of agents. And they’re looking at daisy chaining agents, which means daisy chaining decision making. And as of now, most AI decision making is largely probabilistic, which means that they potentially never make the decision twice. And so this is just kind of where this complication occurs. And then we think about who in the organization is in charge.

Right? So these are all of the things. And so I don’t know, Malcolm, if you want to just start picking apart the questions, but this is what we’re trying to help our clients with right now is helping them to just think through these decisions and make the best decision they can at the time around how to do this. And then how frequently do we need to reassess whether that was still the right decision, whether that was still the right organizational construct, whether that was still all of those right things, because things are changing. I mean, every three to six months we’re like, wow, okay, now what?

Well, so what you just described goes a little bit beyond data ownership.

Oh yeah, 100%. Right?

I mean, we’re so fixated with this idea of making the business own the data. I don’t even really necessarily know what that means. I mean, I do know what it means, but it’s such a nebulous term and I think it’s basically an attempt to offload accountability.

That aside, what you’re talking about is something drastically different. Like put the data aside, right? We’re data people and we love the data and we cherish the data and the data is important. But what you’re talking about is accountability at the point of execution of the decision itself, which is very different than governing the data. You’re governing the behavior of this system.

So what do you when when you’re peeling that onion, what are you finding? Are you are you finding that there there the companies, do they have these things well documented? Do they know who’s responsible for x and who’s responsible for y? I assume that when you start looking at this, what you find is a is a is a is a bit of a mixed bag. Like, accountability all over the place. Yes?

All over the place. And this is also where I think when we try and look at it from an organizational level, that companies really need to think about this. They need to be recognizing that the pressure is coming from their board. I don’t care how big or small or whether you have an official board or an unofficial board, but the board is coming in and saying, you must do AI in order to be competitive.

And then you’ve got the activity happening from the ground up where everybody has their favorite tool that they work in. I don’t care if that is Outlook or if it’s ERP of choice, Salesforce automation of choice, like I don’t care what large vendor you talk about, but there are agents built into everything now. And then people will start using agents without really even knowing that they’re using agents kind of thing. Right? So that is where this needs to eventually come to some.

Consistent decision making in an organization that matches the level of risk and over oversight and ethical guidelines of that company. And it needs to be done thoughtfully and like it just needs to be done. So I had a really interesting conversation, Malcolm, with a guy who reports two steps away from the CIO of a massive airline.

And they never adopted the idea of a chief data officer or a chief analytics officer or any of that. Well, here comes AI, and they already have agentic sprawl that they cannot even wrap their heads around.

So if they never had this idea of a top data job, I said, well, you know what? We need to from my little world, your company should be thinking about a top AI job.

That might be a hard thing for your company because you never adopted the idea of a top data job.

And the reason I make that correlation is because of the complexity of the top data job and therefore the complexity of the top AI job. And so both are very analogous to me. And when we think about the first CDOs that started coming out, the job was undefined. They had like very few resources.

They were trying to figure it out as they go. And this is what’s happening with the top AI job. And it is going to be very important and it is going to be entirely federated. There is very little way to centralize all of this.

And we should still make an attempt to centralize as much as possible and create consistency around anything AI oriented where we can.

This sounds like a minefield to me. Like, so I think back to four or five years ago and everybody’s talking about digital transformation.

Yeah.

Right? And CDOs, whether they wanted to or whether they didn’t want to, many of them were stuck in that role as a change agent Loosely tied to some idea of a digital transformation. Right? All the boards then were talking about digitally transform.

Hey, CDO, you know, you’re you’re horizontal. You know the data. You’re gonna use the data to transform organization. Go do it.

Yeah. And many of them hit a hit some walls in that maybe they weren’t really good at change or they weren’t change agents or maybe they ran into these highly federated organizations where the CDO had a mandate, but the leader of a function didn’t have a mandate and there was friction there. Right. What I’m, what I’m maybe hearing you say now is that now, now CDOs may be walking into this minefield where it’s like, hey, go, go deploy an agent to solve x y z, where you get into actual decision making.

Like that’s authority in an organization where the CDO has to figure out who actually has the authority and maybe what they’re finding is is a political mess. Is mine making any sense?

Yeah, you know what, it is. And I think there’s this resurgence of the importance of the data job, the importance of data governance. So I love that aspect, Right? All of a sudden, people recognize that the data is important.

Love that. So this partnership between data and AI needs to be solidified and needs to be, I know people don’t like to do this, documented, agreed upon, revisited, because there’s so much transition from a job perspective that the companies need to have some sort of structure beyond the people, because when the people start to leave, the structure needs to stay. And so what we’re seeing is that, like from a data governance perspective, they are trying to move as far as they can closer and closer to AI so that the data is ready for AI and AI is ready for the data. It’s fantastic, right?

All of this innovation, there’s agentic value within that pipeline.

From an AI governance perspective, we’re also seeing organizations where AI governance is wanting to then get into data governance because they need to have trust and transparency.

While that’s really, really a good thing, because of course we love it when people are just interested, it can be a massive duplication of effort. And so that’s why things need to be solidified and confirmed so that it’s clear where the collaboration is, where the handoffs are. Otherwise, it’s just colossally unproductive and duplication of effort.

So it sounds almost to me like a decision framework or some idea of the decision framework, right? Where what you’re kind of talking about is something that probably looks a little bit like a racing matrix for a task that an agent has been given to execute on. Is that kind of what the final form or like you figuring out person A is responsible for B, agent A is responsible for C? Is that kind of what it ends up looking like?

You know what? Fundamentally, we go back to all of these fundamentals. So absolutely things like RACI matrices, things like just anything about that solidification and communicating broadly around authority and then really giving people that authority. And so they become essentially agent managers.

And so in the same way that we have our org charts and our operating models, we should have these agentic oversight charts or categories or whatever we want to call it. And then also, want to understand within that decision making, how are we parsing out those aspects of the decision making that we can control and the ones that we can’t control because they’re in a black box that somebody else developed that we really like, you’ll bang your head against the wall trying to figure it out because somebody’s not giving you the source code for that.

So yes, I would reuse all of those traditional concepts as much as possible.

So ultimately though, I assume any decision, if you are empowering an agent to make it, still needs to have a human name under the, you know, accountable column on that racing matrix so that if an auditor comes calling or somebody comes calling, ultimately a human would end up having accountability for the behavior of the machine, correct?

That is what the regulatory environment is demanding as it should be. Because even if we are using AI to help govern the AI agents, at some point, we’re setting up decision rights for those different agents, and we’re saying it’s Okay to decide on this. It’s not Okay to decide on that. That is where the humans still have a lot of involvement. Creating those, like you said, decision frameworks, making sure that those decision frameworks are shared across the organization so other people know about it. And then you can start to program that into the different layers of decision making within the AI solution.

And so I know that we don’t have as much time as we originally wanted, which means that we’re going to have a sequel, Malcolm. Yay. And we can really like start to parse that out in terms of what are all of the layers in which we have the ability to create some control and to create some audit ability, either by other agents or by humans, but ultimately have some trust that we’re not going to expose our company to some massive ethical catastrophe or just a huge amount of risk on a customer by customer basis, employee by employee basis, right? When it comes down to it, it’s how the company is still perceived in the market, whether I’m an employee or a customer or a partner. None of that’s changed.

Well, mean, if I’m if I’m somebody whose name that shows up on that under the a, under accountable for a decision that we have given over to the agents, You said something very early in our conversation. We were talking about these are these systems are inherently probabilistic. And as as much as we may be working to ground them, as much as we may be trying to give them context and and give all of this additional data so that they have every chance in the world of being accurate and consistent and predictable, I still think there’s a level of unpredictability there that would make me a little squeamish to put my name next to a decision these agents are making.

Yes. You see that? Yeah.

Oh, yes. And it should be, right? So it should be that somebody is all of a sudden like, oh my gosh, like, am I really authorizing that at scale?

So it should be. It should be that someone ends up uncomfortable and they start asking questions, because the more questions that you ask, the more thought goes into what possibly could go badly. And that’s sadly, I’m not a negative person. I don’t want to be the one that’s like, well, tell me all the things that can go wrong.

But someone needs to ask all of those questions so that we do understand what are the risks based on these decisions. And then if we kind of, again, daisy chain decisions or we have some level of collaboration, then we have those conversations and the board who’s pushing the AI pressure understands, look, here’s some big decisions we need to make, and we need to understand as a company our level of risk and how much authority we are willing to give some of these agents or not. Those are big decisions that need to be that that decision making structure needs to be created within an organization.

And we have those decision making structures already. We have data decision frameworks many times. And if we don’t, let’s formalize them. There is something that exists in every single company, big or small.

Formalize those data decision frameworks, see how they can be extended into AI and decisions within AI, and then start to use them and determine how to make the company comfortable, and also how is this going to impact an actual organizational chart, because we know that data decision frameworks are not org charts. They are accountability references and communication, you know, frameworks and things like that.

Well, so this starts to look a little different from the perspective of managing risk.

I mean, everything that

I heard you just say almost starts to sound like how an underwriter would talk about writing an insurance policy for, like, you know, black swan events or some other, like, unanticipated weather event where you are where at the end of if you’ve done everything you can do and there’s still a window, a probability of the agent doing something dumb and causing risk to the corporation, whether that’s brand risk, whether that’s cash risk, it doesn’t matter what form of the risk is.

I I would imagine there’s a growing industry here around insurance to indemnify leaders who may be reluctant to put their names onto something that that that you can’t that is inherently uncontrollable. I I don’t I don’t know. That sounds a little bit strange, but I have to think that that’s a growing field.

Well, imagine. I mean, and liability insurance. Right? Right. Exactly. Hopefully, you have it. I have it.

Right.

So, you know, that

You must be talking to the guy behind me.

Exactly. But that’s the thing. So there has been that where there is the you are insuring yourself against professional accountability, you know, but I can’t remember the term. I’m feeling like it’s like, you know, anyway, whatever.

But there is. And if there’s not Malcolm, maybe you have your next career because that would be a very important level of both insurance and risk assessment process. So yes, I never really thought about it as an actuarial activity. And yes, I imagine that it is a very similar activity if you’re truly trying to quantify the risk associated with, you know, that kind of broad agentic decision making.

Yeah, so in our last five minutes, something else we had talked about was this idea of governance as code. Oh. And and I’m very intrigued by this concept because there are many of us who think that governance, whether it’s data governance or whether it’s AI governance or whether it’s all of it, is is this inherently human undertaking where it’s a little bit of art, a little bit of science, a little bit of intuition, little bit and there’s no way that you could ever automate any of this. What you’re saying is that it necessarily must be automated.

Oh, yeah, absolutely. Because if it’s not automated as much as possible, you will never keep up. It is just not possible to keep up. So I would reframe it such that all of that creativity and that human brain power and involvement and all of that, that is done in the upfront. So we’re designing these systems, leveraging all of our brilliance, and then we’re implementing it into code.

So it’s not that that opportunity for human creativity and evolution goes away. It’s just that then it is put into code, whereas in the past, we’ve really kind of and maybe it’s just me. We’ve really leaned into the people and process aspect of governance in order to make sure that we’re making the right organizational decisions around data, which is still fantastic. It’s just then to do it at speed, we need to put it into code as much as possible.

So the same work is done. It’s just then codified. And there are some places where you can codify it more easily. I mean, access control is the absolute minimum viable. But then as you’re breaking down, for example, if it’s a regulatory environment, what truly is an obligation that I have to comply with that regulatory requirement? What does it mean to have that obligation? Let’s parse that out, just like we parse out other aspects of data decisioning.

And then how do we determine what is possible to automate at all of those different levels so that ultimately that obligation, there’s trust that that obligation is taken into consideration?

Again, I’m going back to kind of fundamental things that we’ve done in the data world all along because we, for example, master data management, right? That is essentially a codification of decisions.

Exactly right.

It always has been. So let’s just take that to the next level. And so anyway, sorry.

That brings a whole new definition to or even potentially negates the idea of a human in the loop because there’s no way humans can be in the loop. If you’re if the reason you’re deploying agents is to realize the benefits of scale Right, throwing a human into the middle of all of that will slow it down to human pace and you’re not gonna have the the the same benefits. Now at the same time, I think as a part of your decision framework, you could you could I mean, not all decisions are created equally. Not all not all exceptions are created equally. You could certainly inject a human at the at at when there is a really serious issue going on or when the AI says ding ding ding, hey, there’s something touching PII or HIPAA or who knows, right? Where where you could write code that said in this and these diarist of edge cases inject human else, go forward. But I’m telling you, this is this is interesting.

If you’re a CDO and you have been given an an agentic mandate, be prepared to wade into areas of the organization around authority and control for making decisions that maybe you hadn’t planned for.

And that’s when you may want to call a Kelly O’Neil. Kelly, how would somebody get in touch with you if they wanted to better understand the value that you can bring from the perspective of helping under organizations understand how decisions are made?

Yeah. Well, my marketing person would say go to first San Francisco partners dot com. You know, honestly, it’s super simple. Kelly, kelle@firstSanFranciscopartners.com. It’s super simple and meaning to reach out.

And we all need to be asking those complicated questions.

I think

you’re entirely right, Malcolm.

Indeed. Or go to a data conference where Kelly is speaking because she is a vibrant, charismatic and excellent public speaker. And I recommend you attend her session.

With that, thanks for tuning into an abridged version of the CDO matters podcast.

Kelly, thank you for your

time today.

Welcome.

With that, if thank you. With that, if you’ve made it this far, please take a moment to subscribe, to like, do all of the things. I hope you see us and join us on another episode of the CEO Matters podcast sometime very soon. With that, thanks and bye for now. Bye, guys.