Putting AI Governance Into Practice in Financial Services

Jim Jockel: Welcome to trading tomorrow, navigating trends in capital markets, the podcast where we deep dive into technologies reshaping the world of capital markets. I'm your host, Jim Jockel, a veteran of the finance industry with a passion for the complexities of financial technologies and market trends. In each episode, we'll explore the cutting edge trends, tools, strategies driving today's financial landscapes and paving the way for the future. With the finance industry at a pivotal point, influenced by groundbreaking innovations, it's more crucial than ever to understand how these technological advancements interact with market dynamics.

As AI moves deeper into finance, it's no longer just about what technology can do. It's about how it's managed. From model risk and transparency to new regulations emerging on both sides of the Atlantic, governance has become the defining question for the next phase of AI adoption. To help us unpack what AI governance in practice really means, we're joined by Dave Trier, vice president of product at ModelOp, the leading AI governance platform for enterprise.

With more than two decades of experience in analytics and risk technology, Dave helps firms and regulators bridge innovation and accountability. At ModelOp, he guides some of the world's largest financial institutions in building frameworks that make AI explainable, compliant, and scalable. So first and foremost, Dave, thank you so much for joining us today.

Dave Trier: Yeah. Pleasure. Thanks for having me.

Jim Jockel: So let's jump in. Where does the financial industry really stand on AI governance today? Are we still experimenting or is oversight becoming part of daily operations?

Dave Trier: So just a little bit of background as you well know in the financial industry, they've had model management, model risk management for many, many, many years, right? Many decades, and that's just been a part of their daily daily business. Now AI governance is a little bit different lens on that, if you will. So I would say it's not.

It's beyond experimenting, but it's not quite part of daily operations yet. It's getting closer, so it's somewhere in between those two. Again, they're trying to take some of the principles and foundations that they had in model risk management and extending those to build account for and accommodate AI as well.

Jim Jockel: So the overarching, rule for model risk governance is, you know, Fed, SR 11-7. Is that the overarching rule for AI governance, or is it, kind of built off of that?

Dave Trier: Yeah. So that's a great point. So those are definitely used as a foundation because that's obviously what financial industries by, you know, law are required to adhere to. So that has always been the foundation.

And like I said, they look to accommodate and really extend what they've done with SR 11-7 and their MRM program to account for the nuances of AI overall. Yes. I would love to see a little bit more rigor from the different regulatory, federal, et cetera to have a SR 11-7 for AI. Right?

But that again, they've they've taken the principles there, and financials are obviously very, very smart and risk averse. And so they're making sure that they're carrying us forward to AI as well.

Jim Jockel: So how do you define the term AI governance? And what makes it different from that traditional model monitoring?

Dave Trier: Yeah. So obviously there's a ton of different definitions for AI governance, but for me it's really it's the policies and the procedures for overseeing the safe and responsible use of AI. In short, the way I like to put it in layman's term, are you using AI to do the right thing for the company, for your customers, for your employees, and for the community? Very simply, right?

Just are you doing the right thing? But over as you can imagine, that there's a number of different facets to AI governance, and I'm sure we'll talk a bit more about those. Many of them, again, extend from the basic MRM principles around having the right policy and procedure placed, the right inventory risk tiering, effective challenge, I. E.

Validation, ongoing reviews, attestations, et cetera.

Jim Jockel: Financial institutions don't model risk very, very well. You know, the SR 11-7 came out in 2011. It's it's it's well defined. I think, probably, all the consultant firms made a lot of money, you know, as, all these firms are implementing, their governance oversight.

But AI brings a lot of new challenges, explainability, data drift, even to some extent ethics. How are the risk teams adapting to these core differences?

Dave Trier: Yeah. No. As you can appreciate, the risk teams are just starting by looking at and analyzing what are the new risks, As you pointed out a few of them, what are the new risks that AI introduces? It's not like the traditional regression models out there that you build it once and they kind of don't need to change really.

You might make a few tweaks, but they really don't need to change very often to. Obviously AI is very centered around data is constantly changing. The technology is changing quickly and then really the output is, you know, something that is constantly evolving based on the data itself. So the first thing that risk teams are doing is just saying, okay, what is different about AI and therefore what are the different risks, the new risks that AI are bringing to the table?

And it's things as you just mentioned around transparency. Transparency and explainability of the model itself. But it's also transparency into if you think especially around generative AI, what data is being used itself, right? Are you using data that is potentially copyrighted, right?

Or are you potentially using a vendors? Because financials use a lot of vendor based models and vendor based AI systems. So is do you require the you know the vendor to send data out to the vendor and are they using that? Are they then incorporating that into their model?

So there's that kind of transparency into what's happening with your data overall. Then as you've rightfully pointed out, there's things such as Drift, there's in the GenAI world, there's hallucinations, but also the security comes in much more focus, especially as you get into more agentic AI. Because when you get into the agentic AI, this is where you're using things like MCP tools and the like, which potentially are opening up to external parties as you're using and talking to different systems via these agents that are outside of your four walls. So again, back to your original question, it just comes back to they aren't analyzing.

Okay, what are the new risks that are being introduced? Okay, great. How do we then bake that into an update our policy and procedures to be able to account for those? And then ultimately, we protect ourselves by identifying the risks, putting the right controls in place, and making sure that all of the different teams and parties are actually following those policies, procedures, and have the right controls in place.

Jim Jockel: Would you say at this point all the risk factors have been defined, or as the technology is evolving, are new risks emerging, or is it just same flavors of past risk?

Dave Trier: I wish it was just, you know, same flavors of past risk because then it'd be a lot easier. Right? But, no, it's it's as the technology evolves, there's new risks. As you think about, you know, with generative AI, you never even heard of hallucination, right, from an MRM perspective.

And with agentic AI, now you're talking about, well, how do you how do you the right level of control around, you know, the protecting the in line execution for prompt injection and this incidental more. Right? So, unfortunately, the risks are evolving because the technology and the innovation around that technology has evolved very, very quickly.

Jim Jockel: So regulations are moving very, very fast, but arguably, it's very uneven. You know? For example, you have the EU AI Act and as well as state level US rules. How can global firms stay consistent when the rules aren't?

Dave Trier: Yeah. That this comes back to the fundamentals, Jim, as I mentioned. So even back to SR 11-7, which was, you know, couple a decade old now. Right?

But they had the most of the fundamentals in place. Do you have the policy and procedure? Do you have the right accountability structure across the different teams and multiple lines of defense? Do you have the capabilities such as the inventory, the risk tiering, effective challenge, reviews, et cetera?

So I would just say to your question that these regulations use those fundamentals. They use those principles, and then they put some slight nuances around it. Right? The EU AI Act specifically went into, you know, looking at the developers and the users of it.

It looked at how do we have an understanding of they call it unacceptable risk, high risk, which again is just kind of the risk tiering part that was part of SR 11-7. So what I would say is that the fundamentals are very similar to what we had before. There's just nuances that unfortunately, yeah, each global firm has to kind of keep abreast of what those new regulatory frameworks are. Are there nuances that aren't already accounted for?

Okay, let's make sure we have those in place and just unfortunately roll with it. But that's the reason why, you know, you work with a company like ModelOp that just has those as part of their day to day business and say, okay, great. ModelOp's got me covered. Right?

You can help make sure that I'm staying on top of those. And then more importantly, enforcing those across the organization.

Jim Jockel: In data privacy world, you could argue GDPR is kind of a kind of a foundational, piece of legislation that's out there where others are kind of copying it or different, different components of it. While there's none in the United States, California has its data privacy rules, Japan, Germany, in terms of, where data can leave jurisdictionally. Are any of these regulations like the EU AI Act something of, kind of a foundation, that kind of spans across all of these, types of regulations, or are they or are they mostly vastly different?

Dave Trier: No. They're they're all they're the EU AI Act does have the foundation for sure. Right? It has some of the foundational components.

And if you look at regulatory frameworks like the Colorado AI act, as I call it, right, It has some of the same principles around it. You look at what the California attorney general put out, you know, actually probably a couple of years back now, like a Texas House Bill 2060. They all have the similar principles. So yes, I believe that the to your question, the EU AI Act did lay down some of those principles and foundations.

Unfortunately, there's still the nuances that come out that there's very specific things that each, not even just regulatory bodies looking at, it's even in the industry. Like if you look at the NAIC, the insurance side of the AI bulletin that they put out, right? They're looking for some very specific items. You then jump over to Canada with guideline E-23 and the AI extensions that they have and the slight tweaks to it.

But the foundations of principles, as I said, are pretty similar across.

Jim Jockel: So I when I look at my own life and my own organization, you know, I think back to eighteen months ago where, like, my team was using one AI based solution. You know, now we're using fifteen, and every department's got, you know, hundreds and hundreds of different flavors of AI type, solutions going on. You know, once a firm scales from that handful of AI models to potentially hundreds, you know, what breaks first?

Dave Trier: Yeah. So it's one of two things. I'll answer it from the financials first because most financials are, as you can imagine, very, very risk averse. So what breaks is actually the time to market.

Right? You have each of these different teams that want to use AI and because of the risk averse nature as they should be, they slow that down. They slow it down because there's a lot of manual reviews, manual approvals, ad hoc, back and forth, and it just slows down. So the risk piece doesn't break because they're risk averse.

What breaks is the time to market. Now you go on the other side, well, probably outside the financials where they're not as, you know, they don't have ingrained that MRM type mentality if you will. And so what breaks is that something goes wrong. Just, you know, whether they have, you know, IP leakage or you probably saw in the headlines where somebody downloaded a MCP tool and that caused them to hack all of the emails.

Right? So something goes wrong, whether it's IP leakage or a security breach or, you know, you have just a hallucination that causes some sort of just negative feedback to customers or some sort of, regular sorry, brand exposure type event. So you the kind of two legs of the stool there. The financials, typically what breaks is the time to market because they're just going to do everything manual.

They're okay with slowing things down because they want to make sure they're doing the right thing versus I would say the non traditional traditionally regulated industries. They might have things just break.

Jim Jockel: One could argue, while generative AI has been groundbreaking, game changing, in the ways we work and come to market, et cetera, agentic AI is a whole another thing. And the questions that are getting raised now are really around control and accountability. How are companies drawing the line between autonomy and oversight?

Dave Trier: Yeah. So I think there's obviously the huge buzz around autonomous agents, and they're just going to do everything and act on their own. They can reason and just you give them a problem and they just go and solve it. Right?

What I'm finding, though, especially in financials, and I would just say the Global 2000 and Fortune 500 is that it's nowhere near that. They're not they're not going down the autonomous route yet. They're going down what I call more the guided route. The guided of saying, okay, well, here's the specific use case business problem that I'm trying to solve and we're going to use an agentic solution, but we're going to guide that.

We're going to keep it within the guard, the guardrails or the fence posts if you will. So it doesn't give full autonomous, but rather here is the path. Sometimes they use workflows themselves to say, okay, do this and then this. Yes.

You can take some, you know, some decisions as part of that process, but this is the path that we're going to follow, follow if you will. So in that way, again, just where it is today, Fortune 500 and Global 2000, it's more of that guided as opposed to full autonomous. Now as part of that, naturally, you can get some more oversight because it's guided, right? You can have some oversight into what's happening.

The other way that they're dealing with oversight is just a lot of transparency, a lot of making sure that they're capturing the audit trail around it, both the audit trail around the process that we go through the right steps, that we get all the approvals, that we get all the reviews and tests, et cetera, as well as the audit trail around the actual usage, right? So as that agentic system is making decisions, log what those are, making sure we know and have that if there are questions later, we can come back from an audit perspective and understand when it made a decision, what was that decision, et cetera. So just a couple of areas that I've seen in practice a little bit more of the oversight.

Jim Jockel: Well, there is also a perception, and unfortunately, that perception is probably also rooted in experience, but that more governance means less innovation. So what separates firms that manage to move fast and stay compliant?

Dave Trier: Yeah. It's it's it's really simple. It's it's around having an automated process, right? Where those that have the perception that governance slows things down, it means they have a manual and disjointed process.

What I found in just in practice and having done this over six years within ModelOp is that governance can actually be an enabler of innovation when done correctly. It can speed the time to market AI solutions when When it slows things down, as I said, is that the process is not defined. You have ten different teams that need to be involved and they're pulled in ad hoc and there's not a consistent and streamlined effective way to do that. And so yeah, it slows things down to a halt.

What separates them to your question, what separates though the winners in this space are those that took the time and said like we do with our model of customers and said, hey, here's the process. Let's get legal risk compliance data, IT security, et cetera. The business team to the table. Let's agree on the processes that can go and help to shepherd from idea through usage and retirement.

Let's get those processes defined and then let's use something like a ModelOp to go and enforce those policy using automation. In that way, you have built in inherent trust because everybody agreed to this is the process that will make sure that everything is trustworthy, reliable, accountable, auditable. They agree to it upfront. That way, if you get a new AI system through the door, it goes through the process that, yep, we're good because it's the process we all agreed upon.

It'll pull us in at the right time. It'll make sure that we're doing all the right steps in the process. It'll make sure it's auditable. So using that becomes streamlined first and foremost, but then you use automation.

Use automation to make sure that you are can automate many of those steps, But when you need a human in the loop, pulls them in at the right place at the right time, give them the right information, make their decision and away you go. So we routinely see, as I said, when done right, governance is an enabler. We have customers that use ModelOp that reduce their time to market from six months down to a couple of weeks because we've got that automated streamline process. So for me, that's what separates the winners in the space from those that are going to be laggards and fall behind.

Jim Jockel: Well, the best part is if you don't have a defined process, just open up a Copilot and it could do it for you.

Dave Trier: Yeah. There you go.

Jim Jockel: So, you know, coming back to regulators, and we all love regulators, but the main thing they want is proof and not promises. You know, what new tools or practices are helping firms actually show compliance?

Dave Trier: So, I mean, selfishly, I'll just start with what we do in ModelOp. Our tool is about enforcing the process and enforcing the policy. The first thing regulators want to see, hey. Do you have a process?

Second thing, want to see, show me evidence that you're following the process. And that's what we do at ModelOp. We take what process has been defined as we just talked about, and then we enforce it. But we have the audit trail behind it.

So that gives the regulators really the assurance and understanding that the process is being followed first and foremost. If they have any questions about it, they have the audit trail behind it. So that's just an example around following the process and policy with ModelOp. But of course, there's other pieces of it.

There's the security side. There's some of the data access and policies for which, of course, there's existing security and data, privacy and those types of tools as well that are a part of the big picture.

Jim Jockel: So as AI spreads across trading, risk, compliance, and as well as client interactions, what kind of relationship should regulators and innovators really aim for?

Dave Trier: Yeah. It's just all about trust and transparency. Right? You know, as you rightly pointed out, the regulators want proof.

Right? They're not trying to get in the way. They just want to make sure that the financials are doing the right thing. That they're following the process that's been laid out.

They're following the regulatory guidance, if you will. So I would say that is a relationship built on trust but verified. Meaning, you know, hey, I want to, at time to time, you gotta give me some of the evidence that you're following the process that you're that you if I have specific questions to go and test that you're following the process that you can answer that. So I think it's one around, as I said before, the transparency and the trust that is being that these financials are following.

Jim Jockel: So everybody talks about trust, but what distinguishes firms that just comply from those that lead with credibility?

Dave Trier: Yeah. I think it just starts with I mean, everybody, every large enterprise, Global 2000, and even small mom and pop want to say that we're an AI powered business. Right? And we are we are differentiated.

We're the leading edge, you know, enterprise or financial, if you will, because we're using AI. We're changing the game with AI. But what really happens is that if there's ever a case of something that went wrong, whether it's something minor that happened with the customer or something major like a security breach, you instantly lose all trust with the with your customers, the consumers and really the market at large. So what I did to your question, what distinguishes firms is those that are more forward around here are the measures I'm taking to make you trust that I'm doing the right thing with AI.

Right? Here is what our policy is, whether they call it responsible AI or AI governance. But here's the policy that we're following. Here is the option for you as a customer and consumer to opt out when appropriate.

But just making it very clear and transparent to, again, customers, consumers, and market that we know what they're doing with AI. It's very clear what they're doing. And I do have some options around it. Like, if I don't want to interact with it, they let me do that.

So that's how you build that, you know, level of trust, if you will.

Jim Jockel: Well, there's definitely a couple companies where, I'd I'd like to get a human on the phone every once in a while and not have to go through a very long automated AI messages, to get there. But, unfortunately, Dave, we've made it to the last question of the podcast. We call it the trend drop. It's like a desert island question.

And if you could only track one emerging trend in AI governance of over the next few years, or it's AI over the next few weeks because everything's changing, what would it be and why?

Dave Trier: Yeah. It would it would definitely be around the autonomous portion of agentic. Right? That's an area where huge amounts of potential opportunity, but also really high potential for catastrophic events.

So that's an that's an area where, again, it's all the buzz. Everybody talks about, you know, Terminator and all that. But at the end of the day, there is a area for that to be used, but it shouldn't be used all the time. So it's it's the autonomous agentic space that I'm going to be watching very, very closely.

Jim Jockel: We have been and we'll continue to watch. Dave, I want to thank you so much for your time, your insights, and, yeah, give us a lot to think about.

Dave Trier: Yeah. Thanks so much for having me, Jim.

Jim Jockel: Thanks so much for listening to today's episode. And if you're enjoying trading tomorrow, navigating trends in capital markets, be sure to like, subscribe, and share, and we'll see you next time.