AI Governance: ModelOp's Approach to Enterprise Trust

Evan Kirstel: Hey everybody, fascinating chat today on AI governance with a true innovator in the space at ModelOp. Jim, how are you?

Jim Olsen: Ah, doing good. Doing good today. How are you doing?

Evan Kirstel: I'm doing great. Thanks so much for joining from your off-the-grid location. I see the solar in the background. We got Starlink going.

Really intriguing. But before all that fun stuff, maybe introduce yourself and what's the big idea behind ModelOp. Sure.

Jim Olsen: Uh yeah, so I'm Jim Olsen. I'm uh the chief technology officer of ModelOp and actually did the architecture and design of the original system. Uh so I know a lot about the space and myself and some of my colleagues have actually been working in the space with uh previous previous efforts uh for a long time 10 plus years etc and that kind of things. So have a lot of knowledge about not only the newer generative AI but also traditional AI traditional statistical techniques etc and how they affect your business.

Uh and so that's how we created the ModelOp solution to actually bring in the uh full life cycle management of all kinds of models everything from an Excel spreadsheet to uh an LLM foundational model and now Agentic AI solutions as well. Um so we put that together to make that process a lot easier because we found a lot of companies are struggling getting their um solutions uh using these technologies out to production.

Evan Kirstel: fantastic mission and you released a governance benchmark report on AI recently ideal for this audience. So let's kind of start with the big picture. What was the big idea the motivation behind this benchmark study and what did it tell us?

Jim Olsen: Well, a lot of it was understanding really where companies are at with their uh AI solutions. um you know there's a lot of dispar information and articles and I mean if you listen to the Silicon Valley digital native companies you know everybody's using it for absolutely everything and you know it's it's it's the next biggest thing then you talk to some of the enterprises and they're more hesitant about like how does this impact my business and you know what am I willing to put into place and etc. So there wasn't a lot of great clarity into what the plans for an enterprise business uh actually are uh in a variety of different spaces. And you know what we found is uh a lot of companies are struggling to build trust within their organizations about these solutions because they don't have the insight.

you know, we're seeing a lot of uh IT departments pushing back because they're finding shadow AI where you know, we've seen things where we've heard like uh where hospitals people were posting customer data into ChatGPT-4 to get summarizations going around the it obviously that's a huge risk. I it's breaking several laws and that kind of things. So how do they get that's uh these processes in place? And so that's where we saw that a lot of companies are just struggling with those concepts as a whole.

Um and in the report you can see a lot of our findings where it's a lot of people playing with it. Uh they're not getting the solutions out there quickly. So they're losing on business value, but understandably they also want to make sure they have trust in these solutions. Well done.

Evan Kirstel: And one of the headline stats from the report, 56% of GenAI projects take 6 to 18 months to reach production. So how do we get out of this kind of quagmire?

Jim Olsen: Well, that's that's the found that's the foundation of why we built the company. Um much like today, uh you know, think of software in the 90s. Um people would actually go and just develop it on their desk, throw it out into production. you know there wasn't really processes and because of that things broke um and at least programming is deterministic in nature so you were able to put uh processes in place now what company would not have some kind of a CI/CD pipeline nowadays you know common practice stuff but back then those didn't exist what we found is those same kind of processes for enabling um basically more efficient uh deployment of these solutions and understanding and insights into those solutions and reproducibility uh didn't exist for the basically the model world for AI models, vendor models, foundational models, etc.

So that's what we created was a process where you can actually automate a lot of this to make it easier because software is very deterministic in nature. Uh AI models etc are the exact opposite of that. So what can you put in place to provide those insights and build that trust within your organization so you don't hit all of these red blocks? And we found when we deploy our solution into customers uh you know they were easily cutting that time in half if not even more so uh depending on uh how sophisticated the company was to start with and creating just a formalized uh repository where people can find out who's using this stuff for what use cases and uh you know what is already approved out there could I leverage this etc and that kind of thing.

So having that centralized inventory and then an automated life cycle process to drive the software out to production. Well done.

Evan Kirstel: And you surveyed a number of sectors, financial services, pharma, manufacturing. Did any particular vertical stands out in terms of maturity or challenge with AI governance?

Jim Olsen: Well, I mean, oddly enough, the ones that have the most challenge with uh AI governance are the ones that are the most mature because they're forced into it. So uh you know some of our very first customers were obviously financial customers heavily regulated. You know you can't have models uh making trades or predictions or whether somebody gets a loan or not that isn't well scrutinized and understood. So that was obviously the first place that had the most challenges because they could be audited constantly um and had to have everything documented because you know uh I don't remember which bank it was but it was like a multi-billion dollar fine for not doing this properly.

Um and so you know there's a lot at stake. So obviously for them that created the uh the necessity is the mother of invention. So you know you'd see a lot of homegrown processes within there that ar weren't always so effective because they weren't stepping back from their own business to develop them. And that's by having our own solution that is more neutral and takes all of the ideas into into concerns uh create creates a more efficient solution.

We have se several um firms in the financial sector. What we're starting to see though now is obviously AI is coming into the healthcare industry and we're literally talking life-or-death decisions um when we get into healthcare. So I see that as a space with even more challenges because they weren't born and bred in the statistical nature of a financial institution where you know these things are well laid out well regulated. There's patchwork legislation across different states as to what a uh AI can be used for within healthcare.

And of course, it's just the very real concerns. Nobody wants to have one of these models blow up and be a stain on their reputation. Like uh there was an example in healthcare where long-term care was more uh being not recommended for minority groups than non- minority groups and actually resulted in a lawsuit. Um so you know there's really re real world situations here that come up when you bring these solutions into life or death situations.

Um so you know we've seen definitely a lot of interest there as well.

Evan Kirstel: Yeah we all saw what um the challengers were with IBM Watson many years ago trying to an early stab into the healthc care space with a lot of challenges. is I think we've matured a lot since then but still a lot of work to do and you mentioned the report shows 50 plus generative AI use cases in many cases but only a handful make it into production what's that disconnect why the drop off

Jim Olsen: well I mean there is a natural uh to be fair uh generically there's a natural drop off you know everyone's got great ideas and then bring them to production there absolutely is always a revision on that but what's uh even more driving that now is this is this lack of trust. You know people are more skeptical of these solutions because they are non-deterministic in nature. You know if you have a model that predicts uh uh whether a cell has is cancerous or not that's fairly readily verifiable and uh you know testable to a degree. You know you have known labeled use case case data and things like that.

when we start to get into more like even something as simple as summarize this patient record into a uh into into a recommendation or this prospectus from a company into a summary that I can use to make quick decisions about whether we should be investing or not or these kinds of things that's not deterministic in nature. So people are naturally skeptical because they can't look at it and say for sure it sounds good but is it right? I know that's the one of the challenges is uh you know especially foundational models they're known for sounding very professional and intelligent etc but not always quite so factual so they're very convincing of giving you the wrong information and convincing you it's right so that creates this trust because it's so much harder to you know one bad uh kind of recommendation or something from uh one of these uh situations is a lot are harder to overcome than a thousand correct ones.

Evan Kirstel: Yeah, people tend to remember uh the where it went wrong.

Jim Olsen: Um so, you know, how do I build that trust that yeah, we are holistically looking at the not only the foundational model itself, but it's applicability to the use case that I'm doing it and what risks and mitigations have I put in place to make sure that this use case is well protected. So tying all of these models, all these resources or now when we're getting to Agentic AI, all the different agents involved, the different tools that they use, etc. into a single pane of glass uh inventory like we provide helps provide that clarity of, oh, well, it's also been used over here. It worked really well over there.

Oh, okay. Now I got a little more, you know, you can start to build that trust and these six people reviewed it and these risks were identified and said, yeah, this will be okay because of this. you know, you need that kind of a the story behind the model getting out there uh to build that trust. But then likewise, you can't have building that story be a manual process on an Excel spreadsheet or a SharePoint file or just a bunch of Jira tickets scattered all over wherever that doesn't give you the story.

So that's where, you know, our software helps actually pull that information together into documents and mitigating risks and uh you know, findings etc. that all uh are in one place and you can actually get that and make sure all those Jira tickets etc etc are tied back and happen. So you know by just automating all that making sure it's there and making it readily available you got to do that whether you build this off yourself or you buy a solution like ours because otherwise yeah the process of trying to do the model life cycle then becomes overwhelming

Evan Kirstel: amazing spreadsheets off for AI governance in what's this 1999 I mean come on we need to up our game a bit and that's for you healthcare with your fax machines and your email.

Jim Olsen: It's it's it's it's like a zombie that just won't die.

Evan Kirstel: Um the other challenge in the enterprise as you know is fragmentation, lots of silos, uh lots of technical debt. Um what does that look like in the real world in terms of impacting AI at scale?

Jim Olsen: Well, I mean obviously uh if you have different people taking entirely different approaches using different technologies etc without any consistency it just does create more of a burden on understanding um not only getting these things deployed I said it's just work to do that but then also in doing the reviews of the technologies etc. And a lot of that just naturally comes out because uh you know these are kind of a lot of grassroots efforts that we're seeing initially. Uh it doesn't tend to be as centralized at a higher level. Um so water finds its own level.

So the individual groups are picking their best of breed tools and their solutions and kind of running at it without the knowledge of the other teams and what they're doing because they can't find them. When you get to very large companies that's just a reality uh cross business unit team etc. collaboration is a challenge. So you do want to have a centralized process understandings and then the ability to automatically generate findings about hey have you thought about this and this because you've we've already seen this be a situation in another organization using these so have you taken this into account or you know who's the responsible people to talk to etc.

So you know that's the challenge without having uh some kind of a centralized understandable and automated process is there's an inconsistency even in the process itself which then becomes frustrating to all these individual teams and you know nobody's you're not standing on the shoulders of giants within your own company. You're instead all trying to forge it on your own and we know that never works out as well. That's not.

Evan Kirstel: So let's talk risks. There's still lots of landmines to uh avoid out there on the regulatory side. Uh lots of compliance risk and fines and other challenges. What do you advise customers to be aware of when it comes to real world exposure?

Yeah.

Jim Olsen: Well, it's not just regulation. Regulations are definitely important because obviously if you're not compliant with a regulation that's pretty cut and dry, you're going to get in trouble. Um, and what degree of trouble you get into is going to also depend on how much process you can show cuz nobody's going to be perfect. If you did nothing, just ignored it and everything.

They're going to be a lot harsher on you than if you tried your best and tried to do everything right, things are still going to go wrong. That happens. They're probably, you know, if it goes wrong just because of a black swan event or something like that, you're probably not going to get get in that much trouble um from a regulation standpoint. Um but what we really talk about is more importantly is also even just your brand.

So it's not even uh not even uh having to do just with a regulatory get a fine you know many many products and especially in the consumer product space and we work with several um on that kind of things. Your whole value is in your perception by the customer. You know buy one toilet paper versus another. Yeah, there's some differences in the things, but that's not maybe your way you make money is by making the better toilet paper.

You make it by making the better having the brand that has the name recognition and you trust the quality. If you put AI solutions out there that have a blunder like, you know, McDonald's put out its automated uh ordering scheme and there's tons of videos posted online of people, yeah, I'll take one fry. Okay, added 11 fries. Oh, no, remove that.

I only wanted one. Okay, we have 12 fries. you know, and it kept going on like that. That was a hit to their brand.

It made them look foolish. Now, is it going to destroy a McDonald's? No, probably not. But those do have impacts.

They have real financial impact that is even harder to measure in the long term that can still, you know, cause you maybe even more problems than than the government find. I bet.

Evan Kirstel: So, you're in a very hot space at the moment. A third of companies evidently are budgeting $1 million or more annually on AI governance software. So congratulations on being in a hot market segment. Maybe talk to us about your space in general.

How where it's headed? Obviously up, but and how do you see yourself competing versus other players out there?

Jim Olsen: Well, one of our biggest spaces is, you know, we're always staying ahead to us now. Just straight like a rag architecture foundational model. That's kind of yesterday's news. Yes, everybody's doing it and that yesterday's news doesn't mean not very relevant to an organization and we still have great focus on there but obviously uh all the buzz right now is around agentic AI and what that means because that even has larger implications you're l literally giving autonomy to these foundational models to make decisions about actually changing data within your database or sending emails or any of these kinds of things.

So that's what we've been working on specifically is how do we uh bring uh Agentic AI solutions into the model life cycle process and we've uh done a bunch of work there. Um we actually have webinars uh on it uh on our website. Uh but you can actually start to manage these and things like MCP tools. Everybody's talking about those now.

The Anthropic's MCP, Model Context Protocol is kind of won the tool war for lack of a better term of how do LLMs communicate? uh to actual things that can affect change or read specific data. Uh so like we've uh incorporated Agentic tools right into our solution. So you can actually use Agentic AI to do model governance um itself and that kind of things.

But more importantly, we also have ways of like how would you approve an MCP tool to use and know which use cases are allowed to use it and uh what filters can you put in place like PII protection. Maybe I know specifically that this particular model may have access to PII data. So I want to block any PII data coming from out from it etc. So we've been building things uh in that space knowing that uh the agentic AI solutions are going to uh literally change the landscape in that way and that as these companies put these in place how do they know what they're doing?

How do they know where they use? How do they know you know protect against uh you know deciding all of a sudden to sell all of its stock or something? Uh truly with autonomy comes greater danger. Got it.

For sure.

Evan Kirstel: Including personal danger getting into these robo taxis now all the time. I'm always scratching my head. How's this going to go? But I digress.

So when you talk to a customer, maybe they're a little skeptical or uncertain as to where to start, how to prioritize this journey. What's your advice to them?

Jim Olsen: Well, what we start what we suggest is we have a thing called minimal viable governance which is kind of like here's the minimum you need to do. If you try to start I mean anything if you try to start out doing it all then you know you're never going to get there. It's just like uh you know coding we use more kind of an iterative approach now as opposed to the waterfall design approach of the past. Same thing with governance is get started.

Um start small, get the things in place that you absolutely need. That's going to change by your business. Maybe you have if you're a financial institution, you need a your minimum level is a little higher than if you're just protecting your brand um on that kind of things and get the processes in place, understand what's there and treat iteratively, continue to uh grow and add. And that's where our solution providing a configurable approach to the model life cycles that doesn't require writing code or changing the product itself really enables you to do that iterative process and even version the process to uh carry forward.

So that way you can evolve and if a new regulation comes out tomorrow you can plug it in uh or you have as I said as your business. But the important thing is don't wait. It's the problem's only going to get worse. get started now because getting any process in place means there's a process and there's things identified and you know what's going on versus kind of burying your head in the sand and just waiting until it bites you because it eventually will and that it's going to get harder to unravel it later when there's a whole bunch of them out there than if you get started when as we see there's only so many in production you have that big backfill sitting behind per this report you want the process in place to help that back fill not only uh make sure it's it's governed and doing the right things but also help identify those and push those out into production so you don't lose a lot of those maybe good efforts that are buried within your company.

Great advice.

Evan Kirstel: Uh so we're halfway through the year. Hardly uh I can hardly believe it, but what are you up to the second half? Any travel uh events um beyond the summer? What's on your radar?

Jim Olsen: Well, we're attending a whole bunch of different things. I'll be honest, I don't know all of them because I don't go to all of them uh on that kind of thing. Like we just recently went to the CHAI conference. uh out in Stanford um and participated in that um talking about specifically AI uh usage within the healthcare industry.

You know, we've got CDAO conferences we've been going to constantly doing webcasts. We do our own webinars. Um I just presented one uh actually last week on Agentic AI and what we're doing there on that kind of things. And you know uh a lot is virtual nowadays still but uh you know we're doing some inerson events as well with the conferences etc that are going on and starting to pick up um on that and uh you know but really we're kind of participating everywhere in a lot of different things.

So uh usually usually you know again this is a kind of more of an iterative space so things come up and you never know where you're going to go next week potentially. Exactly.

Evan Kirstel: Well, I'm uh speaking of virtual, I'm admiring your real background, not virtual background. What's up for the summer in Colorado? Any hiking or fishing or hunting or bird watching? What do you get up to there in the woods?

Jim Olsen: Well, yeah, the wildlife we get to watch right from the deck. So, we get moose and elk and marmots and everything come right up to the deck on that. We actually uh myself personally, we have a lot of we I have 14 acres here and we have a lot of beetle kill. So, I'm always working on cleaning that up, unfortunately.

Yeah. So, I don't need a gym membership. I do it uh by moving trees around and things like that. So, we got that.

But, yeah, we get out uh in into the woods and hike and all kinds of things as well. Uh take our UTVs around, etc., too, and just enjoy nature where we can. Fantastic.

Evan Kirstel: Well, thanks for joining, taking some time away from all that and um congratulations on all the success onwards and upwards.

Jim Olsen: Yeah, absolutely. and thank you for taking the time to talk with me today. I really appreciate it.

Evan Kirstel: And thanks everyone for listening, watching, checking out our new TV show at techimpact.tv now on Bloomberg and Fox Business. All right, take care. Thanks, Jim. Okay.

Jim Olsen: Thank you.