Why AI ROI Becomes Guesswork Once Systems Scale

Narrator: Welcome to the Hospital Finance Podcast, your go to source for information and insights that can help you stay ahead of the challenges impacting health care finance. And now, the host of the Hospital Finance Podcast, Kelly Wisness.

Kelly Wisness: Hi. This is Kelly Wisness. Welcome back to the award winning Hospital Finance Podcast. We're pleased to welcome Dave Trier.

Dave serves as CEO of ModelOp, where he leads the company with a clear focus on customer value, product innovation, and enterprise execution. A builder at heart, Dave brings deep technical fluency and real world operating experience to the challenge of helping global enterprises unlock the transformational power of AI. With more than 20 years of experience spanning data science, AI, analytics, cloud, and enterprise software, and as a named inventor on multiple patents, Dave is known for his pragmatic, transparent leadership style. Prior to becoming CEO, Dave served as ModelOp's SVP of product and was foundational in shaping product vision and strategy, working directly with customers to address the operational, delivery, and governance realities of enterprise AI.

Before joining ModelOp, Dave held senior technology and business leadership roles across software, consulting, and industry. He served as vice president of advanced analytics services at Think Big Analytics acquired by Teradata, where he led a 400-person organization across the Americas. Dave holds a Bachelor of Science in Electrical Engineering from the University of Notre Dame. In this episode, we're discussing why AI ROI becomes guesswork once systems scale.

Welcome, and thank you for joining us, Dave.

Dave Trier: Thanks for having me, Kelly.

Kelly Wisness: Alright. Well, let's go ahead and jump in. So hospital systems have substantially invested into AI over the past few years, yet many CFOs still struggle to see measurable returns. Why is there such a gap between AI investment and AI value?

Dave Trier: Yeah. That's a great question, Kelly. And I often get it quite extensively, and there's actually a couple reasons. I guess just first to ground everybody that AI is obviously a new technology, and there is just naturally for large health care systems a bit of a lag between when a new technology is introduced and when you start to see the actual value.

And in particular, this one is very nuanced in that the technology itself does have intertwined with it change management. So I just start with that grounding that AI is a new technology, and there's just a natural lag. However, most of the time, what I see with large health care organizations is that they jump right into the technology for technology sakes, which which is not a bad thing. Right?

It's it's it's very, engaging. It has offers a lot of opportunity, but they need to step back and really look into a thoughtful analysis of what are the top use cases, what are the top business scenarios, the top processes, areas where we can drive the most value and focus on those first. And then from there, it's also about, well, how do I then put the right discipline in place to go from I've got this great idea all the way through testing it, testing with different users, whether it's clinical or back office, and making sure that you have the right approach that works with the existing processes, the existing users, the existing patients potentially as well in order to actually make the most value out of that particular use case. So again, it really comes down to those couple of fundamental principles, if you will, that we see just time and time again.

Kelly Wisness: Yeah. That makes a lot of sense, and I love what you said about focusing on the top use cases. That really resonates with me. You know, so if a hospital CFO asks you how to determine whether their AI program is actually creating value, what metrics should they track beyond the typical number of models deployed?

Dave Trier: Yeah. That's a great one. And it happens a lot because especially with AI and generative AI, that is not exactly a financial metrics tied to it. So I give some guidance to CFOs and other financial leaders across different organizations, health organizations.

Number one, it starts with just usage and making sure that your usage is matching the cost. Right? Because there's a tendency with new technologies, oh, let's just go buy it. It's a new shiny object, But are is there actual usage behind it?

So that's kind of the fundamental. You can apply that to any particular AI technology solution, et cetera. Number two, then if you think about these layers of an onion, if you will, number two comes in, what's the feedback? What's the feedback you're getting from customers, from different users?

Maybe it's an agentic system, and you want to have the overseers of these agents to get feedback. Is it meeting the needs? Did it solve the problem? Did it do its job correctly the first time, or did we have to intervene, in which case, maybe that makes sense to use AI in the first place.

So that's number two, is getting that feedback and having that as a constant, continuous, you know, barometer, if you will, as to whether or not that we are starting to get value. But then number three and the most ideal is there direct financial correlation? Right? And I know there's a lot of buzz around AI, around headcount reduction, which is fair in certain parts, but there are other direct financial metrics, reducing time to market for your product, drug discovery, sales, automating back office tasks that help to save, you know, potentially time, and maybe you outsource some processes in the past, you might be able to bring those in house, if you will.

So there are direct financial correlations around AI that, again, you can start to see and appreciate those benefits over time. So, again, back to your original question, Kelly, it's really the that kind of three pronged approach of usage, feedback, and then ideally direct financial correlation, which unfortunately is not in every situation.

Kelly Wisness: Yeah. No. Those metrics make a lot of sense. So, Dave, many organizations treat AI as a collection of experiments rather than a managed portfolio of investments.

How should CFOs think about managing AI as a portfolio?

Dave Trier: Yeah. It's another great question. AI really cannot be thought of as ex experiments. There's there's just too much on the line.

There's too much investment, too much risk around it in certain areas. So they really need to be thought about and managed with the same discipline that you would manage a portfolio of investments. Yes. Some will fail.

Some will succeed. But you always approach them with that, like I said, that disciplined process to determine what's the business benefit, what's the risk involved, what's the complexity to manage it, and what's our ability to deliver, especially from that change management aspect. So if you think about before you embark on any sort of AI investment, that you'd go and approach it just like you would approach, you know, purchasing your portfolio of stocks, right, that discipline around business versus the cost, and in this case, the risk trade offs around the usage of AI.

Kelly Wisness: Now I've really never thought about managing AI as a portfolio, so that's very interesting. So let's talk about why you call industrializing AI delivery. What does that actually mean from an operational and financial perspective?

Dave Trier: Yeah. That's a great question. And what we see routinely we've been doing this for over seven years working with, obviously, large health care organizations, but also in other regulated industries as well. And routinely, what we've been seeing over the past seven plus years is that when we come into a large organization, they have variety of different teams that are developing or buying AI solutions, and each of them are doing it differently.

They have manual processes, a lot of ad hoc, a lot of back and forth emails, spreadsheets, and it's just a repeat of the wheel, a lot of duplicative costs, and, unfortunately, a lot of an, unknown risk that aren't being managed, unfortunately. So when we talk about this and our, really, our unique approach and differentiated approach around this is to turn that cottage industry, those artisanal type approach of one off crafting of an AI solution into something that is repeatable, consistent, regulated, traceable, and ultimately automated so that you produce this industrial grade approach to how you go from I've got an idea for using AI to actually using it in the business. So, again, it's it's just turning that, like I said, manual ad hoc approach into something that is consistent, repeatable, and, of course, traceable, especially in the regulated industries.

Kelly Wisness: Yeah. No. I never really thought about it that way, but thanks for explaining that for us. So how can hospital system CFOs ensure AI risk oversight is actually happening, not just documented in governance committees?

Dave Trier: I see this time and time again that when you see the word governance as it relates to AI, a lot of spreadsheets come to mind.

Kelly Wisness: Right. Yeah.

Dave Trier: A lot of a lot of checklists. I mean, if you want to go back twenty years, you got clipboards. Right? Unfortunately, that's governance gets a bad rap because people think, oh, man.

It's just another checklist that I have to do, if you will. But it's there, and it's very important to have this in place because AI is inherently risk bearing. It's very tied into data, and especially in GenAI, it can start to hallucinate when starting to produce what look like really, really good answers but are just flat out wrong. Right?

So governance is imperative to be able to do this effectively and safely across, especially, health care organizations. Now the keyword that I would say in response to your question is enforcement, is how do you enforce those governance processes consistently and proactively? So it's not just a after the fact clipboard and checklist, but rather it's just ingrained. It's just a part of the process.

It's just part of going from, like I said, AI idea, if you're developing it, part of your development process, if you're buying it as part of the procurement and testing process. It's just ingrained and enforced as they go through what we call the life cycle of an AI solution from idea through the usage and eventual retirement. So I would just to answer your question succinctly, I would just say to CFOs, make sure that you have an enforcement policy around it, that you're able to trace it and track it and have visibility into what's happening across the entire organization regardless of what type of AI is being used.

Kelly Wisness: Yeah. That enforcement policy, that makes a ton of sense to me. So, Dave, we're entering a new phase with generative AI and agentic systems. How does that change the operational complexity for hospital systems?

Dave Trier: With especially a generative AI and agentic system, there's really a variety of different areas that you have to now think about. Number one, it starts with just the variety of different technologies themselves. Right? So in the past, if you think about how people used AI and ML, there was kind of some some typical approaches that are used, and there may have been some open source and some proprietary technologies.

But now there's such a variety of new AI technologies, both development wise, and there are just, it seems like, hundreds of companies that start every week related to AI. So you just have this massive variety that you're trying to manage. And then you couple that with the pace of change of the technology itself. So not only is you got this wide variety of different AI technologies, but then they themselves are constantly changing.

So how do you how do you keep pace with it? And that's where it introduces, like I said, operational complexity that you have to manage. And then finally, if that wasn't enough, you got a combination of it's not just the, quote, AI models themselves. They're actually integrated into different applications.

They're affecting or a part of existing processes, like I said, whether they're clinical or back office or supply chain. And lastly, they have humans involved as they should. Right? And so you gotta it's not no longer just a I've got a model that give it some data and it predicts an output, but rather you're entwining these, like I said, within existing processes that are involving different teams and users, again, all adding up to the operational complexity.

But don't worry. Right? That's why companies like ModelOp exist, right, to help you to overcome some of those complexities, to help you to have consistent approach to how you go and make sure that the AI is going to be working as you would expect, that it's not going to drive undue or unbearable risks, that you have oversight and visibility into what's happening overall. So hopefully that gives you a little bit context on that one.

Kelly Wisness: Yeah. I mean, I know things are changing rapidly. So, you know, if we look ahead three to five years, what will separate hospital systems that successfully scaled AI from those that did not?

Dave Trier: Again, we'll just start from more of the business value side of things. You'll see that there is better patient care, right, because they're able to use AI to help to inform not only, the clinicians and the staff, but also for consumers, patients themselves as well. Second, you'll help to really take a lot of load off the plates of especially the clinicians and those, that work within the hospital organizations because it's just helping to run some of the routine tasks, some of the menial tasks, ones that, you know, you kind of don't want to do. Right?

So you'll see that there's a lot of time freed up for patient care as opposed to doing some of the back office work that nobody frankly wants to do. So I wanted to start in that front. But for those as you to your question, say, what will separate them is that they're driving those type of outcomes day in, day out. It's just part of what they're doing with AI ingrained into that process.

So they'll just, like I said, take that transformational step to start to look at what are those high value business processes or areas that where AI can help to drive some of those types of improvements that I mentioned. And they'll start to implement those to the point where it's ingrained in how they do day to day operations overall. Who knows what's going to be ahead in three to five years? But I know things are rapidly changing.

They're going to keep changing. Right?

Kelly Wisness: Absolutely. Yeah. So if a hospital CFO listening today could do one thing this year to improve the ROI of their organization's AI investments, what should it be?

Dave Trier: It's very simply, it's it's helped to introduce some of those standards that provide that discipline we talked about upfront. Discipline to have the process industrialized, refined, automated, that first checks upfront, hey. What's that business value versus risk as we mentioned? And then has that industrialized process to do the checkpoints all along the way.

It's during development. If they're developing it or buying it, they're testing it. Are we seeing the value that we want to see? Great.

Let's let it proceed. Then you go into a pilot phase. Awesome. We're again seeing the value.

And then we move on to production, and that's great. You start to use it, and you might get some value or it might change over time. And to the point where, oh, our original thesis around this area, it's actually not returning the value we'd expect. So you now can start to, like I said, rationalize that portfolio of investments and continue the ones that are driving the value, but eliminate the ones that potentially are not.

So it's all about establishing those enterprise discipline and standards and helping to move from that one off ad hoc approach to something that's more consistent industrialized, as we like to say.

Kelly Wisness: Right. Well, thank you, Dave, for sharing your insights with us on why AI ROI becomes guesswork once systems scale. And if a listener wants to learn more or contact you to discuss this topic further, how best can they do that?

Dave Trier: Absolutely. At any time, please go to ModelOp dot com, m o d e l o p dot com, or reach out to me on LinkedIn. Always happy to have a discussion.

Kelly Wisness: Awesome. Thank you for providing that, and thank you all for joining us for this episode of the Hospital Finance Podcast. Until next time.

Narrator: This concludes our episode of the Hospital Finance Podcast. For show notes and additional resources, visit us online. The Hospital Finance Podcast is a production of Besler. Built on partnership, driven by success.