The Medical AI Podcast: Shaking up Healthcare Systems for AI

Dr. Felix Beacher: The Medical AI Podcast. Welcome to the medical AI podcast with me Dr. Felix Beacher. Strap in for a thirty-minute thrill ride of big ideas. Let's imagine a utopian future of medical AI.

Health implants and wearables constantly monitor your health. AI systems process those health data in real time and recommend treatments combining those data with those from your genome. Recommended drugs, naturally, are more efficacious and cheaper than those of 2025. And life expectancy will be a hundred or more, and old age will be longer, healthier, and groovier than ever.

But getting there is another thing. And the problem may not be so much with the technology. Perhaps what is needed is a fundamental shake up of healthcare systems themselves. Joining me to discuss this very important area is Dave Trier.

Dave, a very, very warm welcome to the podcast.

Dave Trier: Thank you so much, Felix. Pleasure to be here.

Dr. Felix Beacher: So, Dave, please introduce yourself to the listeners and explain a little bit about what it is that you do.

Dave Trier: Excellent. Yep. My name is Dave Trier, senior vice president of product at a company called ModelOp. We focus on helping large healthcare and other organizations to scale the use of AI safely, effectively, and rapidly.

Dr. Felix Beacher: Now I've given a fairly fanciful impressionistic overview of what, medical AI might be like in a in a in the in some kind of future. Could you describe from your own point of view, and perhaps a more realistic way what you think the promise of medical AI is?

Dave Trier: Yeah. So I'll first start with that of the patient or the consumer, if you will, and what that looks like. And I don't believe it's very far off from what you mentioned of being able to have real time information, whether it's CGMs or others, as well as combining some of that with your history, your past, and also your your genetics. Right?

And getting an understanding of everything about you from a health perspective, but more importantly, being able to quickly understand, alright. Well, how does this relate to current conditions that I might have as well as current treatments that might be asked of me? Right? So I'm working through current treatment plans.

So how do I decipher some of that Greek, for lack of better terms, that's around different treatments and diseases and other types of scenarios? How do I decipher that into something I can understand, relate it back to my current information to give more tailored, personalized understanding of what exactly I need to do. So for me, from that patient or consumer perspective, it's just insight and understanding of what's happening and a really a very casual and easy to understand view of exactly what I need to do at that point in time. If I turn to that of, say, a provider, for a provider, it's all about helping and assisting with making clinical and other decisions, making sure that we are processing large amounts of data, again, deciphering things such as whether it's radiology, PACS imaging, that sort of thing, as well as notes that it might have been provided from other clinicians to be able to help to inform the decision making, just making that life easier, taking all of the stress and burden off of doing some of the day to day grudge work and so that you can focus on truly a patient centered view.

Then lastly, I'll I'll take it from that of, say, the pharmaceutical or med devices, industry, if you will. In looking at how we can get just groundbreaking new devices or, different types of medicines out into market faster because we have just a wealth of evidence. Being able to combine what we've seen in clinical studies back with, okay, well, here's some of the research of related types of, you know, medicines or other devices in the past. So for them, it's about speeding time to market, but still in a way that is safe and effective, if you will.

So I just wanted to give a couple different views from different stakeholders and personas. Hopefully, that makes sense, Felix.

Dr. Felix Beacher: Yeah. Absolutely. Now, I try not to be too cynical. But, people could people could be forgiven for thinking, well, hang on.

I remember back in 2020, you know, we'd had machine learning systems around for a few years even then, and, nothing has very much has changed as far as going to the doctor and joining a waiting list and Right. You know, nothing has much changed in the last five years. You can forgive people for being a little bit, cynical about the pace of progress of AI systems in health care. Could you comment on that?

Dave Trier: Yeah. Absolutely. So I think if you think about the problems that we're looking at, but just in general, it's just pervasive in the years of working with health care systems. One, you just have some, just general organization and process challenges, right, that most, especially health care provider systems, but pharmaceutical and med devices as well.

They're very large. They're complex. There's a lot of different departments and teams and stakeholders involved. So just overcoming some of the process and organizational challenges is an area that has to be done in order to truly, make leverage make and leverage AI throughout your organization.

Then there's comes as, you know, kind of talked about technology barriers. Right? There are some technology barriers. A lot of those are being ironed out in other industries.

Right? So the medical AI should be able to take advantage of what's actually being worked out in other industries. But the biggest one to me, Felix, is just around the trust and acceptance. So how can we trust that AI is actually making the proper decisions, that it's not hallucinating, right, or it's not providing any sort of misinformation, misguided information.

So that's a that's a major challenge that I would see, again, taking your cynical lens. Right? A major challenge that you have you have to overcome both internally within healthcare organizations as well as that the patient population, consumer population in general.

Dr. Felix Beacher: Okay. Well, why don't we take some of those factors, in turn? So let let's focus a little bit more on the trust and acceptance issue. What's your sense about how that's changing over time in health care?

Dave Trier: Yeah. I think as especially with generative AI and things such as ChatGPT becoming more pervasive with the general population, consumers at large. Right? As you they start to use it more.

They use it in their everyday life. They start to get an understanding of what it can do and what it can't do. So I think just in general, having that broader usage of generative AI is helping it's a step on the way, if you will, to helping them to trust and accept. That's that's not like what it was in the past.

I know you talked about ML and the cynicism around, oh, we tried ML in the past. Right? But it's different. Machine, machine learning and just, I would say, kind of traditional AI, that was really just pockets of teams, data scientists and others that were sitting there in their lab, right, cooking up, hey.

Here's how we could use machine learning, and they're the only ones that really understood it or use it, and it was kind of hidden from the general population. What's different now now is generative AI, it's everywhere. Right? My grandma uses ChatGPT.

Right? So it's it's it's just a different time in which that everybody has been exposed to it and naturally will will help along that trust.

Dr. Felix Beacher: The second thing I would say though as well Let me sorry. Let me just stop you there because Sure. This is a very interesting area that's kind of new because I think, you know, if I think back five years ago, there already For example, radiology systems that you could do controlled studies on, compare performance with radiologists, and you could see an improvement in many, you know, in the machine learning system compared to a radiologist. And

So that's impressive on any level. Right? But then when it comes to LLMs, people are very aware that on the one hand, they can be spectacularly impressive, but on the other hand, they can also be spectacularly stupid. And that kind of level of inconsistency might be a bit worry worrying in the in the context of medicine.

Dave Trier: Yes. For sure. And that's where you see all of the buzz and articles and really pushback around hallucinations. Right?

Of just, hey, this LLM replied back and it sounded really confident. Right, Felix? It sounded like it knew the answer, but by by God, one plus one is not three. So it's, that is an area of concern.

And that's where, yes, my first point was just more around consumers being exposed to the technology. That's that's a little bit different than it was in the past. But my second point would come into it is to help with trust and acceptance is just rigorous testing, rigorous, guardrails, if you will, around how, especially generative AI, is used within medical AI, making sure that there are accepted patterns or processes for what, say, a generative AI system can do, and then just not allowing it to go off the off the rails, no pun intended, but go off the rails for from when it might be starting to hallucinate or others.

So that's the second piece around the trust and acceptance. It's just having a process, having a rigorous testing, validation, and approval capability to ensure that we are using this system purposely for one specific usage, if you will. We thoroughly tested it. Here are the results around it.

We validated that we have all the specific guardrails in place. We've done some negative testing, if you will, to be quite technical, some negative testing of scenarios that we're trying to make it to break it, if you will. So that's where the you help along the journey towards trust and acceptance is that second area of just rigorous testing, validation, and understanding that there's the proper guardrails to go and keep generative AI in line, if you will. So I'll pause there and see if you have any questions on that one.

Dr. Felix Beacher: No. No. I think this is very important. Of course, you know, the trust and extent trust and acceptance is an interesting issue from because it differs with respect to clinicians who presumably know when, machine learning or AI systems are being used and the general public who, most of the time, probably don't even know when AI systems have been used.

So I guess the trust and acceptance is more important within clinicians than the general public. Do you think that's fair?

Dave Trier: I think it's actually both, but certainly that clinicians will obviously, being highly educated will be much more cynical, to use that term again. But they'll be much more cynical because they obviously know the right answer. Right? So the they will want to have very, very deep details, very a deep understanding of how this AI system came to a certain conclusion.

So oftentimes, what we see is that there are link backs. So here, it'll answer the question, and then it'll say, alright. Well, here's where I got my answer from. And typically, you know, for clinicians especially, it would need to be in, obviously, trusted sources, whether they're, journals or other types.

Right? So just giving the clinicians, here's the details of where I got it certainly helps in establishing trust. On the consumer side, though, they also want to have an understanding of when an AI system is used. And this is where that you see some of the regulatory frameworks, whether it's like Texas House Bill 2060 or the California attorney general came out and say that you need to publish here are all the different AI systems that you are using and then also make consumers, patients aware that this is an AI system that is being used behind the scenes.

So in that way, yeah, consumers want to know, Felix. They do want to know this information that, know that an AI system is being used. And many of them are getting a little more astute in terms of okay. I also want to know the facts behind it.

Where did you get this information? Was it of a trusted and reliable source? So I think that same level that, clinicians, it certainly will be a different level that clinicians expect, but consumers do have some expectations about understanding. Yeah.

I do want to know that AI system is being used and that it is actually trustworthy, verifiable.

Dr. Felix Beacher: Okay. So trust and acceptance is a key issue for, commercialization of medical AI. What else would you highlight in, in this area?

Dave Trier: Yeah. It's organization and process. Right? So we work with a lot of health care providers, pharmaceuticals, and other large organizations.

And you'd be surprised how little of the challenges around technical integration, Felix, and how much is around. Alright. We gotta align the organization. We gotta align the different processes.

Because in reality, what it is even for one AI system, just on average, there are about ten different stakeholders that can touch that AI system before it even sees the light of day, right, across the, obviously, the medical team, the potential data science team, IT, security, legal risk, compliance, et cetera. There's over ten different teams that are touching it, and then there are typically anywhere between five and ten different systems that have to be involved in the process. Think about your security, your data, your infrastructure, et cetera. Right?

So there's just the people side. There's the technology side, but then there's a process side. There are typically a myriad of different processes that are involved. So it's just as you can imagine, each one of those are a barrier.

There's they say, okay. Well, I've got this great system. It's shown good results, but then you get stopped. Right?

You hit another you hit another wall because of the, again, process or organizational construct that has been established in these large organizations, over the time. So we see that routinely that what you obviously want to try to get this innovation out the door quickly. Routinely, we see some of it grinding to a halt because of just the process that's involved, throughout the organization.

Dr. Felix Beacher: So could you give some specific examples from your experience of those kind of organizational barriers?

Dave Trier: Sure. Yeah. So I think the first one is that, especially large organizations that are just trying to grapple with using AI, commercializing, so to speak, scaling the use of it, they don't have a repeatable process, believe it or not. They may have documented it in a policy document or maybe some very ex extensive PowerPoint presentation, but they don't have just here's an ironed out streamlined consistent process so that anybody that's within the entire organization, they say, I've got this great idea for AI, for this medical AI, system.

And what do I do? Right? So the first challenge that they come across is they say, I want to go and put this in production and they say, okay. Well, what do I do?

Well, they reach out to their manager. The manager says, okay. Go talk to this person in legal. Legal says, well, this is something where you actually need a data security, data and security review.

Well, then you reach out to them and then they say, okay. Well, you also need to get out to compliance. So first and foremost is just navigating the structure. Right?

Just navigating. Who do I need to talk to? What do I need to do? And that changes whether it's it's something that's used for clinical decisioning versus, say, back office.

Right? And so it's not just a here's the one process. It's well, it's dependent. Are we you developing this internally?

Is it for clinical decision? Are you purchasing it from a vendor? Did you talk to procurement if so? Right?

So there's different pathways, different types of processes, approvals, and other depending on the, really, the target usage, clinical back office, et cetera, as well as, the type of its internal or vendor base, and even just the delivery mechanism. Is this something that's going to be a SaaS based solution or cloud? So you can just imagine that this is a kind of a spiderweb, right, Felix, that you have to navigate and try to figure out on your own. And here you are.

You're just a, you know, part of the, I don't know, radiology team, and you just want help with some of your imaging. Right? So can you imagine trying to navigate that structure if you're just trying to do your day job and but also you really want to use this AI? So that's just what we commonly see is just these teams that want to use AI just get so frustrated with just the spider web of here's the different things I need to do, and I don't even know who I need to talk to.

So that's just, again, a common example we see.

Dr. Felix Beacher: Now the US is a somewhat litigious society as, as people tend to be aware of. Now when it comes to medicine, of course, we are dealing constantly with issues of life and death. So given that, I would imagine that the fear of litigation could be especially pronounced is how much is this a factor in generating a kind of conservatism when it comes to adoption of new technologies like AI?

Dave Trier: Yeah. It's it's high. Right? Especially for the clinical decision making.

Right? So this is where a lot of health organizations, they do start with some of the more back office optimization type use cases. You think about nurse triage, optimization, that sort of thing. So a lot of them do start in the back office just because of that fear of litigation.

Right? But then when they do get into, alright. Well, we've got this, again, AI system that is just going to help us tremendously as when it comes to radiology. Then that's when they need a lot more trust, a lot more evidence, a lot more testing verification, and sometimes even independent third party reviews of it before that they will go and, you know, take something on because of the fear of litigation.

And sometimes even that, you know, as part of it that they may require, especially if they're, buying a vendor system from, you know, this AI system from a vendor, if you will, that they'll require them to take some sort of stance from a liability perspective as well. So that's where legal and procurement, especially for vendor purchased AI, comes into play quite extensively to make sure that the terms and conditions, the liability, right, all of that is ironed out before the AI system is being used. Right? So and then that's where, again, yes, high level of litigation, but it's a it's a real challenge in that back to my previous point around the organizational process challenges is that most large organization, it's a one size fits all.

Here's the process. It doesn't matter if it's a simple back office thing that doesn't ever even touch consumer patient data. It's the same process as it would for something that is a high impact patient facing type AI. And that's a real challenge because it doesn't need to be like that.

Dr. Felix Beacher: Now could you also talk about challenges with IT infrastructure and, health care settings in the US?

Dave Trier: Sure. Sure. In short, it's a challenge. Yeah.

Right? In the US, quite. Yeah. It's a challenge everywhere.

It's not just health care, but it's certainly in health care that you've got the combination of legacy infrastructure. You have your on premise on premises and your cloud based infrastructure. But, really, the data is where a lot of the challenges come into play that you have siloed datasets across the organization, some, some of which are in legacy databases and data warehouses, but you need to combine that with some information that is in vendor based systems and PACS and EHRs, et cetera. So, really, the data is one of the biggest challenges because, you know, AI doesn't work without data.

You have to have access to it. You have to be able to train, fine tune your your AI based on data, and then you need to be able to use it in actual, you know, making decisions, inferences, predictions, et cetera. So, yeah, the infrastructure is a challenge, but it's really the data that is, the most, difficult to overcome in order to commercialize AI, if you will.

Dr. Felix Beacher: Well, yes. Because data, as you say, is the food of AI systems, isn't it? They can't exist without it. But at the same time, when a new AI system is introduced, it is a major disruption potentially to established clinical workflows.

I mean, could you think of, based on your experience, an example of that going very badly wrong?

Dave Trier: Yeah. I think, a good example of that going, badly wrong is where something as simple as helping to summarize, summarize some of your notes. Right? Your clinician notes.

Yes. And yeah. I mean, it's it's right? It's so simple, Felix.

It's like, this is going to help you. Right? I just want to help to summarize The notes that we took as part of a, an office visit.

Right? And when it goes badly is that, first and foremost, EHRs can be quite involved. Right? So and just having that unknown or really just untrained, process to add on to it is where it can go very, very poorly where, you know, all of a sudden you roll out this clinician note summary capability within your EHR, and you just you don't have enough training and education around it, and you get the clinicians that don't know what to do.

And so they'll start recording stuff and then not know that they're recording or they'll start recording it and just talk to it in a certain way, and it just produces just absolutely atrocious results. Right? And so the clinician, you know, they're very, strapped for time, of course. And so they'll they'll try it once, they'll try it twice, and then they'll give up and say this thing is worthless.

So that's when something goes really, really poorly if you don't have the proper change management and training in place. This is just one example, of course. But without the proper training and change management for the how do it affects clinical workflows, it will result in a disaster and unused. Right?

The AI system will just go unused, and then a lot of negative feedback and say, oh, the AI is not what it's built up to be. And, no, I don't want to use this anymore even if this is a great use case that somebody else is proposing.

Dr. Felix Beacher: And I'm I'm guessing that as LLMs gets more integrated into healthcare systems, it's going to it's going to result in challenges that seem very strange. I mean, in the same way that a people have used LLMs to, support their lawyers have used it to support, legal cases And have got caught out because the LLM has just generated these fake case case law.

Dave Trier: And in the same way, I can well imagine that an LLM could just create completely new diagnostic categories and diagnose people with previously unknown conditions that kind of It's possible, but this is where back to the you put out your here's the utopia vision. Right? And bringing it back down to earth a bit that overall, we what I generally see in practice out in health care is that very controlled use around LLMs, especially. So you have more of a deterministic system that is guiding the process.

Yeah. It will call and use an LLM at the appropriate point to do things like, look up across, you know, some clinical case studies and the like. So it'll go and look some some of that information up, but it's still a deterministic path. It's not going to allow you to go and create new new diagnoses or treatment plans or anything like that.

So it's that's where health care is absolutely is doing the right thing of being cautious around making sure that you have the right guardrails in place or using repeatable deterministic patterns and pulling an AI at certain points of it, but not letting AI drive the whole process yet.

Dr. Felix Beacher: Sure. Okay. Now, I like to ask about disasters and worst case scenarios because I think that, firstly, they're fun and secondly, instructive. But why don't we try and be more positive?

Now I know that you have, experience in radiology AI systems. Could you talk about examples of a radiology AI systems, which you think are really good use cases of, how these kind of AI systems have been commercialized?

Dave Trier: Yeah. I'll talk through I talked about some of the challenges, organizational process, infrastructure, trust challenges. So let me give you a here's how it went well in the context of a radiology system. Right?

So we worked with a large health care organization that a health care provider that is that was, in the radiology department. They had, a particular vendor. So this is external technology, AI technology, that they wanted to bring in. And they wanted to bring it in to help them with some of their, studies, pulmonary embolisms, and the like.

And so what they what as part of the process in using our software and helping to streamline that process, They went from what in the past before was a very rigorous, arduous process, ten different teams, ten different systems, took anywhere between six to twelve plus months to something that was streamlined. It was consistent, and it offered the right level of trust, acceptance, and oversight throughout that entire process. And it went like this where they said, okay. The radiology department said, we got this great vendor.

I'm going to go and register this as an AI system because that's what we need to do per just good best practice and regulatory guidance. And from there, the system said, okay. Well, is this for clinical? Like, yes.

For radiology. Okay. Great. So then let's go and do a risk analysis around it, determine this to be a little bit higher risk because we're dealing with patient data, imaging data, et cetera.

But from there, it knew what to do with it. It said, alright. Well, based on this, you need to go and get a legal review. You need to make sure that the procurement team is helping you, with the appropriate t's and c's around it.

You also need to make sure that you're getting from the vendor the various, testing results, case studies, et cetera, related to it to verify the results. So all of that, the system helped to automate much of that process to pull it in. And then, lastly, it went to say, alright. Let's do a validation or review of it from our AI, committee, if you will.

Did that review. They identified a couple areas of risk. Let's catalog those risks, make sure we're following up on them, if you will. And then finally, once it said, okay.

You're approved to use it. Let's start using it. But then the system starts to collect the information. Right?

Think about monitoring. How is this actually working? Is it doing comparable or better results than the radiologist might be doing? So just that ongoing continuous verification or monitoring that the system is actually indeed delivering on the value and that we don't have any just very much off the radar type projections or, you know, types of, inferences that are part of the system.

So we went from a place that was, say, you know, twelve, the eighteen month process to go from idea to usage to something that is in the realm of, say, you know, nine to twelve weeks. Right? But you have the right level of visibility and oversight and making sure everything is done. So that's just an example, Felix, of how you can really turn AI and help to commercialize AI in a way that's repeatable, consistent, and has that right level of visibility and oversight.

Dr. Felix Beacher: Okay. So if we just wrap up, tie all tie all of that together, what would you say to leaders in the sector or government or the general public about what the biggest lessons are from your experience overall?

Dave Trier: Yeah. I think the biggest lessons are in order to commercialize the use of AI, one, you do need to balance innovation and oversight. Right? You can't go too off the deep end on oversight to the point where it slows innovation to a halt.

So having the right level and I'll use the Goldilocks of governance analogy, not too much, but not too little to ensure that you are had the right level of trust, oversight, visibility into AI, but you're not slowing down innovation. The second thing that I would say is to there are a lot of buzz. There's a lot of buzz around AI in the market, and there's a natural tendency for especially technologists in the area to go and chase every single shiny ball. So it's for a large organizations and leaders in those organizations, it's around vetting what are the particular use cases or areas that we might use AI.

Vetting that across, okay, what's the impact it can provide to our organization versus what is the potential risk and making sure that you have the most impactful use cases being the focus in the short term and don't chase those those shiny balls. Right? And then the last thing I would say, number three is just having the a consistent and repeatable process for how you move from idea through all of the various vetting, analysis, approvals, reviews, usage, and eventual eventual retirement of a system. So just having a consistent and repeatable process so that people aren't guessing so that you're there isn't a miscommunication or lacking of expectations across the different teams and stakeholders that might be using it.

Dr. Felix Beacher: Okay. Good. Look. I think you've given a very, very nice, overview there.

Let me ask you though, last question. Apart from lessons about deployment and commercialization, what is your favorite specific medical AI system out there?

Dave Trier: Yeah. Mine mine is definitely in, the genomics area, right, of being able to use various sets of AI, LLMs inclusive to help with genomics research. I have a personal, close to heart story around this. And so anything that I can see that's going to help in the innovation area around, genetic diseases is something that's near and dear to my heart.

So I'm I'm continuing to watch how generative AI AI as a whole is helping to evolve, speed up some of that, genomics and genetic based research. That's definitely one to watch.

Dr. Felix Beacher: Dave, thank you so much. It's been it's been very, very interesting talking to you, and very, very best of luck in the future.

Dave Trier: Thanks so much for having me, Felix.

Dr. Felix Beacher: Okay. Bye bye then. Have a good one. Thanks for listening.

Ping me on LinkedIn if you have any ideas on new subjects or guests.