The Model Life Cycle: Why It’s Important and How to Get Started

Jay Combs: Hello, and welcome to the second episode of the Good Decisions podcast, which is ModelOp's AI governance insights, podcast. And I am your host, Jay Combs, the VP of marketing here at ModelOp. And with me today is our guest, Christina Morandi, who is the director of customer success at ModelOp and works with, all of our customers and is very familiar with all the different kind of trends and, things that are happening in the world of AI governance. Welcome, Christina.

Christina Morandi: Hey, Jay. Thanks for inviting me. Glad to be here.

Jay Combs: Awesome. It's great to have you. The last time or in our first episode of, Good Decisions, we talked about risk tiering and why is that important, to AI governance. And we started talking about how risk tiering does ultimately impact the different model life cycles and workflows that we have.

We don't really have time to get into what that is or why it's important. So that's what we're going to talk about today, model life cycles. So, let's get into it, and we'll start with a really straightforward question. What is a model life cycle, and why is it important?

Christina Morandi: That's a great question, Jay. It's also a loaded question because we haven't we need to be on the same page as far as what is a model. Right? So, you know, we like to cast a wide net when we're talking about models.

So we use, we refer to a model as any, like, AI initiative and or any, like, statistical, mathematical, or kind of, rules based decision making model. And so, like, some examples could be, as simple in regards to model, something as simple as a spreadsheet with some calculations or something more complex like a machine learning model of, you know, machine models machine learning models of various complexities. These could be like in house models that are that are built for purpose or vendor models or generative AI, LLMs, neural nets, like I mean, the list goes on and on and on. Right?

Again, we're just we're, so in regards to models, we just try to cast a wide a wide net here, because because each one of those model types, regardless how simple they are or how complex they are, especially when we're talking about, like, ensemble models with multi with multisteps, Each, regardless of the complexity of the model, it's really important to have a models a model life cycle in place. And the model life cycle is basically just the workflow, right, that model goes through from ideation to retirement. It's all of the steps that model has to go through from development and testing, to monitoring monitoring controls and reporting. Basically, anything that you're going to do with that model, you want to make sure you understand.

Jay Combs: Got it. And why does a business need that? I mean, everything you made set, like, logically makes sense. But, like, why is that important, to a business?

And more broadly, why is it important to, like, the larger enterprise?

Christina Morandi: Modeling and AI initiatives, typically aren't cheap. Right? Everything around, you know, developing these models or purchasing these models, Right? It's it's expensive, and so, we really need to have a plan in place in order to make get the most value, out of those investments.

And there's also a lot of risk around, models. Could be the model itself, could be how the model is being used, what the exposure is at the model. We talked a lot about this a little bit. You guys talked about this a little bit on your last podcast, in regards to defining risk tiers.

And so if you're dealing with, especially when you're dealing with higher risk models, you want to make sure you have a full grasp of the requirements that need to be in place in order to be able to work with that model. So, again, it's really important for businesses to get a handle on what the full model life cycle is as soon as possible so they can make sure that they can actually use that model in the real world, that it's going to be compliant. It's not going to, cause any issues, and that while it's in operations, it's not causing any issues. And you have the steps in place in order to be able to put those controls, and you can the controls for, monitoring, reporting, in place.

Jay Combs: Got it. And so what you said a couple you had a couple terms in the, like, requirements, controls, regulation. So is it fair to say that, like, you know, a big part of this is making sure that, especially across a large enterprise, we might have different teams using different tools, doing things differently, trying to create, like, consistency for compliance with regulations or internal policies So everybody's on the same page and enforcing kind of the rules and regulations in a consistent way, across the whole enterprise.

Christina Morandi: Correct. So you want to be consistent for the model types that you're investing in. And so, you know, for example, if you know there is policy in place that is relevant to a generative AI model that you're using because it is customer facing or client facing. High risk, if you will.

High risk. Right? Then it's it's it's really important that you're that you're governing all of your GenAI and LLM models, not only comprehensively, but consistently.

Jay Combs: Got it. And that's kind of the tie, like, I guess, to the risk tiering. You might have high, medium, low risk defined differently, and different model types or initiative types might have different checks and balances they need to go through. So it's not certainly not a one size fits all, but, you know, consistency having that risk based approach is real important to, to the workflows themselves.

Christina Morandi: Yeah. You know, a lot of our customers, they'll start with that risk tiering in order to define the model life cycle steps per risk tier. Because the steps the controls the monitoring controls and reporting requirements that need to be, put in place are typically associated with that risk tier.

Jay Combs: So interesting. So it seems like this can get pretty complex with low, medium, high risk model. So can you give us some insight into, like, in the real world where these life cycles get complex for organizations and what impacts those have?

Christina Morandi: Yes. So there's two areas of complexity. One, as you can imagine, is obvious. Right?

It's the model life cycle itself can get pretty complex depending on all the all of the controls and monitoring that need to be in place. There might also be some validation and review steps in there that require different groups of people to be involved in that in that model life cycle. You know? And as the investments in AI increase, as more AI is getting developed or purchased, within our customer environments, we're seeing more complexity as well in requirements for a variety of reasons.

So, anyways, for example, like, a year ago, you know, we our customers on average would have model life cycles with, you know, twelve to twenty steps max. Right? And now we're seeing model life cycles with 300-plus control points. So just to give you an idea of the complexity there.

So, we're talking about that many control points. We're talking about that many steps within the model life cycle, managing that through manual processes, through spreadsheets, through individually emailing people in order to get their input, it really becomes, more difficult, which is one of the reasons why a lot of our customers prioritize their higher risk models, right, as far as getting them, you know, governed and managed as quickly as possible. Now so we talked about the model life cycle complexity specifically, but the other aspect that we need to consider is the actual volume of models. Right?

So just as, you know, a year ago, maybe, you know, most of our customers were dealing, you know, had a dozen, models that they've been developing for a while, in house models, with the rise of GenAI and LLMs and vendor models, and just AI being used for more purposes, Our customers that may have had may have had to manage twelve models a year ago are now managing hundreds of models. So you can imagine how difficult it gets to, to handle a model life cycle, for hundreds of models with hundreds of control points, using manual processes. It pretty much managing these workflows manually, it is we're we're at a we're at a point now where it's just not possible without automation.

Jay Combs: Yeah. It sounds almost overwhelming because this isn't just like a data science team issue. Right? We're working with compliance, risk, legal, infosec, data privacy, various other teams that need to be involved with the enforcement and review validation of the models that are going going through.

And it's going to be changing. Right? Like, as more regulations are coming online every week, it seems like at this point. It's just a massive scale problem.

And so, yeah, I can see why this is so challenging. So can you maybe take us into how do you scale us? How are we going from ten to, you know, three hundred, model ideas and running them through these, these lifestyle, cycle workflows. How do we how do you even get started with that?

Like, so how do you get started, and how are folks doing it today?

Christina Morandi: So having the right tools in place is really important. As I mentioned, you have to be able to automate a lot of these steps, especially around the monitoring piece, the controls piece, testing and validation, ongoing reviews as you mentioned, and other, you know, risk management reporting, components. You have to be able to automatically integrate with other systems and kick off different processes and notify individuals if there are still manual processes.

Jay Combs: Like, what kind of systems? Just that I can ask you.

Christina Morandi: Just any any, any of the systems that are used in order to manage the model, like any ticketing systems that you need

Jay Combs: Confluence or something like that.

Christina Morandi: Yes. Correct. And then, obviously, from a communication standpoint, the emailing, you know, being able to, to be able to automate emails. And then there's, from a monitor monitoring perspective, the way that we're we're monitoring our models is getting more complex and more varied as well.

So being in being able to integrate with the with different systems or be able to ingest those, monitoring calculations and approaches, that's that's also something that being able to put that all into one place and automate it will will help from a scaling perspective.

Jay Combs: Yeah. So IT systems, data science, machine learning tools, Got it. So I guess yeah. It can be an integration nightmare.

Right? It sounds like. I mean, yeah, you're basically trying to work with an existing tech stack and do the communication around it, know who's responsible for what, when it's their turn, making sure nothing slips through the cracks, especially, with something that's that's high risk is really, really, intense and concerning and probably keeps people up at night.

Christina Morandi: Yeah. And it's a symptom of, again, more and more teams using AI, investing AI, and the need for there to be disparate systems. Whenever you have a need for there to be disparate systems, you have a need for more complex integrations. Right?

And so having manual processes in place really doesn't work for that, or having tools in place that are vendor locked and specific to very specific model types that may not work with a broader investment in AI across a broader set of model types and model tooling?

Jay Combs: So there's a lot of complexity here, certainly at different risk levels. So when do you as an enterprise or somebody who's involved in model life cycle, when do you get started, I guess, kind of documenting the policies and trying to figure out what that life cycle and control should look like?

Christina Morandi: Well, you want to figure out what your requirements are and how you're going to use your model as soon as possible. And the answers that you obtain there will result in your, in the steps of your model life cycle.

Jay Combs: So it's really important to So right away is basically the answer. Like, as soon as you come up with the idea, you're almost starting to you're thinking about the life cycle.

Christina Morandi: Exactly. You want to think about it right away because at the end of the day, you don't want to spend all of this money investing, in house developing in house models or paying for third party models and then not be able to use those models because you didn't have, like, a proper workflow in place in order to be able to properly monitor or manage and control those models according to the policies that those models have to adhere to. So the sooner you figure out what the model life cycle looks like, the sooner you can start moving your model through through that model life cycle, and you can start getting value out of out of, out of your model and out of your, investment.

Jay Combs: That makes a lot of sense. So you start right at the beginning. Right away is when you want to start the documenting and preparing, the process, which then begs the next question because that can feel daunting. How do you actually get started?

What's the starting point?

Christina Morandi: So everybody starts at the same spot. Right? Everybody has to understand what the requirements are, what the policy is, specific to their model, and what the steps need to look like. Right?

And that is probably the most daunting task. And that is why at ModelOp, we have spent a ton of time, researching, the latest policies and guidelines. For example, the EU AI act, both high risk, low risk, SR 11-7, NIST standards. I mean, you name it.

Like, we live and we breathe researching, these policies so that we can extract the requirements from those policies into templates and give our customers a place to start that isn't so daunting. Right? And so I think everybody who is focused on building model life cycles and making sure that their in model investments are adhering to the latest policies. Everyone should be building up a library, right, of templates that they can that they can just pick up, based off of the latest policies, how they're going to use their model, what the risk tier is, to have a place to start and then also build a standard, right, for each.

Because at the end of the day, right, at the end of the day, it's all about how quickly can we get and we're going to sound like a broken record here. How quickly can we get a return on our investment? Right? How quickly can we see value, and how quickly can we start using the insights from, our models used operationally?

Jay Combs: Got it. It makes a lot of sense too. And not just I think, like, yeah, certainly, you have the value in the ROI is important, but also that risk mitigation piece of making sure which is tied to the to the value, obviously, making sure that everybody's using and enforcing the same policies and regulations, especially across, like, a large enterprise. Everybody's on the same page, and it's consistent, which makes going on market more efficient, which ultimately leads to better ROI.

So you're working with, leaders at Fortune 500 companies every every day, implementing, you know, these model life cycles, understanding the policies and regulations that need to be enforced. What key lessons or insights have you collected, from, from this experience that you wish, you know, anybody getting started in their AI governance journey and starting to define the processes around model life cycles should know.

Christina Morandi: So, yeah, we're still learning every day, but some key takeaways are you don't want to start from ground zero. You want to use, templates, around the latest policies that are relevant to the models, that you're working with. You want to use templates as much as possible. You want to use out of the box metrics, and monitoring as much as possible.

Again, you want to you want to what you want to use what you have. And if you have something like a template around a specific policy available to you, like, we offer at ModelOp, you want to take advantage of that. And you also want to be really smart about prioritizing the model life cycles that you're defining. And so, you know, a lot of our customers will jump right into their highest risk, highest exposure models to get a handle on and have, you know, have, proper controls and monitoring in place, off the bat, to reduce their exposure, reduce their risk, and, continue to see value from those higher risk models or higher exposure models.

Automation is also this, you know, this may be obvious, but the faster you can move, through your governance, processes, the faster you're going to see your return on investment. I don't understand. Without automation, how this would be possible at an enterprise level, especially with the volume of models and the complexity. It seems like you're just asking for trouble without having a consistently, defined enforced process that's done through automation.

Right. And, we work with a lot of customers who have tried to throw people at this problem and have been unsuccessful. And so, yeah, I mean, automation is really key, to be able to ensure that the processes in place are followed and they are standardized and they are transferable, and they can exist without the need for a specific individual or a specific organization. So another thing, that we haven't really touched on yet, a best practice is having defined model life cycles, having defined governance processes for all of your models.

So we talked about maybe templatizing. We talked about prioritization. We talked about automation. But there's a visibility piece in there as well that becomes really impactful, to make sure that you are implementing your governance processes consistently across all of all of your models, across all of your groups that are investing in using models.

Jay Combs: Got it. So automation is obviously very important. But, you know, another thing I'm curious about, if I'm an executive or, you know, a leader at an enterprise, I'm maybe not in the weeds of, you know, the individual steps of a model life cycle. How do I get visibility, or how do I understand how compliant we are with certain regulations or how at risk we are with certain, certain issues.

Is there a way to report or quickly give insight to leaders in an enterprise, based on, you know, where all the different model life cycles are across the organization?

Christina Morandi: Yeah. So you really need to have a report at the end of the day on all of your model life cycles. It tells you how many models you're even you even have running, what those models are doing, what their compliance score is, and I think we're going to talk about this one in the next on the next, podcast. But, you know, having this concept of a governance score that measures the health of your of your models is key.

So it's it's one thing to understand the model process, the model life cycle itself. It's one thing to be able to implement that process consistently. It's another thing to be able to get the full view of everything of everything running. And before ModelOp, a lot of our customers couldn't answer those basic questions like, how many models do I have running?

What are those models being used for? What is what is the risk associated with these models? What is the exposure? And so, we've we've invested a lot of resources into being being able to answer those questions.

Jay Combs: Awesome. Well, I think it's a great place to stop. Covered a lot of ground today on model life cycles. And you're right.

I think in the next episode, we're going to start talking about some of those, those reports, the AI governance score that you just mentioned because there's a lot more to dig into, on that front. So, Christina, thank you for the insight today. Thank you for joining us, and hope to have you back on the podcast again. Any final thoughts?

Any final words?

Christina Morandi: No. This is really fun. And, you know, if I haven't said it enough on this podcast, nobody should be starting from ground zero here. No nobody should be sending emails manually or filling out really complicated spreadsheets.

So we can we can do better than that.

Jay Combs: Awesome. Great to hear. Thanks for joining. Thanks for listening, everybody.

Don't forget to subscribe to the podcast. We'll be trying to come out with new content, on a weekly or regular basis, and look forward to chatting again on the next one. Bye, everybody.