Risk Tiering: What is it and why is it important to AI Governance?

Jay Combs: Hello, and welcome to the Good Decisions podcast. It's the podcast for AI governance insights. I am Jay Combs, the VP of marketing here at ModelOp. And this podcast is really going to be a series on practical insights and, frequently asked questions that we hear from our customers, from our prospects, and from folks throughout the industry about enterprise AI governance, including topics on, governance frameworks, regulations, generative AI, responsible AI, and it's meant to be, like, pretty tactical on answering these questions.

We hear a lot of, different, you know, jargon and lingos from data science teams, from governance teams, from ops teams, from development teams, from the enterprise team. We want to help people just understand what everybody is talking about. So we're all talking the same language and, understanding what is happening with AI governance. So that's our mission is to help, people understand what's going on with AI governance.

And each week, we're going to try to drop a new podcast and have a new guest on, with specific expertise on the industry or on a governance or on some topic that we're going to talk about. And with me today is Amanpreet Kaur, who is our customer success and implementation engineer. She has over ten years of experience with data science and machine learning models and doing research, and she has a PhD in astrophysics. Welcome, Aman.

Amanpreet Kaur: Hi, everyone. Thank you so much, Jay, for having me on the very first podcast for ModelOp.

Jay Combs: Yeah. That's really exciting. And Yeah.

Amanpreet Kaur: I'm I'm more than thrilled to be here.

Jay Combs: Awesome. We're thrilled to have you here. I know you're you're pretty busy, working with a lot of our customers, and, obviously, you're getting a lot of questions. And you I think you have a kind of a unique position to hear, what folks are asking or curious about, across a bunch of different verticals, a bunch of different customers, and you're on prospect calls as well.

So, we want to pick your brain as to what folks are really curious about or just need some, need some insight on to. So, for today's, podcast, we're going to talk about risk tiering. Specifically, you know, what it is, why it's important. It comes up a lot.

Obviously, like, different teams, are dealing with risk and risk management. But risk tiering has, like, a very kind of specific meaning in data science, with, you know, managing AI initiatives and models. So let's break it down. So, yeah, the first question is risk tiering.

What is it, and why is it important?

Amanpreet Kaur: Yeah. Of course. So if you think about risk tiering, it is, just putting, AI initiatives into different categories based on the potential risks, based on their severity and the potential impact, which basically help enable various organizations to prioritize their risk management workflow. And basic basically, what it does is it helps in the directional focus of an organization.

For example, a high risk model, based on this tier definition would be prioritized in many ways, and as compared to a low risk model, which can kind of might have to deal with less number of steps, to be dealt with in an organization. So that's that's a basic definition. Just basically putting a model into a different categories based on its risk assessment.

Jay Combs: Got it. And so, I mean, obviously, that innately sounds important for an enterprise to manage and to know what buckets their different initiatives go into. But why is it important to, you know, somebody I don't know. Somebody say, like, I'm on marketing team.

Why why does that matter somebody on a marketing team or an engineering team? Like, why does it matter to the business? Like, what outcome does that get me?

Amanpreet Kaur: Yeah. Very important question and good question. So, yes, it is it is very important. That's not something that we just do, but, the reason behind it, as I already kind of mentioned that it helps in prioritizing the focus as well as the resources, for an effective workflow for an organization, so that we can have the right size of governance to enable the business in the right direction as well as accelerate the innovation.

So for example, we have we have all heard the especially in the last few months, actually, now more than a year, we have been hearing a lot of horror stories of data leakage. And, all these, stories, they come up because they were not proper risk tiering identification provided to that model. If that is in place, then the model will go through a lot more scrutiny. And that way, we make sure that nothing, that is, very nothing should be leaked.

Nothing that should be a public information should be, leaked out. So that is just one process as an example.

Jay Combs: No. I assume you're talking about I know there's, like, the examples of Samsung, I think, putting, some proprietary or trade secrets into, ChatGPT or specific data that was, kind of a no no, and there's a bunch of other examples out there.

Amanpreet Kaur: Exactly. Yeah. Yeah. Exactly.

We have we have enough examples to not tell the organizations that risk tiering is important, and we should not skip this process.

Jay Combs: Got it. I noticed it the word process or the word, you know, governance or compliance can sometimes feel really heavy, but it seems like as you're describing the risk tiering, the process, it's it's designed to help, like you were saying, like, accelerate innovation. So if something is lower risk, like, as a marketer, I can use that model, like, more quickly. It has less checks and controls in place than maybe, you know, a high risk model.

Is that is that the case? Is that the right way to think about it?

Amanpreet Kaur: That is that is exactly right. Yes. A low a low risk model, will go into production. Basically, you can, yeah.

So that is something that is feasible, on a shorter time scale. Whereas something that is marked as a high risk model, so you have to look at its feasibility and what time frame, how much time it's going to take to get implemented. So it also gives you a great idea about the whole life cycle of the model. So that is an important aspect as well.

Jay Combs: Got it. And I know, you know, there's, you know, some new regulations out like the EU AI Act was, approved. I know it's most likely going to go into force, this summer, but the, you know, the EU AI Act has, like, different tier. It's a risk based, legislation.

So you have, think, you know, prohibited, risk that's, like, unacceptable risk. You have high risk, and you have low risk. But how do how do you actually kind of implement this? How do you I mean, it's one thing to have, like, these buckets of risk, but how do you actually implement a, like, a risk tiering function within your larger AI governance framework?

Amanpreet Kaur: Yes. That's that's a very complex question, because there is no one way or one rule based on different industries. So, the very first point, as I mentioned it, is very industry specific. So you can have different risk criteria for a financial institution as well as a health care one.

Right? So that way, that is the very first point. You have to see what is the high risk in health care versus finance. Like, in finance, you can have the credit card report, whereas in health care, does it involve, clinical trials, or diagnosis for patients?

So that is pretty high risk. Right? So that's that's one example of, like, how you do it. That's just one bullet point if you think about it.

And then within those organizations, you can have different policies for each and every branch. So it depends upon those policies. And then the next very important point is the data privacy. So, again, because in again, taking going back to the health care and, the financial institution both involve data which involve people.

So the data privacy as well as the quality, that is extremely important. Safety, compliance, regulation requirements, and the environment in which you deploy these AI initiatives or the models, how robust they are, like, security wise. That is also important. And, as well as the model ownership, Like, how when they develop it, what kind of risks they are taking.

So there are a lot of questions. So, basically, it can be ten controls, five controls, or, like, a hundred-plus controls. So the risk calculation is not very simple in that sense. But these are some of the factors that do play a role in defining a category.

Jay Combs: Got it. And, yeah, there's obviously a lot to kind of dive into and break down here. But one question. Like, does the risk tiering have to apply to all types of models?

Like, for instance, like, Microsoft has, you know, Sales Copilot or Salesforce Einstein or embedded AI in those types of systems, or if I develop, initiatives in house or models in house using my own tools, do all these different types of models have to be tiered, or is it just apply to what I develop within my own organization?

Amanpreet Kaur: So, I mean, every model that is going to go out there in production, which is going to have some sort of impact, should have a risk tier defined. If you are starting an initiative, you're developing something locally, but you are not putting that into production, but there is a reason you are doing it because you are ultimately going to make that use that template to make something more robust and which will go out, the that whole initiative. So I would I would say that it is important to start about start thinking about risk tiering from very early stage.

Jay Combs: Yeah. I mean, that's something we've heard at some conferences where people ask, hey. I'm, you know, using, you know, software that's got embedded AI or a third party model that am I am I responsible for that? And the answer is yes.

Like, with a lot of the new legislation, even when it's when when it is risk based, like, you have to account for all model types, all initiative types.

Amanpreet Kaur: So Exactly.

Jay Combs: I just wanted to kind of dive a little bit deeper into that, there. Aman, you mentioned a bunch of bullet points there on how to actually implement, risk tiering. But can you be a little more specific or maybe kind of bucket those bullet points into, like, a broader approach for tackling risk tiering, just kind of make it a little easier to understand?

Amanpreet Kaur: Yes. Definitely. So, so, basically, what you can define, as three different categories here. So the very first one is the model complexity, which talks about the mathematics, the statistics, like, what kind of is it a linear regression model or is more complicated work?

So that is how complicated the model is. So that is the very first point. Right? Like, how many resources it's going to need, and that is also a big factor.

And then the second one is its intended usage or purpose. Why are we doing this? People can throw in a lot of models, but if we don't have a purpose, that doesn't really go anywhere business wise as well.

Jay Combs: Is that like, hey. This is a we're approving somebody for, like, a home loan or analyzing, like, radiology data or MRIs and that kind of thing.

Amanpreet Kaur: Exactly. Exactly. So let's say if there is a model that involves people, like, for example, in terms of banking, if it involves financial reports, credit card reports, which is going to be the most important factor for providing loans at this high risk. Right?

And then health care, is this model going to be used for, providing diagnosis or clinical treatments? High risk. Right? If it is just about, let's say, optimizing a schedule for nurse, that's not high risk, and similar kind of examples on both sides.

And the third important factor would be the exposure. What is its impact? And, again, coming from the second point, which is the purpose, the impact would be if you're diagnosing a patient, based on what is provided by a model's output. So you are talking about, a very high risk impact.

So you have to take that part into consideration as well. And same thing for the financial. So model complexity, its purpose, as well as what exposure it has, what impact it has, these are the three broad categories that you can define, in terms of to define risk tiers overall. And then these can be broken down into different subcomponents.

Jay Combs: Got it. So complexity, purpose, impact.

Amanpreet Kaur: Yes. Exactly.

Jay Combs: Okay. Well, let's let's keep rolling. There's a couple other, I guess, like, big things to mention here. You talked about, like, the scaling at the end, whether it's, like, five points of control or five hundred points of control, which is a lot.

Like, and if there are that many points of control, how do you actually bring these models to market? Like, that seems almost kind of impossible to do manually or without an AI governance process. So how do you kind of scale up this risk tiering with that many checks and balances?

Amanpreet Kaur: So you would you would hire someone to do it manually and sit there for ten days. No. I'm kidding. No.

Yeah. I'm kidding. So, obviously, not. So yeah.

So you have to, that's that's where the scaling part comes into play. Right? Like I said, it could be ten points. It could be hundred points.

It could be more than that depending upon the organization's requirement and the model requirement. So you should have the ability to programmatically calculate all these risks. So, basically, automate that. And, so, like, for example, we have seen some use cases where, you know, you can you can define the criteria.

Like, you just have, like, ten criteria that can be done in a day, and then people might be like, oh, I don't really need to automate this process. This is fine. But then the next day, they come up with a model which now needs hundred criteria, like, hundred control points. Then they're like, oh, now we need to do automation.

So the idea is that we can have this automation process in place, where we can make every model go through. And depending upon its risk tier, it can either go through these ten control points or can go through those hundred control points. So that is essential. So we do need to automate this process based on risk.

Jay Combs: Yeah. It seems like automation is just absolutely critical to doing, yeah, doing this at any level of scale. It just seems impossible to take something to market. It would just take way too long to do without

Without that, automation. So that's that's a huge point. And you're talking to a lot of individuals at Fortune 500 companies, folks from data science teams, governance teams, IT leaders, AI leaders. What's the big trend, or what's a top trend you're seeing related to risks, risk tiering, and AI governance?

Amanpreet Kaur: Well, I would think that the obvious one is the scaling, and the one that is not so obvious, but that happens, with scaling is a complexity. When I say scaling, I'm talking about the number of use cases which use AI. So a lot of companies are now investing in use cases which use AI, and these AI initiatives bring their own risks. And so the risk calculation becomes more complex.

For example, we have seen customers go from three or, five to over a dozen to define the risk criteria. And if you throw GenAI in and they bring their own risks, some of them we haven't seen before. So we have definitely gone from calculating, risk using a handful of criteria to these giant spreadsheets that now we are seeing from all these customers. So these are the main changes that we have seen.

Jay Combs: Interesting. So, yeah, that's really interesting. What So with that kind of rising complexity, what, if any, impact does that have on, you know, the controls for managing those initiatives throughout the model life cycle? Is there an impact on what those workflows look like for the especially the more high risk models?

Amanpreet Kaur: Oh, yes. Definitely. So, overall, the complexity for risk changes, so everything inside it, which is the control points, yes, that also changes. So, for example, like, almost a year ago, we saw these model life cycles with about a dozen control points to calculate this.

And now we are going upwards of hundreds. So that's definitely that's something that we can no longer manage manually. So we do need to, automate the whole process. So the scaling and complexity is definitely making the need of automation almost, necessary.

Jay Combs: Yeah. Got it. Well, that in itself is a is another topic to dive into. Maybe we'll do that.

We'll talk about, the broader model life cycle and the complexities, the need for automation in our next podcast. But I think we'll we'll stop it there for now. And thank you so much for joining us and giving us the insight into risk tiering. So in short, like, just to kind of summarize what we covered, risk tiering is a way of, categorizing AI initiatives based on, complexity, purpose, and impact.

And, obviously, there's different guidelines and criteria depending on if you're in, say, like, financial services or investment management, or you might have some guidelines based on, like, the SR 11-7 letter. Or if you're in, health care or medical devices, you might have guidelines from the FDA or, software as a medical device. So there's different criteria for creating this, but the scale and number of models and the for calculating the risks of those, of those models is just growing, and the need for automation is really critical. And, for, you know, for ModelOp shameless plug here, like, that's really where automation, can help.

And, you know, for folks who are struggling with this issue and seeing these scale and complexity challenges, with our experts. If you go to www.modelop.com, you can get in touch with us. And that's it for the first episode of the Good Decisions podcast on risk tiering. We'll be doing another episode soon.

In the meantime, please subscribe to the podcast, and, we look forward to talking to you all again soon. Thanks Thanks so much.