Turning AI Policies Into Workflows

[00:00:00] Kevin: Hi, I'm Kevin Werbach, professor of legal studies and business ethics at the Wharton School of the University of Pennsylvania. For decades, I've studied emerging technologies, from broadband to blockchain. Today, AI is promising to transform our world, but AI needs accountability, mechanisms to ensure it's developed and deployed in responsible, safe, and trustworthy ways.

‍

[00:00:32] Kevin: On this podcast, I speak with the experts leading the charge for accountable AI.

‍

[00:00:40] Kevin: Modelop provides AI governance solutions for large enterprises. I talked with CEO Dave Trier about what it means to turn corporate policies into automated workflows, why pre-approved patterns for low-risk AI can speed, deployment, why traditional governance risk and compliance, or GRC approaches break down for AI, and what it takes to govern agents all the way down to the network layer.

‍

[00:01:06] Kevin: Here's my conversation with Dave

‍

[00:01:11] Kevin: Welcome, Dave, and thank you for joining me on "The Road to Accountable AI."

‍

[00:01:16] Dave: Thanks. Appreciate you having me, Kevin. Why don't we start by talking about what exactly Modelop is and where it fits within the larger AI governance stack? Sure. Be happy to. We're an enterprise software company that's focused on large scale AI delivery and governance, and we'll talk, I'm sure, quite a bit more about where that fits into the overall ecosystem.

‍

[00:01:38] Dave: But at the end of the day, we help large enterprises to go and implement AI governance just as part of how they do AI delivery, and we have a very powerful software platform to enable them to do that at scale What's different about doing AI governance for large enterprises? Yeah. With large, with large enterprises, there is a lot that's involved from a policy perspective, right?

‍

[00:02:02] Dave: So for large enterprises, it's not just the regulatory policy. There's all kinds of corporate policies, as you can appreciate.

‍

[00:02:09] Dave: There's corporate policies that are related to, you know, the use of data, corporate policies related to security, corporate policies related to the type of information that can be exchanged, the type of usages that they may allow or disallow from a corporate perspective. So at the end of the day, it comes down to how do you help them implement and enforce those corporate policies at scale?

‍

[00:02:32] Dave: There's a few other minor ones, Kevin, but that's, that's at the, the core of it, the key difference when you talk about enterprise AI governance. What does that look like in practice, taking those corporate policies and translating them? Yeah. It's, that's, one of the, the big things that you have to do is that generally there's corporate policies that are 80, 100-plus pages, right?

‍

[00:02:53] Dave: And so how do you translate that into something that is really straightforward, but also embedded into how the different business teams want to go and develop and use AI? So kind of a-- In practice, what happens is that, and this is obviously where our software is great at, but what happens is we take those 80 or 100-page documents, which have a lot of process flows as well, right?

‍

[00:03:16] Dave: There's a number of different processes. So we take those, and first and foremost, we take the processes and turn those into something that is really orchestrated, right? So instead of somebody going to have to read the 120-page document, rather it's orchestrated. So if you're just a developer of an AI solution or you buy a AI solution, you just kind of get shepherded through the process.

‍

[00:03:37] Dave: You get shepherded through being able to adhere to all the different steps that are pertinent in the policy to what you're trying to do with your AI solution. So a lot of what we do is help to translate the policy into something that is usable, actionable, and obviously scalable, around that policy, set of policies and procedures throughout the overall, enterprise.

‍

[00:03:58] Dave: How automated is that? Yeah, we try to automate as much as possible, right? Obviously, we can't automate everything. And the purpose of governance and the risk teams and the control partners is really to have effective challenge, right? So we d- you know, you never want to get rid of the actual effective cha- challenge that's done by the different humans involved, right?

‍

[00:04:19] Dave: From legal risk, compliance, governance risk, et cetera. But we try to automate all of the really just the mundane routine tasks. Things as simple as you know, we, we do need to analyze what data is being used, and are there any potential PII, PHI in there? You know, a lot of times that is done manually, right?

‍

[00:04:39] Dave: Another example is that you want to do a risk tiering. Very simple concept, but a lot of that's still done with an Excel spreadsheet. You go fill it out, and, this Excel macro kicks something out that says, "Oh, this is a high risk," or, "This is unacceptable use," in the EU terms.

‍

[00:04:53] Dave: So on average, to answer your question, it's, for the, the lower priority, or sorry, lower risk, maybe medium risk, you can actually automate between 60 and 90% of that process. For the high-risk ones, that's where you do want to take a little bit of extra time, as you well know, to have humans go and take a, a very in-depth review, do their effective challenge around it, identify where there's potential findings or validations, conclusions in the banking world, et cetera.

‍

[00:05:20] Dave: But so that one you probably, you know, a little bit lower percentage. But, like I said, the low-risk ones or one that we're coming up quite a bit around, Kevin, a concept that we, we call just pre-approved patterns, especially in the GenAI and agentic world. If there are, at the end of the day, that people are just creating chatbots, right?

‍

[00:05:36] Dave: And so there are certain patterns that you've identified that say, "Okay, well, we know this pattern. We don't need to redo the whole risk analysis around it because, you know, 90% is the same as the last one," right? So that's where we get some real, very nice savings, time savings, accelerations, because we say, "You know what?

‍

[00:05:56] Dave: 90% of this is the same. We've got all the evidence, and we've got all the reviews. Everything's already been done with that, so we can just focus on the 10%." So again, just, that's just an area where you actually can help to streamline and automate a lot more than, than you would think through the use of, of technology like ours.

‍

[00:06:13] Kevin: But presumably, as you said, companies have different policies, especially if we're talking about large enterprises, and, and certainly some of it is, industry-based, how do you help them decide what are those patterns that for them they can automate?

‍

[00:06:27] Dave: Yeah. So first off, absolutely on every single enterprise has a slightly different policy- Mm-hmm ... and sometimes drastically different. And, you know, just in general, that's why we really, from our company perspective, only focus on the Fortune 500 Global 2000- Mm ... is because we really designed it so that every company can make slight tweaks, they can configure and tailor to their specific policies and procedures as they put forward.

‍

[00:06:50] Dave: So that's that, that... On the first bit, I completely agree with you, and that's why we've designed it as such. Mm-hmm. And then on the second one, which you said of, well, how do I then take the learnings of, from, say, one customer to the next when they do have a different policy? In general, that's where you do start to see some repeatable patterns, believe it or not, right?

‍

[00:07:08] Dave: It all comes down to the usages. What are the target usages? What data do they have access to? what other tools in the, agentic world are the tools they have access to? Is this internal versus customer facing? So there are a number of core, questions or inputs that- 

‍

[00:07:23] Dave: can help to determine what those patterns look like, even if it's you're looking at a bank versus wealth management, if you will. they get, repeats that are fundamental to determining things like the risk rating and how we want to handle it from an ongoing process perspective.

‍

[00:07:39] Dave: So let, let's just drill down a little bit more. Sure. You talked about n- you know, the lower risk tiers where it's possible to have a lot of auto-approval. Yeah. are there consistent factors that would put something in that bucket, or does that have to be a bespoke process each time to say what's at a low tier for your organization?

‍

[00:07:59] Dave: Yeah, so, two parts of that. So in terms of the, the tiering, that is something that's often very well defined- Mm-hmm ... especially for the regulated industry. So the- Mm-hmm ... the actual risk tiering itself is something that is, again, well defined. Mm-hmm. It's very easy to help to automate based on the inputs, if you will.

‍

[00:08:15] Dave: Then the second part of that is then you can further do, like I said, those pre-approved patterns- Mm-hmm ... after the risk tiering's done. So let's just say that the risk tier kind of kicks it out as a low risk, if you will, and then you can look at a number, to your question, of factors, and those factors often come involved of, okay, is this a, a pre-approved LLM?

‍

[00:08:33] Dave: So oftentimes that's, you know, the, the, the core team technology and risk and governance teams will go and kind of validate and approve an LLM version in particular to say, "Yep, we've looked through this one. We've reviewed it. We've done all the, the tasks. We've done all the testing. We've done all the due diligence."

‍

[00:08:49] Dave: And so they'll list this as approved for production usage. Okay, great. So if you are now developing a, a chatbot or an agent that's using that pre-approved LLM, that naturally says, all right that's one item we can check to say- Mm-hmm ... "Yep, we're into that pre-approved bucket." Another one, obviously very common, is this internal versus external.

‍

[00:09:06] Dave: Internal, that gets, obviously, has a lot less risk potentially around it, so that's another one that you can look at. And then the, another one is also if you are connecting to different data sources, especially the corporate data sources, around it. If it's something that you're really not connecting in and pulling in other corporate data, if you will- Again, that's something that you can generally say, "All right, well, we're limiting our exposure because we're not gonna feed it any potentially sensitive data."

‍

[00:09:33] Dave: Whether or not that data source has sensitive data, it still limits your exposure around it. So those are kind of just a, a couple of examples. and, you know, each company is actually refining this as they go- right? So they'll get a new, you know, a new gen AI, agent that comes through the pipe- 

‍

[00:09:47] Dave: and they say, "Oh, wait a second. Yeah, this looks the same as this, and this, but these two new things came in the door." But I think we can see that, one of those two items, we'll see a lot again. So then they'll go and approve that and have that as part of that kind of pre-approved pattern, if you will, put that into the cycle, et cetera.

‍

[00:10:04] Kevin: So it's a- Mm-hmm ... it's a learning piece as well from the, the overall kind of execution, protocol from the different governance teams. All right, so at, at the bottom there's things that get pre-approved. Presumably there are some things that are absolutely prohibited at the top. What do the middle tiers look like, and, and what's the, the functionality that you help, provide to organizations to go through the necessary processes?

‍

[00:10:27] Dave: The middle tiers is where you get into just making sure that, well, first and foremost, you're following all the steps in the corporate- Mm-hmm ... policy, So this is where part of what we do is we're mainly focused on workflow automation and automation- Mm-hmm ... of those processes, right?

‍

[00:10:39] Dave: So this is where the integrations come into play a lot. So integrating with your security scanning capabilities- Mm-hmm ... integrating with, say, your data profiling, data integrity, type capabilities to make sure that, yep, security cans, security scan came back clean- Mm-hmm ... data profile came back and indeed it does not have any PII.

‍

[00:10:55] Dave: And then pulling in, excuse me, other corporate policies like looking at if it's a third-party AI, pulling in the Ts and Cs that might come from a procurement system. Hey, do we have the Ts and Cs? Has legal signed off on them? Terms and conditions- Has procurement signed off? ... Right. So those are the types of things, and then you move on to testing, right? So e- for the, the kind of that middle tranche, you can have a set of, of tests. It's not gonna cover every- Use case specific nuance from a testing perspective, but there are some baseline tests that you can take a look at that, again, if you said, "Hey, there's no PII, let's go and check," right?

‍

[00:11:27] Dave: Let's go and test it and make sure there isn't any PII coming in or out of it. You can look at things such as, hey, we said that this is going to be, you know, a, a chatbot and that this chatbot, we wouldn't expect it to be overly negative. Let's go do, run a sentiment analysis- Mm-hmm ... to determine, hey, is this coming back overly negative?

‍

[00:11:42] Dave: We wouldn't not want that for even if it was an internal chatbot, if you will. So there is a set of tests that then you can, run and help to automate as much of those as possible, looking at deviations from what was requested, as well as kind of the accepted norms and risk tolerances that you have around some of those tests, if you will.

‍

[00:12:00] Dave: So that's kind of what happens in that middle tranche of just- Mm-hmm ... let's try to, again, run all the different scans and the tests, help to prepare some documentation so that the reviewers can then take a look as like, all right, well, here's a summary. Yep, it passed security scan. Yep, it passed the data integrity.

‍

[00:12:16] Dave: Mm-hmm. Here's the tests that run. Here's the results of those tests. Here's where it actually failed a threshold. So you gotta just prepare a package, right? You prepare a package for the different reviewers, whether it's legal or the risk team or the technology team.

‍

[00:12:28] Dave: And so in that way you're trying to streamline the process. You're pulling in the right teams at the right time to try to, again, ease that process as much as possible Who are you typically selling to in these organizations? Yeah, so we're typically selling to, at the end of the day, the person or persons that are accountable for AI, right?

‍

[00:12:48] Dave: Mm-hmm. That are accountable for what's happening, decisions made- Accountable for accountable AI, yes. So the person... and I say it because that role is different- Yeah ... within each environment, within each, sorry, enterprise. It could be a chief AI officer if they have one, a chief data and analytics officer.

‍

[00:13:05] Dave: Could be a CIO in, in certain cases as well, if they are the ones that are accountable to the, executive C-suite around the use of AI. But ultimately, that's the main person just to ensure that we can actually safely scale AI out to all the different teams. That's obviously the end goal for these large- Mm-hmm

‍

[00:13:23] Dave: enterprises. Let's get AI into the hands of everyone to help to improve various aspects of the business. But they can't do that until they know that they got the foundation in place. They know that if John over in accounting de- has an AI idea and wants to use Copilot Studio, that it's gonna go through the process.

‍

[00:13:42] Dave: It's gonna go through that engine that says, "All right, we check, check, check, check, check. Came out, everything looks green. Now we can use it." That's really hard to do at the enterprise when you've got accounting that's, you know, not- Mm-hmm ... very technology or, risk, knowledgeable, if you will, no offense to them, versus somebody that is potentially part of the data science team, super technical and pushing the bounds of it, but they aren't, they're not experts, they're not subject matter experts- Mm-hmm

‍

[00:14:05] Dave: in risk and policy, right? So how do you make sure that all the different teams, technical, non-technical, are able to effectively go from idea to usage in a way that's scalable and also, of course, responsible back to the organization- Mm-hmm ... that we're following those corporate policies? How do you do that?

‍

[00:14:24] Dave: Yeah. That's the that's the, the reason we exist, to be quite honest and overall, Kevin. So we do that with a couple of key pieces. first and foremost is that we do have a very, comprehensive system of record, we call it an inventory, obviously- Mm-hmm ... of just so you know every single AI- 

‍

[00:14:41] Dave: that's out there. That could be AI, could be ML, could be regressions, could be vendor. It doesn't matter to us. We took a lot of time to design how do we make sure we encompass and be able to track every single AI system. Of course, with that, we have a discovery capability to help organizations go and find what's out there.

‍

[00:14:56] Dave: But then the real secret sauce from there is what we call our, our Modelop AI Delivery engine, or MADE, and that's what takes, as we started the conversation, those policies and turns those into enforceable workflows so that you can, it doesn't matter if you've got 40 different flavors of how you need to do the different steps in the process, but you have the ability to go and say, "All right.

‍

[00:15:21] Dave: If an idea comes in, here's the workflow that must be followed." Those workflows will pull in the right people and the right systems at the right time to make sure every- check is done, every tick in the box every analysis, any evidence that needs to be captured, all of the, as I said, the risk tiering, the automated testing, the triggering of security scan, all of that is check.

‍

[00:15:43] Dave: Mm. And really the powerful part about it, and the most important part, is that there's actually not one process to rule them all, right? Mm. There's actually a variety of different processes. It's based on risk, like we talked about, high risk versus low risk. Mm. High risk has a lot more involved versus a low risk.

‍

[00:15:58] Dave: There's also different processes for, say, customer-facing versus internal, or external, you know, third party purchase SaaS-based AI versus internal. Mm. As well as in target environment. Is this gonna be running within our network, or is it running within a, a partner network, or maybe it's in a cloud? So there's a variety of different processes, and that's what, again, we took a lot of time to say, for the Fortune 500, we have to accommodate all those different scenarios.

‍

[00:16:25] Dave: Mm. And so what happens is the, again, back to your original question, we take their policies with all those variations, right, the 30, 31 flavors, right? And then can turn those into actionable workflows that are really AI and model context aware to know, okay, it's high risk- Mm ... go down this path, and then make sure every step is followed.

‍

[00:16:45] Dave: It's low risk, go down this path. Mm. Oh, it's low risk and it's cloud-based, okay, well, we know what to do here. So it's that kind of delivery engine is what we call it, that turns that, again, very manual paper-based policy that- Mm-hmm ... often spreadsheets, emails, super ad hoc- Mm-hmm ... into something that is scalable, repeatable, consistent, and of course auditable and traceable to make sure- Mm

‍

[00:17:06] Kevin: you know, no stone is unturned, if you will. And just to, to drill down a little more in where you started, the, the system- Sure ... of record. Yeah ... you know, makes sense, seems obvious that you need to know what AI you have if you're going to govern it, but we're talking about large enterprises, all kinds of things going on, shadow AI, people bringing things in, third party.

‍

[00:17:25] Dave: What are, what are the big challenges and, and the keys to building that foundation effectively? Yeah, I think the, the, the first one is just making sure that you have a system of record, to be honest. But then from there it's, th- then making sure that you can tap into the various sources, that you can discover where those AI, shadow AI, and other components are.

‍

[00:17:50] Dave: So that means that you can tap into the procurement system. So for some reason they purchased some SaaS-based AI that, you know, they didn't report back to the governance team. God forbid. Hopefully they don't. But- Mm-hmm ... you know, being able to tap into the different accounts. So you think about your Azure, your Google, your AWS accounts to make sure you can go and scan, hey, out in AgentCore right now, I see 10 agents.

‍

[00:18:12] Dave: Five of them are registered, five are not, right? You know, being able to tap into your Copilot Studio, your Agent 365 on the Microsoft side, et cetera. But then also being able to tap into, at the network level, which, so going way down in the stack, right? To determine, okay, well, we look like we actually detected some MCP traffic, if you will.

‍

[00:18:32] Dave: The common security thing to do is just block it. If you don't know it- Mm ... you block it, right? which is what they should do, right? But then from there you want to say, "All right, we detected this traffic. Maybe we do want to put this in the queue to get approved," right? So that, again, you're kind of taking at multiple levels.

‍

[00:18:46] Dave: At the network level, the kind of account level for cloud and other type of- Mm ... platforms, if you will, as well as maybe even tapping into the procurement system. So now I'm, and I make it sound easy, but as you can appreciate in a large enterprise, there's- Yeah ... some red tape involved in that. but that's the way that you get into kind of full scope and understanding of here's all the different places AI might be used.

‍

[00:19:09] Kevin: What do you see about the organizations that are more effective in implementing AI governance versus the ones that struggle? Yeah. So I think the first is just top-down support, and I know that sounds so fundamental and so, "Hey, Dave, that's, like, what a first-grade answer would say." But it is so, it is so true because there's so much, focus.

‍

[00:19:29] Dave: There's a lot of, different leaders or different technologists that are kind of chasing the latest and greatest technology. And so there's a lot of emphasis and a lot of focus on, "Let's just go and find the best LLM," right? "Let's go and build some bunch of agents." And so that's kind of the-- unfortunately gets too much attention from leadership, and then you forget, "Okay, well, that's not gonna be effective.

‍

[00:19:52] Dave: If we can't get any of them over the line, it's not gonna be effective." I.e., you need to make sure that you have the right governance, and as I like to say, governance can actually be an accelerator, doesn't have to be the brakes. So it's, it's first is the top-down to say, "We're going to do this. We're gonna build these different AI solutions, but we're gonna do so in a way that has governance embedded in about how we do delivery."

‍

[00:20:12] Dave: You can't have one without the other. And so that, for me, that's the first one. The ones that do it the best have the top-down support, and then obviously the different program teams underneath it to have that tight interlock- Mm-hmm ... between AI development, delivery, and governance overall. So that's number one.

‍

[00:20:28] Dave: the second is I'll just say, really as a kind of a lesson learned from enterprises out there is that you can't just use your traditional approaches to governance. Kind of you think your traditional GRC type approaches, which were designed for a whole bunch of corporate reasons, right? Mm-hmm. So you could...

‍

[00:20:45] Dave: You know, there's physical, there's access to the location itself, right? There's a whole bunch of policies, but AI is its own beast, right? It's constantly changing. Mm-hmm. It's related to data. It's a, a, a unique combination of software, data, risk that's inherent because it's not deterministic, right?

‍

[00:21:01] Dave: So trying to take what has been done in the past and apply it to AI just leads to a lot of- ... slowdowns, a lot of failures on the delivery side, if you will. So a- again, the best ones think about, okay, this is fundamentally distinct from what we've done in the past from a governance perspective.

‍

[00:21:19] Dave: Mm-hmm. And second, it's fundamentally distinct from the development and execution side of the house- Mm ... a- as well as distinct from security. And I'm sure you'll have questions, and we can get onto that, right? But these are different disciplines, right? You develop and you, a student cannot grade their own papers, right?

‍

[00:21:35] Dave: So developers cannot grade their own papers, right? So governance and developments are distinct. Security is focused on a different area, if you will. So again, back to your original question, it's, it's a recognition that these are distinct disciplines that are interrelated and need to work together, but they need to be there, you know, again, in a distinct basis overall.

‍

[00:21:54] Kevin: Mm-hmm. You made the argument that governance is actually an accelerator, and I've, I've made this argument as well, and I've had conversations like this with a number of guests, and so I, I believe that's true. But I'm really curious in practice- Yeah ... how do you make that case and where do you see, evidence in companies that that can actually happen?

‍

[00:22:11] Dave: So just based on what we've done with our customers- over the many years. Before walking into our customers, we did some analysis, and the different Governance and risk activities generally were causing anywhere between 40 and 70% of the delays- in what I've talked about from the beginning, the idea to actual production usage.

‍

[00:22:29] Dave: That's a pretty big range. So if you think about, hey, you've got this pent-up business demand for this AI solution that's gonna deliver X dollars in benefit- and the governance tasks are the things that are slowing it down to the point where, in some of our customers before us, it would take them between 9 and 12 months to get a- an AI or ML solution out the door.

‍

[00:22:48] Dave: Right. By the time you get it out the door, it is deprecated, right? You know, it's, it's- Mm-hmm ... it is fundamentally, not going to provide the business value that you would expect overall. So that's where we start, is just saying, "Okay, let's look at where it is today." And Kevin, to your original question, a lot of it is the- 

‍

[00:23:03] Dave: manual processes, the ad hoc, you know, cycles that are involved. You know, you're a developer and you develop something wonderful, and then you go and you say, "All right. Well, I'm ready to push a button. Let's get it technically deployed." And they're like, "Whoa, whoa, whoa, whoa. Did you talk to Johnny?" "Oh, wait.

‍

[00:23:17] Dave: No, Johnny says you're using, you know, uh, uh, uh, PII data." Then you gotta talk to Jane. Jane says, "Oh, nope, you gotta talk to legal." So then you get this runaround, and you're just kind of a ping-pong nature of going out to the different teams, and nobody has that fundamentally understanding. They don't have the blueprint at the end of the day of how do you go from idea to production.

‍

[00:23:34] Dave: It's a constant, you gotta go and figure it out yourself, read the policy document, et cetera. So that's the way that we help to become an accelerator. I'll go back to that case of the- Mm-hmm ... before we came in, it was a large financial, it was between 9 and 12 months to get an AI and even an ML solution out the door.

‍

[00:23:52] Dave: And that was because, as I said, there was just not a cohesive collaboration across the teams. There wasn't a documented process across here's the different steps involved, here's the different teams involved, the different approvals, the evidence, et cetera. And so we, with our, product overall, but first and foremost, lay down here's the blueprint.

‍

[00:24:09] Dave: Here's the playbook, right? Here's the playbook across all the different teams, the different flavors, the different plays. They use a football analogy. You got your playbook, and you got a bunch of different plays. Those plays, as I said, high-risk play versus low risk, right? You know, cloud versus internal.

‍

[00:24:22] Dave: So you develop that playbook, and then you obviously Modelop is great about taking that playbook and enforcing it, using it with automation, putting those processes as live orchestrations, et cetera. So in that way, we took that 9 to 12 and reduced that to under three months. And that was purely because they had the repeatable playbook.

‍

[00:24:39] Dave: They had the, streamlined understanding of who needed to be, doing different approvals, reviews, testing, et cetera, and orchestrating as much of that as possible via automations and integrations. Mm-hmm. So that, that's like, again, why both you and I agree that it can be an accelerator as, as it's proven through some of our case studies with our customers thus far.

‍

[00:25:00] Kevin: speed to market, is one way companies can think about how effective they are with AI governance, whi- what you've talked about. What are the other ways that, that managers in these enterprises, are able to evaluate how they're doing? Yeah. so I think the, there's a couple ways. Speed to market, obviously, from a business perspective.

‍

[00:25:19] Dave: The, the second is that the demand for AI, especially with gen AI and now agentic, it's off the charts, and I'm sure, you know- Yeah ... you've talked about this with some of your guests before. Sure. So if you think about it from the con- I'll just say control partner perspective, the legal risk, compliance, governance, et cetera, they're staring down a gauntlet.

‍

[00:25:38] Dave: Oh my gosh, we're gonna have to go and review, you know, 1,000, right? Or, we have one customer that's, you know, in the many, many thousands of different AI solutions. So they're staring down this gauntlet. What do I do? Like, there's no way I can keep pace with this. So the, the second reason is actually they're faced with two choices: I go and scale up and have a massive team to handle this demand, or I turn to, you know, software that can help us to keep pace with this demand.

‍

[00:26:03] Dave: So yes, time to market, but second is, can we have that pace of innovation without massively increasing the cost, the human cost, the human headcount from that, again, control partner perspective? So that's a second reason to do, you know, have a, a effective, a responsible, accountable AI program. And then the third, of course, is just why it exists in the first place, protect the organization, right?

‍

[00:26:28] Dave: Yeah. So protecting the organization from, you know, undue risk, from potential brand exposure, hopefully that never happens. Obviously, any audit or regulatory concerns, but at the end of the day, you just wanna protect the organization from something going wrong. It will go wrong, but how do you protect it from catastrophic wrong, if you will?

‍

[00:26:48] Kevin: Yeah, but, you know, how do executives assess that? Because if, if the catastrophic wrong hasn't happened- Yeah ... then you're proving a negative. And, and clearly legal and compliance matters a lot, but- Mm-hmm ... there are all these gray areas. and, you know, we've got the EU AI Act and we've got the, you know, GDPR and different data protection requirements.

‍

[00:27:06] Dave: And, and, and those at least it's, it's concrete what the obligations are. But there's still a lot of these areas where, for most of these organizations, there's not a definitive legal mandate or regulatory- Yeah ... mandate they have to follow. How should they think about how far to go?

‍

[00:27:22] Dave: Yeah. So, again, first off, we always hope they do the right thing and protect the organization and many leaders will do that. second, that their AI will go wrong, and there's plenty- Mm-hmm ... of examples out there that you've seen where, I think one of the first ones was that a chatbot, you know, responded to an airline passenger and said, "Oh yeah, sure.

‍

[00:27:40] Dave: That's fine. You can get the, uh, the grievance rate for, you know, the funeral you have to go to." So there's actual financial implications should AI go wrong, around it. But then third, I would say of why, a- again, a really a catalyst to take this seriously is actually the spend, believe it or not. Yeah. So the spend that is starting to come about because of just the token maxing and just in general the variable nature of how the different frontier lab providers are doing this and changing kind of their variable rates, the spend is growing quite significant, right?

‍

[00:28:16] Dave: And so it's not just a regulatory bit, it's also a financial and operational governance, if you will. So you can, in, in a way, if you have the proper, governance in place, you're not just looking at did I, as I said, do all the reviews, approvals, regulatory compliance, et cetera, but you're looking at what's it costing us?

‍

[00:28:37] Dave: Yeah. What's the value we're getting out of it? Does this make sense, right? Mm-hmm. And so having that level of financial due diligence, just to have that, again, fiduciary duty back to your company, does it even make sense to continue on with this AI solution because the cost is X- ... value TBD, right?

‍

[00:28:55] Dave: So i- in that way, it, it is something that I'm seeing. I'm sure you've heard, Kevin, with some of your conversations and what you're seeing. That is being a ca- that is a catalyst. Mm-hmm. It's a, it's a big catalyst to- Yep ... say, "All right. It's time to rein it in. Let's get a little bit of control around this Wild West of AI," just purely from a, a fiduciary perspective.

‍

[00:29:14] Kevin: And then how much does the rise of agentic AI change what you do? it's been exciting, Kevin, from our perspective, it's been super exciting. It doesn't change our technology because we designed our technology from the very beginning to be able to support all types of AI, both now and in the future.

‍

[00:29:31] Dave: So when LLMs came out, we added it overnight 'cause we just designed our technology. So it hasn't changed our technology. What is it changing again, why it's exciting, is because agentic AI really has been the catalyst that's pushed over the edge these enterprises that said, "Nah, you know what? We can just get by with what we have with our responsible AI program.

‍

[00:29:52] Dave: Yeah, we can just continue on with SharePoint and spreadsheets." That doesn't work with agentic AI. There's too many. It's too much. There's too much that could go wrong because now the agents can make decisions on behalf instead of just informing employees of what decisions could be made, but they're making decisions on it, which could have real actions.

‍

[00:30:12] Dave: And even if they say, "Well, I don't want it to make decisions," there's still the opport- there's still the chance that the agent does it anyway, right? Which we've seen all those stories of, out in the, the wild about different agents that have gone rogue and even despite the directions not to go rogue, right?

‍

[00:30:26] Dave: So for us, it's changed in that, as I said, agents are so, in such high demand from the different business teams that it has, as I said, pushed over the edge to say, "You know what? We need to fundamentally do something different here. We need to, have this as a, a comprehensive program and not just an afterthought Yeah.

‍

[00:30:46] Kevin: And we're almost out of time, but that's why... so h- how can agents though be effectively governed? I understand we're talking in the enterprise, we're not talking about, you know, unreleased frontier models in testing at the labs. But there is still this problem that, you know, we have this alignment issue where you can tell the agent something, but it may still do something else.

‍

[00:31:03] Dave: Yeah. So there's a couple levels as you probably, well know from a control perspective. And the first thing on the controls is just limiting the usage and making- ... well, first defining the usage, and then defining what are the limits of that usage, so that you have an understanding, okay, hey, this agent is designed to do this.

‍

[00:31:20] Dave: Here's the autonomy that we wanna give it or not give it. Here's the limits of that autonomy. Here's where we need human in the loop, et cetera. And then making sure that, again, those limits and usages are, part of how the agentic system is designed. Then second, you'll go down a level to more the system level controls, and you get into place in making sure that especially the tool and data and system access is well understood, and that there are appropriate low level controls in place to limit it to only those tools, data, and systems that you have defined that are possible.

‍

[00:31:56] Dave: Mm. And that's where you, need to get down into the execution, and potentially even the network layer, because you're defining, hey, we don't want it to reach out to the internet, we don't want it to reach out to MCP servers, et cetera. So it multiple levels, as you can appreciate, Kevin, from 

‍

[00:32:09] Dave: again, the limits and the here's the usages, down to the platform level of here's type of access and integrations that are allowed, and then all the way down to the network level of, okay, now let's enforce that, especially for don't let it outside this network, right? if the network level is saying nope, can't get out there, then it can't do anything.

‍

[00:32:27] Kevin: So all of those kind of- Mm ... examples in the wild that we see- Wow ... wouldn't be possible if the appropriate network level, uh, controls are in place around it. So again, multi-layer. Got it. All right. Well, as you say, it is an incredibly exciting time. and, I am glad to have the opportunity to talk with you about what you're doing at Model N.

‍

[00:32:44] Dave: Thanks a lot, Dave. Yeah. Thank you so much, Kevin. This has been The Road to Accountable AI. If you like what you're hearing, please give us a good review and check out my Substack for more insights on AI accountability. Thank you for listening.

‍

[00:33:01] Kevin: If you want to go deeper on AI governance, trust, and responsibility with me and other distinguished faculty of the world's top business school, sign up for the next cohort of Wharton's Strategies for Accountable AI online executive education program, featuring live interaction with faculty, expert interviews, and custom designed asynchronous content.

‍

[00:33:21] Kevin: Join fellow business leaders to learn valuable skills you can put to work in your organization. Visit execed.wharton.upenn.edu/acai for full details. 

‍