AI Initiatives: Demand, Challenges and Architecture

Krish Palaniappan: Hey folks, welcome to Snowpal Polyglot software development podcast. Our guest today is Dave Trier. Uh Dave is the VP of product at ModelOp, a leading AI life cycle automation and governance software for enterprises. Uh Dave, thanks for taking the time to chat today.

Thank you.

Dave Trier: Pleasure to be on this.

Krish Palaniappan: uh if you could give a brief introduction about yourself and your company to our audience and then we can just get into the topic.

Dave Trier: Sounds great.

Krish Palaniappan: Excellent.

Dave Trier: And again, thank you so much for having me. My name is Dave Trier, VP of product with ModelOp. As Krish mentioned, we are a software company that specifically focus on helping large enterprises overcome the challenges as it relates to AI life cycle management and governance. We focus mainly on the Fortune 500, really helping them get AI solutions to market faster with the right level of oversight and assurance throughout the way.

At the end of the day, just helping organizations use AI at scale with the appropriate level of trust. Sounds good.

Krish Palaniappan: And I have that uh your company page pulled up here. Did I pulled up the right one? Right.

Dave Trier: You did. Yes. Thank you.

Krish Palaniappan: Okay, perfect.

Dave Trier: Yep.

Krish Palaniappan: and I can I we'll include the link to your company page and your LinkedIn profile on the podcast so folks can uh can check out you know the company and your products and solutions. Um so without further ado we can jump into the topic uh and what you had recommended was a great place to start which is um why the topic in itself is why enterprises struggle to implement AI initiatives but that being the topic Dave we can start with where it makes the most sense for you.

Dave Trier: Yeah, absolutely. Thank you so much. I've had the pleasure of working with large enterprises actually my whole career over 20 years. Uh but in particular for this latest so six years with ModelOp on how they can take advantage of it but most importantly how do they actually take those AI ideas and put them into actual business usage.

Now what's been really interesting throughout this whole journey or the journey with our customers and enterprises at large is that they have a large pent-up demand for AI especially around generative AI that every aspect of the organization every department every team has been able to take advantage of generative AI or wants to take advantage of generative AI in one part or another. But what's really been challenging to them is well how do I get those solutions to market faster especially with generative AI which has been something that's a bit opaque to many parts of the organization but just first starting on the demand side I always start there is that okay well technology is great right Krish technology is fantastic but unless it's actually going to move the needle on my business whether topline or productivity savings or bottom line cost and operational savings then technology is worthless, right? So, it al but it starts with the demand. And what's been really fascinating about this space, I've been doing the data and analytics space for over a dozen years now.

But what's been really fascinating is in the past 18 to 24 months with generative AI is that they have recognized every department every team has recognized here is a actual huge opportunity for us to leverage a technology to change the way we do customer service or operational processes or internal you know as on a customer and how we help with answering common HR and compliance questions right so every aspect of the business has the opport opportunity to use generative AI to make improvements to how they're doing it. So in that way it's been as I said a really pent up demand around how do I take advantage of this very transformational technology for all of my processes every aspect of my business. That's been the part to be honest Krish that's been really exciting is that it's not just a technology in one silo of the business but it's every team every department has an opportunity to use this technology

Krish Palaniappan: you know that's a great introduction thanks Dave so let me ask a couple of questions uh and I've had recent a few conversations recently with other founders uh around this topic as well and some of these questions stem from what you said and also from my learnings from these prior conversations You mentioned uh demand, right? There's no point using technologies unless it's actually providing meaningful business value. Mike, my first question to you as it has been to a lot of the other podcast guests in the space as well is uh do you are you are you looking for problems that you could potentially solve using AI or are you looking to use AI to solve your existing problems differently? And I can repeat the question if it's it's not clear.

Dave Trier: No, it makes sense. It's it's actually a combination of both. There are certainly existing problems that customers of ours, enterprises at large have, but there are also ones that there are new opportunities that the technology unlocks. So, it's actually a little bit of both.

I'll give you an example of the first one, right? So, again, I was on with a very large customer this morning and they have in the past they had to a team that was having to field a whole bunch of compliance questions, right? So that internal employees had a question about the travel policy or they need to look at or they wanted to understand what's the time off policy, right? And in the past that was something where they would have to have a team of internal support staff that were answering these questions, having to go and look it up in over 30,000 different compliance documents.

I kid you not, over 30,000 compliance documents to make sure they get the right answer. Fast forward. So that's an existing problem, right? And then fast forward now with generative AI is that with a generative AI rag architecture, they're able to use the 30,000 policy documents and have an a foundation model, an LLM, help to answer some of those questions, those common questions that internal employees have.

So that's just an example of how you can apply generative AI technology to an existing problem. Now you go on the other side as I said to the other the flip side is well are there new ways that generative AI can unlock the answer is absolutely yes that there are uh things that you couldn't have done in the past that you are able to do with generative AI. So again to answer your question I would say it's a little bit of both.

Krish Palaniappan: Okay let's go with your example there you mentioned 30,000 documents and those were I presume in your example there they were internal documents right in other words they were private to that particular organization. So prior to the days of generative AI, uh the team would have to have some sort of training to be able to read or understand some of the documentation and at least be able to find the rest of the documentation. So when somebody's asking them a question real time, they are able to go quickly and find those answers, right?

Dave Trier: That's that's is that's correct.

Krish Palaniappan: Okay, that's a fair assumption. Y now with those internal documents, you mentioned the rag architecture. That's something very interesting as well. you know if you can talk a little bit towards the retrieval augmented generation which is essentially we're saying the LLMs are not going out there into the public space to fetch answers they are but before I actually finish that question in your example here was were those answers to be found within one of those 30,000 documents or was it a combination of hey let's find some part of the answer in these internal documents but we may have to go outside that space to maybe the public uh domain to complete those answers says, "Oh, is it limited to just those documents?"

Dave Trier: Yeah, primarily for this use case, they tried to get the answer from the 30,000 documents, right? Because they're a question about policy, right? So, they should be, I would hope, in 30,000 documents, you got the answer to a given policy question. So, they focused on putting those 30,000 documents into a vector database, standard rag architecture, put them into a vector database, go and try to look up and find the answer from there first.

Now what was so to answer your question yes they try to use those internal policy documents then what as they're finding as they're starting to use the system employees are using this say oh okay well actually we're seeing some of the questions you know what we need to augment some of those the 30,000 documents with some additional information about policy that aren't in the 30,000 but they're still looking up internally they're not going out to the internet and therefore trying to open it up to potentially wrong answers they do try to focus for this use case at least in the particular policy documents and then some of the augmented u knowledge that they have around those policies as well.

Krish Palaniappan: I think that's a great example. So I want to dig a little bit deeper because that example is not good for many reasons. One is it's not only easy to understand for anybody watching or listening to our conversation here. Uh but also something that is probably generic enough so other people would have similar requirements for whatever it is that they're trying to implement.

Um so let's say again the use case here is people there had to be a team of humans uh to support this request. People had questions about you said travel policies within the company and whatnot because it's a large enough organization. Now using uh genai and in this example the rag architecture those answers the first of all you know let's say you had 10 people I want to sort of incrementally find my way into my end question here. Let's say you had 10 people who had to be trained, uh, hired, staffed, trained, yada yada yada, to actually, uh, be put in place to be answered these, to answer these questions using, uh, this AI technologies.

I presume you still need somebody. Is it two people instead of 10 or is it eight people or is it six people just in terms of percentages?

Dave Trier: Yeah, that's some of the information I probably can't share, but you can say that it's uh significantly smaller than than it would have needed. And it's not just that you have to train them one time. As you can imagine, policy evolves um a lot more than you would think, especially in a large organization with 30,000, you know, policy documents, right? Uh so it's not just the upfront training, it's the ongoing training of, hey, this policy has been updated.

Now we got to go and train our support staff to go and be able to answer that andor we have to update the internal legacy search I should say as well to try to make sure that this gets into it. But in short yes it does it substantially decreases the amount of uh you know support staff that you need for this particular scenario. Okay.

Krish Palaniappan: And then you know that again I'm asking these questions it doesn't have to be for that particular client. I'm just saying these questions are generic. So more the answer that would help our audience is how they might be able to solve a similar problem right just just just to give context there. Uh back to this architecture how does this rag architecture look like?

If somebody wanted let's say there is an organization that has a similar problem where uh it doesn't have to be for these types of questions but for whatever other reasons they have a few people humans doing this work and they want to be able to go to AI technologies using rag and whatnot. what should that how does it look like where do they start and how do they go like if you can speak to some of this architecture at a high level and to whatever levels of detail you could go to that would be great just in terms of are they using AWS which products within AWS or some of the hyperscalers and just some technical context to this problem uh to this solution

Dave Trier: sure yeah I'll just give a little bit of a high level then maybe we can segue into how do you so there's the architecture right and then segue into how do we help to ensure there's the right oversight and across the full life cycle of this AI solution if you will. But from some from an architecture perspective, I would say before you even get into architecture, it's just making sure that you are clearly and concisely identifying here's the situation that we're trying to solve. Right? This one is very simple.

That's why I picked it because it's easy to understand. You have some questions around policy. You had a whole bunch of documents on policy. How do I get an easy and quick answer with the appropriate references?

Right? So first starts with making sure you put in the in scope if you will put some some rails around what the problem you're trying to solve. Then as you get to architecture yeah there's a couple components. There's a foundation model and this can vary just depending on the organization.

You may use a GPT-4o. You may use one of the foundation models in AWS bedrock which could be an anthropic or it could be a Llama. It you know you pick the right model based on your specific situation and some of your partners that you go with. So you have the foundation model which is one of the core right then you have as I mentioned before the vector database which is just a database that helps to well you vectorize all the documents.

So you take all those 30,000 policy documents and you put them into a database that allows for quick reference and typically that's the most efficient using a vector database. The third piece that you have is typically what's called an orchestrator. Think about it. It helps to stitch together the full application if you will from I take a user that has a question.

They type in question. All right. What do I do with that question? It turn it into a prompt if you will that then sets off to the foundation model.

Well, first looks up with the vector database. Looks up to can I find an answer there? Sends it off to the foundation model gets an answer and then gives the response back after some post filtering. Right?

So you have that orchestrator that helps to put put together all the different steps as part of that process. And then finally you probably have a user interface as you can expect of just allowing a user to type in their question and get a response back. So that if you think about that those are some of the high-level components that are part of a typical scenario like this

Krish Palaniappan: and you explained them beautifully. So thank you uh uh you know those components. So let's take them really quick one by one. The first one is the foundation model.

I know you mentioned you know whether it's AWS or you could go to bedrock you can find lama. one of the models that make the most sense that is you know from by directly by experience and by talking to people Dave that's one of the starting points where a lot of people get stuck because there are just so many options that are out there. uh how do you go about like in other words do you have like a model or two or three that you fancy that you're like that's your go-to model or do you actually have a you know you start on a clean slate saying you know what I have no biases and I'm going to look at the problem and then go do my research as to which model I should pick uh regardless uh what how do you go about picking the right model because that is can be a bit overwhelming to find the right model to start off

Dave Trier: Yeah, just to be clear, our company ModelOp, we're agnostic. Uh we're agnostic to the foundation model, the technology, if you're AWS, Azure, GCP, Oracle, it doesn't matter to us. We're we're agnostic. We uh our company is focused on regardless of the technology, use the foundation model, the situation, we're just helping you to oversee the full end-to-end life cycle.

So, I just wanted to be clear on that. Sure. But to answer your question, this is something where it is just dependent on the scenario. Um whether it's you're focusing on textbased or kind of a chatbot based uh you're focusing on more summarization, maybe you're doing an image generation as part of marketing campaign.

So you know there are different models that and you can go look at all the published studies around those that perform better in different situations. So my general advice would be again define the problem and what you're trying to solve. go and look at some of the recommended best choices for that specific problem area or problem set and then you know experiment with a few of them. Now if you set up your architecture in a way that you have your orchestrator and your vector database you should be able to do it because we've done it before you should be able to just switch out the foundation model behind the scenes and then you can do some sideby-side comparisons.

If you look at any of the benchmark studies around foundation models, that's what they're doing, right? They're setting up here's my problem. Here's the application, the orchestrator, vector database. Let me just switch out these two and see how which one performs better.

So, long-winded way of saying is that define the problem, figure out what type of problem set it is. There's some good recommendations for each of those different problem areas, and then just test it, you know, but design your architecture in a way that you can flop out, swap out, I should say, the different foundation models to get some good results.

Krish Palaniappan: Okay, lovely. Right. So that's actually a very good great answer. What you're saying is have you know as a developer as somebody who runs a startup I tell people that when you're changing something you want to change one thing at a time so you know what the impact of that change is.

You know a lot of times even when writing code as developers you know when trying to fix a bug you try to change three different things and you don't know what's causing the problems. You want to do them iteratively. So your answer here is very much along those lines which is keep the rest of your architecture in place and maybe swap the models out so you know how it's performing assuming ceteris paribus conditions with regards to the rest of the stack. Awesome.

Yeah, that's that's right.

Dave Trier: And just one thing that we help a lot of our customers with is that as part of the that end-to-end life cycle of the models, we really think about use cases. What's the business problem I'm trying to solve? So in this case it was, you know, a chatbot for common compliance questions. We have the use case.

Separate from that, we have the model. And especially with generative AI, we actually have a process to help our organizations, our customers to onboard foundation models because they have certain ones that they approve that make sense for them based on the security profile, their actual kind of contractual or commercial agreements, if you will. Um so what we do is we help them to onboard those and have a catalog of here's the approved foundation models so that these enterprises the developers in the enterprises can say okay my use cases is again a compliance chatbot what is available and approved for me to use oh I can see in the ModelOp you know catalog of foundation models these are the ones that have been approved I'm good to go with going and trying to experiment with a couple of them as I mentioned before so I just wanted to throw that concept um out there that's a it's a very practical way for large organizations to narrow some of the choices down to the ones that have been vetted approved when when we're talking about foundation models. Makes sense.

Krish Palaniappan: And then the second thing you mentioned in the architecture was the vector databases. Uh one of the things that I have looked at I actually had a chat with the founder of this company based out of Netherlands if I'm not wrong. It's called Weaviate. That's one of the vector databases that I've actually looked at a little bit more than others.

like in your example like or your experience, what are some vector databases that you actually like that you could you would recommend that people check out?

Dave Trier: Yeah, again we're we're agnostic to that. It's it's um for our customers it's their preference commercially um what they've what they've defined that works within their ecosystem and stack. So I honestly I don't have a strong preference on that one.

Krish Palaniappan: Okay, no worries. The only the reason I ask is at least for people who are watching to understand is there are databases that actually have added support for vector design like is one of the databases that we use at Snowpal the time that is a NoSQL database but it is not built you know it was built as NoSQL but they do have support for a vector database component if you will uh it's much it's along the lines of you know Oracle was you know an RDBMS but I remember at some point they had some support for NoSQL But when you use databases that were created for one purpose, but they actually try to solve the more recent use cases, they I mean it just depends whether you like them or whether they're you know it makes sense or not. So I'm just calling out for it saying my recommendation, personal opinion is you want to pick one that was designed from the ground up as a vector database as opposed to one that was built for some other purpose but also has support. But again, this is just my opinion that's not to say that you can you cannot go with and use their vector database.

You know, their support for vector databases as opposed to using something that is built from the ground up. But I've I've seen as somebody as a developer that the experience of using one versus the other is actually quite different. Okay. And then the third part, you mentioned the orchestrator.

Dave Trier: Are those when you say an orchestrator that actually deals with all of these is it is it software that's interacting with uh you know is it the actual the software tier uh that's solving this problem a bunch of different pieces that are working with the vector databases and the models and putting all of them together the core backend piece if you will if I imagine the orchestrator as that would that be correct or what would the orchestrator be exactly in this case yeah it's it's the code that helps to stitch together the different pieces, right? If you think about UI, you got to accept the input from the UI. You got to, you know, go uh call out using typically an embeddings model, right? And call out to the vector database.

You then have to go and pass that information off to the LLM, the foundation model itself. You got to have the response. You likely are filtering out some of the responses if they're not great coming back from the LLM and then present it back to the UI. So think of it as the code that helps to pull together all those pieces uh you know across those different steps in the process and make sure that they all run smoothly if you will.

Okay.

Krish Palaniappan: And I'm pretty sure your work is language agnostic. We are a polyglot shop at Snowpal. But that being said, I know when you deal with machine learning and AI and all of these tools. Some languages are, you know, better for whatever reason like Python for instance, you know, we use Golang and a bunch of other languages at Snowpal, but we've realized at least it's our opinion that Python seems to have support, you know, the support for plugins and SDKs for Python when it comes to AI.

Is that your experience Dave or again do you see support across languages?

Dave Trier: Yeah, we see them across languages but uh yeah, Python obviously and has been in the data science world the most preferred language uh for a while now. Um but you know again there's support for other languages as you rightfully pointed out and so it's just really a preference for you know your your enterprise as well what's the what's their preferred language but Python you're right is pretty prevalent has been for many many years now in the data science space. Sure.

Krish Palaniappan: And then the last part of the stack that you mentioned was the UI. Uh you know uh do you see user interfaces change where from where they are today? In other words, today these interfaces are built more traditionally. You log in, you know, you go click a bunch of buttons and you get stuff done.

But you talked about being able to search for whatever it is that you're looking for within those 30,000 documents. So do you see the future of web interfaces or any interface being very conversational in meaning being 180 degrees different from how we are used to interfaces today?

Dave Trier: I do see that starting to happen, Krish, that again because I own the product, right? So, even our own product that, you know, obviously we've built out a beautiful product and have all of the various menus and pages and, you know, the cards and all the graphs and charts and everything like that. But now with generative AI, we're we have introduced the ability to just allow for a chat and response. Go show me, you know, the all the models related to this area.

Show me the use cases that use generative AI. Show me the latest test results that are for this particular compliance chatbot. Right? So we are even within our own internal product moving towards an interface where it's more conversational of I have this question.

Can you just give me the answer instead of me trying to navigate through the different buttons even if that is a good user experience. But now people are getting a little bit more used to that conversational kind of give me the answer type paradigm if you will for better or for worse.

Krish Palaniappan: And then I have to believe we are in the early stages that because you know um I'm yet to see a product uh including our own at Snowpal. We I'm yet to see actually a product that I'm using that has switched over from a traditional interface to a conversational interface. Now that's my experience. Do you have you used any that you can recommend that we can check out to say okay here is how the future of web interface is going to look like.

Dave Trier: I don't think that's going to be when I say I think it'll eventually go there. That's not going to happen overnight, right? People are still they still want their traditional navigation because you know you want to make sure that you have a triedand-true method. Um so I would say that those are going to operate in parallel for a while.

Um, so I wouldn't say there's any thing that's moved over other than the purpose-built software that was as just a chatbot or a summarization, right? An AI product that was just meant to do that. Uh, so again, in short, I would say that those two user experience paradigms are going to op operate in parallel for a while. We'll see.

We'll see if it eventually switches over to just the conversational side. It, you know, time will tell. Sure.

Krish Palaniappan: And then maybe this is a segue to uh what are some of the reasons you think enterprises struggle to implement these initiatives because everyone has good intent right and I think a lot of people and companies and individuals have come to you know the realization that AI is not a hype and it's it's not fat it's here to stay and they're beginning to make these investments yet there are challenges because we are you know you probably a year or two in where everybody's talking about this it's still early stages so what are some of the challenges that you uh have encountered or you've seen other people encounter when they go to implementing these initiatives? Right.

Dave Trier: And as I said, we focus on the large enterprise. So some of the most common challenges with enterprises are really impediments that uh are in the way to getting to production and these are a variety of different ones. But it all comes down to there isn't agreement and therefore enough trust across all the different teams that these AI solutions aren't going to cause harm whether that's reputational, financial, regulatory, uh systematic, that sort of thing. And what I mean by that is that you have these AI solutions and even before AI, whether it was ML or statistical or just any data science analytical tools, there's a process to get those into usage, if you will.

For an enterprise, there's typically an IT and data and security. If it has any, you know, customer or other transactional proprietary information, you have some legal potentially uh risk type. So there's, you know, 10 different teams that are involved. Typically there's 10 different systems that it touches across the infrastructure the again security the data the uh you know looking at legal and risk and other um ITSM type systems.

So there's 10 different teams 10 different systems all of these just become impediments. Now what made it worse with AI is that these 10 different teams didn't have a great understanding around what AI is and didn't have a great way to understand well how do I know that this is not going to cause harm right again regulatory financial uh systematic harm etc. So the biggest impediment that we've seen is that these different departments and teams want to use AI, but there's not a consistent way to manage the end-to-end process that everybody can trust. And so that's why we started our company was just to have that consistent process, that consistent oversight, that legal and data and risk and security and infrastructure.

All of them can say, "All right, well, I trust that this is doing the right thing, that we've we've actually gone through the paces. We've put it through its paces. We've identified the risk because it's gone through this model process. So, I trust that we're good.

Like, I don't need to slow it down. I don't need to have a bunch of manual uh processes, hold it up for months at a time because I'm trying to understand everything." No, we've got the blueprint. ModelOp is that blueprint.

We know if it goes through that blueprint, then we can say, "Yep, I'm good on signing off." And by doing that, that helps to shorten all of that manual processing time, the ad hoc reviews, the ad hoc back and forth. It helps to shrink it by more than half. So again, for us, it's really about helping enterprises get those AI solutions to market faster, overcoming those different processes, technologies, people, other red tape, if you will, overcome those in a way that allows every part of the organization to trust it and get it into business faster.

So that's a long-winded answer, but obviously we're it's an area that we spend a lot of time.

Krish Palaniappan: So you know this question is more for both large enterprises and midsize businesses as well. Let's say there are a fair number of teams in any midsize or large company even in a mid-size company. Uh what is the general approach that you would recommend? Does each team should each team come up to speed to the same level a little bit at a time with AI and all the teams come up to speed at the to the same level and then they keep chugging along or would you is there an approach where you work with one team and get them completely ported over if you will from their existing systems to a genai based futuristic system and showcase that as an example to the rest of the teams because you know those are I mean there's many ways to skin the cat but I can think of two ways where you know you say every team should make a little bit of progress at a time so nobody's left behind that's got its own pros and co cons and challenges and the other approach you have one team do it all the way through and have that be showcased as an example if that makes sense Dave what it does what would be a good way to go

Dave Trier: it does and I think that's just where you need to take a look at your organization and the skills in the organization to be quite frank where if you have varied and mixed skills If across it, then it probably makes sense to have a focused team that has a lot of background, especially around the risks and understanding the risks and the potential impacts. Have them start first and set the here's the example, here's the frameworks, here's the approaches, as I said, here's the blueprint, if you will. Have them start first and then disseminate that to other teams. Now if you are a digital native company like a Google or a Facebook or Meta sorry then you probably can trust every team to just do it on their own.

But again if you think about the very large organization the large enterprise you do need a bit of introspect around okay do we have the skill sets across every single team to allow them do their own thing. That's not a question I can answer but it's something that I would advise that just have a think about that. And a lot of what we're seeing is that enterprises recognize, you know what, let's start this in an AI center of excellence as an example. Let's set the benchmark.

Let's set the here's the shiny example and then again kind of disseminate and train out to other teams and organizations. So that's a bit of a yeah bit of a mix. Perfect.

Krish Palaniappan: We can end our conversation conversation with agentic AI. Before we uh go to agency AI, I want to just ask one high level question which is you know when we talk AI and people are trying to get themselves up to speed which everyone has to uh I think the Fiverr CEO I read recently that he had said that all the easy I'm paraphrasing what he said he said all the easy jobs are going to be gone. All the difficult jobs are going to become easier with AI and all the jobs that were impossible to do are going to become possible. So you know I believe I read this somewhere maybe on LinkedIn where someone shared this it was for to his employees and to the freelancers on the platform as well right so all of us have to sort of buck up and then you don't want to be left behind.

Um so this question is a high level question which is when we talk AI and if people have to be like hey you know what I know little about this I want to go read up on these things what are what are some jargon or buzzwords I'm going to start with some and you can add to that list and then we can switch to the last question which is agentic uh LLMs like uh you know large language models are SLMs or small language models um those are things I can think of you mentioned vector databases very much so you could be doing AI work without vector databases but it's simply not going to be the same. So I think it kind of goes hand inand that's one to look at. Uh you mentioned uh conversational interfaces again when you're talking web or mobile the way you're going to interact with these tools is going to be quite different from how we've done in the past. uh you mentioned rag retrieval augmented generation which is if you're not going to go with data that's out in the public domain you want to take your internal private data uh and make it part of your vector databases and persistence layer you need something like rag in your architecture those are some I can recall from our conversation I'm sure I missed someday but what are there other jargon or buzzwords you can throw in here that you can look up

Dave Trier: I mean for buzz and jargon if you will one that uh I would add to your list there would be guardrails, right? So guardrails around which just explained to the audience that haven't heard of them before. Guardrails are think of them as the filters on what's coming back from say LLM or SLM basically to say all right well we want to filter out certain responses. The most kind of obvious one and LLMs and SLMs do a good job today but filtering out profanity, toxic language, that sort of thing.

But guardrails are important because you also need to have your own specific ones in place based on the use case. So for example, you may have a scenario where um you know you don't want it to return some confidential information or you don't want it to return PII, right? So you may need to filter those out. I the guardrails are helpful in that certain situation as part of it.

So guardrails I would say is another buzzword that would I would add to your list there.

Krish Palaniappan: And then I think the other two we can just talk about it at as the very last item here which is AI agents and agentic AI. Uh first of all you know when I think about them with very little experience I struggle to tell the difference between the two they seem like synonymous like AI agents and agentic AI. I mean is there like a difference? I'm sure there is like what is the difference?

What are the difference between the two?

Dave Trier: Yeah that's it's obviously a topic that a lot of people are have discussion around. Well, the way I think about it is that there's an overarching agentic AI system and that system has a number of components just like we talked about with rag and agentic AI solution has a number of components and one of them is the LLM or SLM, right? You have the foundation model, but then you have these helpers, right? They may be called uh agent tools is one of the most common that's used and using things like MCP and those are just think of them as helpers.

they're helping the language model to go and do some actions. You know, a common one might be, all right, well, uh, again, I'm doing a customer service, right, chatbot as part of it, and you're talking to this chatbot, and it may say, okay, well, I need to go and update my Salesforce record. Those helpers are there to go and call the Salesforce API, update a record, like, hey, I talked to, you know, Krish today at uh, May the 22nd. So those helpers are there to go and call the APIs uh on behalf of the LLM and you know get the response back and then the LLM or the chatbot if you will will say okay Krish great thank you for your inquiry we went ahead and updated your customer record I don't know your billing address right so think of the agentic system as the overarching piece and the LLM is really um you know helping to be the conversational piece but then you have these helper tools like the a they're actually called agent tools that will go and help you with whatever task that you're trying to accomplish.

But I consider that part of the overall agentic AI system with a number of components. Does that make sense?

Krish Palaniappan: It def it certainly does you know and just to add to that a lot of these words you know as somebody who's who's you know been building software much like yourself Dave for over two decades. There's only so much you can learn from the theory right you can read you digest it you understand it. When you start doing it is when you're like you know you can truly conceptualize the differences. And here's what I tell people.

There's no there's no singular single right answer in my opinion. But there are people who sometimes believe that they have to understand the fundamentals and the concept thoroughly before they can start doing anything. But that I mean if it works for you, more power to you. But for a lot of us, for me and for people I know, uh you want to start doing it incrementally.

You know, you will not be doing it right. You might be doing it wrong. It's all right as long as long as you start doing it. Doesn't matter if you pick the right uh vector database or not like you said right a lot of these are agnostic you know you want to be you want to treat them as commodities to some extent LLM is a commodity vector databases are commodities and then finally when you start using them as a developer you start to recognize that well not all of them are truly commodities because one works better than the other or one simply you're able to connect with one more so than the other and that's why when I asked you about Python uh we have a lot of languages in our ecosystem.

Python is not one of them yet, right? But as we build our next API, we've recognized that, you know what, we could do that using Golang. We love the language, but the support for Golang in the space of AI and all of these tools and platforms is not very natural, right? It's very it's contrived, if you will.

So, you know, we have to find and make do with what's there what's out there. Uh, as opposed to Python where things are a bit more natural from what I've seen. So those are some things I want to just add to you know what you've already said here. Yeah, that makes sense.

Dave Trier: Especially you need to make sure that there's an ecosystem and support from the community for that ecosystem. Otherwise, it becomes a maintenance nightmare, right? What happens when you know the one of the libraries goes out of um you know out of end of life if you will and then you have a bunch of dependent libraries on that one. If there's not community support for those then it's a real hassle.

you end up having to maintain some dependent libraries that are just unrelated to what you're trying to do, right? So that's just an example, but your point is absolutely correct is that you want to pick languages and frameworks that you're seeing community support around.

Krish Palaniappan: You mentioned Agentic, right? So seems like MCP and a A2A are two different uh you know protocols and frameworks that are starting to emerge, right? So again, it's about picking things that you're seeing the community support around MCP. MCP is a model context protocol.

What is the other one that you mentioned?

Dave Trier: Oh, it's just A2A. It's just another one that's becoming uh more, you know, prevalent, if you will.

Krish Palaniappan: Is it agent to agent or something like that?

Dave Trier: Yeah, agent to agent. Okay, cool.

Krish Palaniappan: And I think uh this is amazing. You've shared a bunch of really useful things here. So, as we wrap this up, Dave, uh any questions you have for me? I've I've been throwing questions at you.

It's the easier thing to do. Any questions you have for me that I can answer?

Dave Trier: No, I just again, thank you so much for the time, Krish. And um I would just you know say to your listeners out there that uh overall this is a very common problem right a very common problem of just understanding well I know we have a lot of demand or we got a lot of requests to put AI into usage especially generative AI into usage and it's very common to run up to impediments a wall if you will at a certain point. So you know feel free to reach out to myself or our company if there's anything that we can we can help you with. Absolutely.

Krish Palaniappan: Um, I think that's all I have. If you have nothing else, Dave, I want to say thank you so much. You know, you've uh, you know, brevity is not my forte, but I'm glad, you know, folks who are show who show up at the podcast are able to say things very correctly and in quick time as well. That takes a different level of skill that I honestly don't have.

You've packed a number of things in the 30 minutes or 35 minutes or so that we've chatted here. Um so at least you know uh for folks who are again watching this listening to it they can go back pause and then pick up what you said there digest it do some research okay what is MCP what is A2A or what is agentic AI what are AI agents and then and then go from there so uh folks uh we talked to Dave uh Trier this was amazing conversation plenty of learnings for me here Dave is a VP of product again at ModelOp a leading AI life cycle automation and governance and software for enterprises. Uh I will include the we'll include the links to the company and to Dave's profile and anything any other links that you want for us to include Dave we are happy to do it. Um thanks so much for your time Dave and just you know after I hit stop just hang on for a second so I know the upload completes.

Thanks Dave.

Dave Trier: Thank you Krish. Pleasure.