In this episode of the AI Engineering Podcast, host Tobias Macey works through generative AI governance with ModelOp CTO Jim Olsen in unusual technical depth. Jim's organizing argument is that governance attaches to the use case, not the foundation model: generating an image for a blog deserves little oversight, while summarizing a doctor's notes — where an omitted medication could lead to a double dose — deserves a great deal. On vendor models he is pragmatic, noting that legal exposure over training data lands mostly on the model's developer rather than its consumer, but that consuming a vendor model doesn't excuse you from establishing your own baselines and monitoring. The conversation covers shadow use and hospitals blocking domains outright, how model governance works alongside data governance rather than replacing it by tying dataset references to model instances, tokenization to keep PII out of requests, and automated PII-disclosure testing using jailbreak prompts. Jim is blunt about the ceiling: LLMs are fluent but not factual, they remain uninterpretable, and the practical response is constrained application interfaces, flow control tools like NVIDIA NeMo, and cosine similarity between RAG answers and source documents. He also covers multimodal risk — an offensive image being far harder to classify than an offensive word, and culturally dependent — the cost calculus between hosted and self-hosted models, signs that ultra-large models are no longer outperforming their counterparts, and his own bias tests that varied only a protected-class attribute and got visibly different answers.
- Why governance attaches to the use case rather than the foundation model itself.
- Where legal exposure over training data actually lands — the developer, not the consumer.
- Why consuming a vendor model still requires your own baselines and ongoing monitoring.
- Shadow use and why some hospitals simply block the domains at the network layer.
- How model governance works alongside data governance by tying dataset references to model instances.
- Tokenization to keep PII out of requests, and why consistent substitution preserves training results.
- Automated PII-disclosure testing using jailbreak prompts, plus ongoing review of real answers.
- Why LLMs are fluent but not factual, and why that makes hallucination the hardest failure to detect.
- Constrained application interfaces and flow control tools rather than open prompt access.
- Using cosine similarity between a RAG response and its source documents as a hallucination check.
- Why multi-turn conversation raises hallucination risk and token cost together.
- Human-in-the-middle deployment as the cautious first pattern at large enterprises.
- Why an offensive image is far harder to classify than an offensive word, and culture-dependent.
- Bias testing by varying only a protected-class attribute in an otherwise identical persona.
- The cost calculus between hosted models, self-hosted Llama, and small models that run on a phone.
- Early signs that ultra-large models are not outperforming their smaller counterparts enough to justify cost.
[00:05] – Introduction
[00:50] – How Jim first got into machine learning
[01:40] – What governance even means for foundation models
[02:10] – Why the use case determines the level of oversight
[03:12] – Transparency in how foundation models were built
[04:15] – Where legal exposure over training data actually lands
[05:18] – Policy creation as a business and technical hybrid
[06:23] – Why organizations struggle to know what they even have
[07:25] – Patient data sent to a public chatbot
[08:27] – Why testing models is unlike a CI/CD pipeline
[09:31] – What sending data over an API actually means
[10:34] – Private instances, self-hosting, and secure cloud
[11:38] – Why this is an enhanced problem rather than a new one
[12:43] – Shadow use and the virality of these tools
[13:47] – Blocking at the network layer, and the education gap
[14:51] – How model governance works with data governance
[15:56] – Closing the loop between datasets and model instances
[17:01] – Technical controls: context filters and token masking
[18:04] – Why consistent tokenization preserves training results
[18:40] – Automated PII-disclosure testing with jailbreak prompts
[19:07] – Baselines, cross-LLM querying, and detecting drift
[20:11] – Why static guardrails fail against the mutability of language
[21:15] – Fluent but not factual
[22:18] – Flow control versus pre- and post-filtering
[22:50] – The car dealership chatbot and the danger of moving too fast
[23:21] – Why hallucination is the hardest failure to detect
[24:25] – Constraining inputs and outputs
[25:27] – System prompt attacks and an immature tooling space
[26:31] – Cosine similarity against reference documents in RAG
[27:36] – Multi-turn conversation, hallucination, and cost
[28:38] – Human-in-the-middle as the first deployment pattern
[29:41] – Flow control and preventing conversations going off the rails
[30:44] – How organizations weigh internal versus customer-facing use
[31:46] – Bias testing across protected classes
[32:49] – Why traditional ML adoption outpaces generative AI
[33:51] – Multimodal risk and why images are harder to classify
[34:56] – Cultural context in what counts as offensive
[36:03] – Where the ecosystem most needs investment
[37:07] – Trust, expiry, and the DevOps precedent
[38:10] – Why explainability may never fully arrive
[39:15] – The most common mistake: underestimating the problem
[40:18] – Why regulations are obscure and peer interpretation matters
[41:21] – Lessons from a rapidly shifting space
[42:25] – Seeing through the hype cycle
[43:27] – Trends being monitored, and EU reporting obligations
[44:31] – Breaking down organizational barriers
[45:35] – Operational cost and the compute question
[46:37] – Hosted versus self-hosted trade-offs
[47:40] – Small language models and diminishing returns at the top end
[48:43] – Where regulation does and does not make sense
[49:45] – Decisions that affect real lives
[50:48] – The biggest gaps in tooling today
[51:52] – Industrializing models for business use
[52:54] – Closing remarks
In this episode of the AI Engineering Podcast Jim Olson, CTO of ModelOp, talks about the governance of generative AI models and applications. Jim shares his extensive experience in software engineering and machine learning, highlighting the importance of governance in high-risk applications like healthcare. He explains that governance is more about the use cases of AI models rather than the models themselves, emphasizing the need for proper inventory and monitoring to ensure compliance and mitigate risks. The conversation covers challenges organizations face in implementing AI governance policies, the importance of technical controls for data governance, and the need for ongoing monitoring and baselines to detect issues like PII disclosure and model drift. Jim also discusses the balance between innovation and regulation, particularly with evolving regulations like those in the EU, and provides valuable perspectives on the current state of AI governance and the need for robust model lifecycle management.
