AI Evals & Discovery

Show notes

What you’ll learn in this episode:

• What “evals” actually mean in the AI/ML world
• Why evals are more than just quality assurance
• The difference between golden datasets, synthetic data, and real-world traces
• How to identify error modes and turn them into evals
• When to use code-based evals vs. LLM-as-judge evals
• How discovery practices inform every step of AI product evaluation
• Why evals require continuous maintenance (and what “criteria drift” means for your product)
• The relationship between evals, guardrails, and ongoing human oversight

Resources & Links:

• Follow Teresa Torres: https://ProductTalk.org
• Follow Petra Wille: https://Petra-Wille.com

Mentioned in the episode:

• How I Designed & Implemented Evals for Product Talk’s Interview Coach by Teresa Torres Teresa’s - Interview Coach
• ML (Machine learning)
• Story-Based Customer Interviews - On Demand course by Teresa
• LLM (Large language model)
• AI Evals for Engineers and PMs course (get 35% off through Teresa’s link) on Maven
• V0
• JSON (JavaScript Object Notation)
• Anthropic
• The Product Leadership Wheel - A Framework for Defining and Growing Product Leadership at Scale by Petra Wille
• Lovable
• Behind the Scenes: Building the Product Talk Interview Coach by Teresa
• Previous episode: - Building AI Products

Coming soon from Teresa:

• Weekly Monday posts sharing lessons learned while building AI products
• A new podcast interviewing cross-functional teams about real-world AI product development stories