About this episode
The episode discusses the evolution of AI evaluations and their importance in product development, featuring insights from experts in the field.
There are a lot of reasons why we should do AI evals. For many companies doing AI evals is the way to build the feedback loop into the product development lifecycle. So it is like your compass. We’re using AI evals as a compass to guide product development and also product iteration. And also, many times we need evals to function as the pass or fail gate in release decisions. Whether this product is good enough for release or whether it is good enough for experiment, evals are also used in that. Stella Wenxing Liu , Head of Applied Science at ASU , and Eddie Landesberg , Staff Data Scientist at Google , join Hugo to talk about why AI evaluation is evolving from “vibe checks” into a rigorous, multi-disciplinary science and how causal inference will take AI evals to the next level in 2026 . Vanishing Gradients is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber. They Discuss: * Team-Centric AI Evals , integrating product managers , data scientists , and SMEs under a “benevolent dictator” (or not!) to ensure comprehensive and effective evaluation; * Custom Evaluation Metrics , moving beyond generic vendor metrics…
People in this episode
Host: Hugo Bowne-Anderson
Guests: Stella Wenxing Liu, Eddie Landesberg
Topics covered
- AI evaluation
- product development
- causal inference
- custom evaluation metrics
- team-centric evaluations
- policy evaluation
Keywords
- AI evals
- product iteration
- feedback loop
- evaluation metrics
- causal inference
- AB testing
- data science
Mentioned in this episode
Organizations: ASU, Google
More episodes of Vanishing Gradients
- The Future of Agentic Data Science · May 25, 2026 · 1h 5m
- Agent-Harness.ipynb* · May 20, 2026 · 1h 20m
- Agentic Engineering and the Lost Art of Verification · May 12, 2026 · 1h 32m
- Privacy Theater Is Not Privacy Engineering: What It Actually Takes to Ship Safe AI · April 15, 2026 · 1h 7m
- LLM Architecture in 2026: What You Need to Know with Sebastian Raschka · April 13, 2026 · 1h 18m
- Episode 72: Why Agents Solve the Wrong Problem (and What Data Scientists Do Instead) · March 20, 2026 · 1h 34m
Explore listener stats, chart rankings, contacts and more on the Vanishing Gradients podcast page.