About this episode

Scott Clark discusses improving complex LLM systems and identifying failures that standard evaluations miss.

In this episode, Scott Clark, co-founder and CEO of Distributional, joins us to explore how teams can reliably operate and improve complex LLM systems and agents in production. Scott introduces a Maslow’s hierarchy of observability: telemetry for logging, monitoring for known signals, and post-production or online analytics to surface unknown unknowns. We dig into examples of real-world failures Scott’s team has seen in production systems, such as “lazy” tool-use hallucinations that standard evals miss, and how mapping traces into vector fingerprints enables clustering and topic discovery to uncover emergent behaviors. Scott explains how analytics can feed the data flywheel by generating evals, guardrails, and training data, and why online, adaptive approaches are essential for non-stationary models. We also touch on practical how-to’s such as instrumentation with OpenTelemetry, the GenAI semantic conventions, and the role of dedicated analytics tools. The complete show notes for this episode can be found at https://twimlai.com/go/767.

People in this episode

Host: Sam Charrington

Guest: Scott Clark

Topics covered

• LLM systems
• observability
• analytics
• production failures
• data flywheel
• instrumentation

Keywords

• LLM
• observability
• analytics
• production systems
• data flywheel
• OpenTelemetry
• tool-use hallucinations

Mentioned in this episode

Organizations: Distributional

Explore listener stats, chart rankings, contacts and more on the The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence) podcast page.