EVA - A Framework for Evaluating Voice Agents by ServiceNow

About this episode

The episode discusses the challenges of evaluating voice AI agents and introduces ServiceNow's open-source framework for establishing industry standards.

Voice AI agent evaluation — why it's fundamentally harder than text, how cascade failures derail conversations invisibly, and ServiceNow's open-source framework to establish industry evaluation standards. Featuring real audio examples showing authentication failures, leaked reasoning, and latency problems. WHAT WE COVER  TARA BOGAVELLI — Research Engineer, ServiceNow Leading the open-source voice agent evaluation framework. Explains why existing benchmarks don't measure what matters and what ServiceNow is releasing to establish industry standards. KATRINA STANKIEWICZ — Staff Machine Learning Engineer, ServiceNow Cascade model architecture expert. Breaks down STT → LLM → TTS failure modes, named entity transcription challenges, and real audio example analysis. GABRIELLE GAUTHIER MELANÇON — Staff Applied Research Scientist, ServiceNow Multi-language evaluation specialist. Reveals why Large Audio Language Models lag behind, the native speaker requirement, and bot-to-bot simulation methodology.  CHAPTERS 0:00 Introduction — The evaluation gap 1:11 ServiceNow's Open-Source Framework Announcement — Tara Bogavelli 2:43 Meet…

People in this episode

Guests: Tara Bogavelli, Katrina Stankiewicz, Gabrielle Gauthier Melançon

Topics covered

• Voice AI evaluation
• Open-source frameworks
• Cascade failures
• Machine learning challenges
• Audio language models

Keywords

• Voice AI
• evaluation framework
• cascade failures
• machine learning
• audio examples
• named entity transcription
• evaluation metrics

Mentioned in this episode

Organizations: ServiceNow

More episodes of ServiceNow Podcasts

• 6th YEAR SEASON FINALE: Juan and Tim Rant · May 29, 2026 · 35 min
• Being AI Native at ServiceNow · May 27, 2026 · 24 min
• Pragmatic Use-Case-Driven Data Governance with Jason Doerr · May 20, 2026 · 43 min
• TAKEAWAY - Pragmatic Use-Case-Driven Data Governance with Jason Doerr · May 20, 2026 · 6 min
• Data Catalyst³: Governance × Change Management × Data Fluency w/ Bob Seiner · May 15, 2026 · 49 min
• TAKEAWAY - Data Catalyst³: Governance × Change Management × Data Fluency w/ Bob Seiner · May 15, 2026 · 5 min