An LLM Evaluation Framework for High-Stakes AI

About this episode

Violet Turri and Katie Robinson discuss the Evaluating Large Language Models (ELM) library for rigorous LLM evaluation.

Experimentation and validation of LLM performance is critical when building LLM-driven systems that must reliably deliver a service, from customer service chat bots to intelligence analysis tools. To help teams meet the need for rigorous evaluation methods, a research team in the SEI's AI Division led by Violet Turri has developed the Evaluating Large Language Models (ELM) library, which is built on best practices for LLM evaluation and benchmarking. In the latest episode from the Carnegie Mellon University Software Engineering Institute, Turri sits down with Katie Robinson, a design researcher also in the SEI's AI division, to discuss the ELM library, which turns evaluation from an ad-hoc process into a repeatable, extensible framework.

People in this episode

Guests: Violet Turri, Katie Robinson

Topics covered

• LLM evaluation
• AI systems
• benchmarking
• customer service
• intelligence analysis
• research methods

Keywords

• LLM
• evaluation framework
• AI
• benchmarking
• customer service
• intelligence analysis
• research

Mentioned in this episode

Organizations: Software Engineering Institute, Carnegie Mellon University, AI Division

More episodes of Software Engineering Institute (SEI) Podcast Series

• Protecting AI Systems Against Data Poisoning · June 4, 2026 · 20 min
• Goal-Line Defense: A Tool to Discover and Mitigate UEFI Vulnerabilities · April 15, 2026 · 41 min
• Leadership, Legacy, and the Power of Mentors: Insights from Dr. Paul Nielsen · April 6, 2026 · 19 min
• With a Little Help from Our Civilian Friends: Cybersecurity Reserve Is Both Feasible and Advisable · March 20, 2026 · 49 min
• Maturing AI Adoption: From Chaos to Consistency · March 2, 2026 · 26 min
• Temporal Memory Safety in C and C++: An AI-Enhanced Pointer Ownership Model · February 9, 2026 · 24 min