From Data to Performance: Understanding and Improving Your AI Model

About this episode

The episode discusses the challenges and solutions in evaluating AI and ML model performance, focusing on the AI Robustness tool.

Modern data analytic methods and tools—including artificial intelligence (AI) and machine learning (ML) classifiers—are revolutionizing prediction capabilities and automation through their capacity to analyze and classify data. To produce such results, these methods depend on correlations. However, an overreliance on correlations can lead to prediction bias and reduced confidence in AI outputs. Drift in data and concept, evolving edge cases, and emerging phenomena can undermine the correlations that AI classifiers rely on. As the U.S. government increases its use of AI classifiers and predictors, these issues multiply (or use increase again) . Subsequently, users may grow to distrust results. To address inaccurate erroneous cor r elation s and predictions , we need new methods for ongoing testing and evaluation of AI and ML accuracy. In this podcast from the Carnegie Mellon University Software Engineering Institute (SEI) , Nicholas Testa, a senior data scientist in the SEI's Software Solutions Division (SSD) , and Crisanne Nolan, and Agile transformation engineer, also in SSD, sit down with Linda Parker Gates, Principal Investigator for this research and initiative lead for…

People in this episode

Guests: Nicholas Testa, Crisanne Nolan

Topics covered

• AI
• machine learning
• data analysis
• prediction bias
• model evaluation
• AI robustness

Keywords

• AI
• machine learning
• data correlation
• prediction bias
• model performance
• evaluation methods

Mentioned in this episode

Organizations: Carnegie Mellon University Software Engineering Institute, Software Engineering Institute

More episodes of Software Engineering Institute (SEI) Podcast Series

• An LLM Evaluation Framework for High-Stakes AI · June 11, 2026 · 17 min
• Protecting AI Systems Against Data Poisoning · June 4, 2026 · 20 min
• Goal-Line Defense: A Tool to Discover and Mitigate UEFI Vulnerabilities · April 15, 2026 · 41 min
• Leadership, Legacy, and the Power of Mentors: Insights from Dr. Paul Nielsen · April 6, 2026 · 19 min
• With a Little Help from Our Civilian Friends: Cybersecurity Reserve Is Both Feasible and Advisable · March 20, 2026 · 49 min
• Maturing AI Adoption: From Chaos to Consistency · March 2, 2026 · 26 min