AI incidents, audits, and the limits of benchmarks

About this episode

The episode discusses AI safety and the challenges of verification and evaluation in real-world AI deployments.

AI is moving fast from research to real-world deployment, and when things go wrong, the consequences are no longer hypothetical. In this episode, Sean McGregor, co-founder of the AI Verification & Evaluation Research Institute and also the founder of the AI Incident Database, joins Chris and Dan to discuss AI safety, verification, evaluation, and auditing. They explore why benchmarks often fall short, what red-teaming at DEFCON reveals about machine learning risks, and how organizations can better assess and manage AI systems in practice. Featuring: Sean McGregor– LinkedIn Chris Benson – Website , LinkedIn , Bluesky , GitHub , X Daniel Whitenack – Website , GitHub , X Links: AI Verification & Evaluation Research Institute AI Incident Database 38th convening of IAAI BenchRisk State of Global AI Incident Reporting Upcoming Events: Register for upcoming webinars here !

People in this episode

Hosts: Chris Benson, Daniel Whitenack

Guest: Sean McGregor

Topics covered

• AI safety
• verification
• evaluation
• auditing
• machine learning risks
• benchmarks

Keywords

• AI incidents
• audits
• benchmarks
• red-teaming
• machine learning
• AI systems
• risk management

Mentioned in this episode

Organizations: AI Verification & Evaluation Research Institute, AI Incident Database

More episodes of Practical AI

• Zero Trust for AI Agents · June 11, 2026 · 47 min
• Breaking down the 2026 Stanford AI Index Report · June 4, 2026 · 47 min
• Rebooting Enterprise AI with MCP and Kubernetes · May 28, 2026 · 48 min
• Hermes Agent: Agents that grow with you · May 21, 2026 · 52 min
• U.S. Congressman Beyer on AI challenges facing America and the World · May 14, 2026 · 45 min
• The Myth of Model Wars: Open vs Closed AI in 2026 · May 7, 2026 · 42 min