What Could Possibly Go Wrong? Safety Analysis for AI Systems

About this episode

The episode discusses safety analysis for AI systems, focusing on the complexities and vulnerabilities of LLMs and the STPA technique.

How can you ever know whether an LLM is safe to use? Even self-host ed LLM system s are vulnerable to adversarial prompt s left on the internet and waiting to be found by system search engines . These at tacks and others exploit the complexity of even seemingly secure AI systems . In our latest podcast from the Carnegie Mellon University Software Engineering Institute (SEI), David Schulker and Matthew Walsh, both senior data scientists in the SEI's CERT Division, sit down with Thomas Scanlon, lead of the CERT Data Science Technical Program, to discuss their work on System Theoretic Process Analysis, or STPA, a hazard-analysis technique uniquely suitable for dealing with AI complexity when assuring AI systems.

People in this episode

Guests: David Schulker, Matthew Walsh, Thomas Scanlon

Topics covered

• AI safety
• hazard analysis
• System Theoretic Process Analysis
• adversarial prompts
• complexity in AI systems

Keywords

• AI systems
• safety analysis
• LLM
• adversarial attacks
• hazard-analysis technique

Mentioned in this episode

Organizations: Carnegie Mellon University, Software Engineering Institute, CERT Division

More episodes of Software Engineering Institute (SEI) Podcast Series

• An LLM Evaluation Framework for High-Stakes AI · June 11, 2026 · 17 min
• Protecting AI Systems Against Data Poisoning · June 4, 2026 · 20 min
• Goal-Line Defense: A Tool to Discover and Mitigate UEFI Vulnerabilities · April 15, 2026 · 41 min
• Leadership, Legacy, and the Power of Mentors: Insights from Dr. Paul Nielsen · April 6, 2026 · 19 min
• With a Little Help from Our Civilian Friends: Cybersecurity Reserve Is Both Feasible and Advisable · March 20, 2026 · 49 min
• Maturing AI Adoption: From Chaos to Consistency · March 2, 2026 · 26 min