About this episode

David Manheim discusses the shortcomings of AI evaluations and proposes ways to improve their reliability and transparency.

David Manheim is head of methodology at AI Evaluation Consensus. He joins the podcast to discuss how AI evaluations can become more reliable, transparent, and useful for decisions. We cover common failures such as unclear reporting, training to the test, benchmark saturation, and models changing behavior when they know they are being tested. The conversation also examines real-world tests, biosecurity, persuasion, forecasting, human oversight, and why even “normal” AI progress could be disruptive. LINKS: David Manheim website AI Evaluation Consensus Statement CHAPTERS: (00:00) Episode Preview (01:04) Evaluation consensus project (07:01) Evaluation awareness challenges (12:28) Reporting capabilities clearly (19:38) Benchmarks beyond humans (29:52) Proxies and biosecurity (42:01) Persuasion and democracy (53:59) Forecasting with AI (01:08:44) Oversight and disruption (01:16:42) Supporting better evals PRODUCED BY: https://aipodcast.ing SOCIAL LINKS: Website: https://podcast.futureoflife.org Twitter (FLI): https://x.com/FLI_org Twitter (Gus): https://x.com/gusdocker LinkedIn: https://www.linkedin.com/company/future-of-life-institute/ YouTube…

More episodes of Future of Life Institute Podcast

• How AI Is Replacing Children's Ability to Think (with Randi Weingarten) · June 30, 2026 · 44 min
• The Gulf Between AI Progress and Political Understanding (with Dex Hunter-Torricke) · June 25, 2026 · 1h 4m

Explore listener stats, chart rankings, contacts and more on the Future of Life Institute Podcast podcast page.