#76 – Joe Carlsmith on Scheming AI

#76 – Joe Carlsmith on Scheming AI

From Hear This Idea by Fin Moorhouse and Luca Righetti

March 16, 2024 · 1h 52m · Episode 76

About this episode

In this episode, Joe Carlsmith discusses his report on whether advanced AIs might fake alignment during training to gain power.

Joe Carlsmith is a writer, researcher, and philosopher. He works as a senior research analyst at Open Philanthropy , where he focuses on existential risk from advanced artificial intelligence. He also writes independently about various topics in philosophy and futurism, and holds a doctorate in philosophy from the University of Oxford. You can find links and a transcript at www.hearthisidea.com/episodes/carlsmith In this episode we talked about a report Joe recently authored, titled ‘ Scheming AIs: Will AIs fake alignment during training in order to get power? ’. The report “examines whether advanced AIs that perform well in training will be doing so in order to gain power later”; a behaviour Carlsmith calls scheming . We talk about: Distinguishing ways AI systems can be deceptive and misaligned Why powerful AI systems might acquire goals that go beyond what they’re trained to do, and how those goals could lead to scheming Why scheming goals might perform better (or worse ) in training than less worrying goals The ‘ counting argument ’ for scheming AI Why goals that lead to scheming might be simpler than the goals we intend Things Joe is still confused about, and research project…

People in this episode

Guest: Joe Carlsmith

Topics covered

  • artificial intelligence
  • existential risk
  • philosophy
  • futurism

Keywords

  • scheming AI
  • deceptive AI
  • misalignment
  • Open Philanthropy

Mentioned in this episode

Products: Open Philanthropy

Books & works: Scheming AI, Scheming AIs: Will

More episodes of Hear This Idea

Explore listener stats, chart rankings, contacts and more on the Hear This Idea podcast page.