“Stop doing decision theory without metaphysics” by Elias Schmied
About this episode
Elias Schmied discusses the complexities of decision theory and metaphysics in the context of blackmail scenarios.
[Epistemic status: rant] There's something that annoys me about the reoccurring debates on decision theory in this corner of the internet. Take a simple blackmail scenario: Omega, a near-perfect predictor, knows a piece of embarrassing information about you. He threatens you that he will release it to the public if you don’t pay him 100$. However, making the threat is slightly costly to him, and he wouldn’t have done it if he hadn’t predicted that you would pay. Do you pay? Let's say we want to argue for the Functional Decision Theory (FDT) answer that you shouldn’t pay. It was originally motivated by the observation that such agents seem to achieve higher utility (“rationality is about winning”), since they don’t get blackmailed in the first place. But that only leads to making yourself into such an agent in advance (which everybody generally agrees you should do[1]) - it doesn’t clearly apply when you are already being blackmailed and have never thought about the question before, or if you are an AI who was just created and is instantly blackmailed before being able to self-modify or make precommitments. I see three broad ways to make FDT's recommendation make sense in [...]…
People in this episode
Guest: Elias Schmied
Topics covered
- decision theory
- metaphysics
- blackmail
- Functional Decision Theory
- rationality
- AI
Keywords
- decision theory
- metaphysics
- blackmail
- Functional Decision Theory
- rationality
- AI
More episodes of LessWrong (30+ Karma)
- “What do I mean by “Artificial General Intelligence”?” by Steven Byrnes
- “Epistemics and Coordination: It’s complicated!” by Raymond Douglas
- “Differential acceleration of alignment-relevant capabilities is a bad bet” by Zephaniah Roe
- “Measuring Reward-Seeking via Contrastive Belief Updates” by Jérémy Scheurer, Axel Højmark, jenny, Felix Hofstätter, Theodore Ehrenborg, Bronson Schoen, Alex Meinke
- “11 Open Empirical Problems in Reward-Seeking” by Alex Meinke, Jérémy Scheurer, Axel Højmark, Theodore Ehrenborg
- “Demis Hassabis on the New Coming Age” by Zvi
Explore listener stats, chart rankings, contacts and more on the LessWrong (30+ Karma) podcast page.