“11 Open Empirical Problems in Reward-Seeking” by Alex Meinke, Jérémy Scheurer, Axel Højmark, Theodore Ehrenborg
About this episode
The episode discusses open empirical problems in reward-seeking models and their implications for alignment.
We recently published our paper on "Measuring Reward-Seeking via Contrastive Belief Updates". We're excited about research like this, and there are many more open problems than we can work on. Here's a list of open problems that we think are valuable. If you work on/solve these problems, we'd be happy to signal-boost your research. If your next research project is one of these problems, feel free to reach out to alex@apolloresearch.ai to discuss it in more detail. Reward-Seeking and its Implications 1. Is a Reward-Seeking Model more difficult to align? The strongest case for expecting reduced "train-time corrigibility" due to reward-seeking, applies to Instrumental Reward-Seeking, where the model actively reasons "I will please oversight now, in order to accomplish some other thing later". Alignment training a model like that may update its beliefs about graders and oversight, without reshaping its underlying values. Current forms of reward-seeking are likely better understood as terminal, i.e. models try to please the grader without ulterior motives. There is likely a continuous spectrum between Terminal and Instrumental Reward-Seeking. Thus, we can hopefully study the effects…
People in this episode
Guests: Alex Meinke, Jérémy Scheurer, Axel Højmark, Theodore Ehrenborg
Topics covered
- reward-seeking
- alignment
- empirical problems
- research
- instrumental reward-seeking
Keywords
- reward-seeking
- alignment
- empirical problems
- instrumental reward-seeking
- research
Mentioned in this episode
Organizations: Apollo Research
More episodes of LessWrong (30+ Karma)
- “What do I mean by “Artificial General Intelligence”?” by Steven Byrnes
- “Epistemics and Coordination: It’s complicated!” by Raymond Douglas
- “Differential acceleration of alignment-relevant capabilities is a bad bet” by Zephaniah Roe
- “Measuring Reward-Seeking via Contrastive Belief Updates” by Jérémy Scheurer, Axel Højmark, jenny, Felix Hofstätter, Theodore Ehrenborg, Bronson Schoen, Alex Meinke
- “Demis Hassabis on the New Coming Age” by Zvi
- “AI 2040: Is it Actually a Deal?” by 1a3orn
Explore listener stats, chart rankings, contacts and more on the LessWrong (30+ Karma) podcast page.