About this episode

Beth Barnes and David Rein discuss the implications of their graph on AI timelines and the nuances of model behavior.

Beth Barnes and David Rein on the one graph that ate the AI timelines discourse, and why the two people who built it are the most careful about how you read it.**SPONSOR**Prolific - Quality data. From real people. For faster breakthroughs.https://www.prolific.com/?utm_source=mlstInterview: https://youtu.be/cnxZZTl1tkk---Beth Barnes and David Rein from METR on the one graph that ate the AI timelines discourse, and why the people who built it are the most careful about how it gets read.Beth founded METR after leaving OpenAI alignment. David is first author on GPQA and co-author on HCAST and the METR Time Horizons paper. Together they built the measurement Daniel Kokotajlo called the single most important piece of evidence on AI timelines: the log-linear line of "how long a task a frontier model can complete at 50% reliability" vs release date.The conversation opens on reward hacking. Current models can articulate in chat why a behaviour is undesired and then execute it anyway as agents. From there: construct validity, Melanie Mitchell's four-problem taxonomy, and the ARC-AGI 1-to-2 collapse as a worked example of adversarially-selected benchmarks regressing once labs…

People in this episode

Guests: Beth Barnes, David Rein

Topics covered

• AI timelines
• reward hacking
• construct validity
• adversarial benchmarks
• model evaluation
• task reliability

Keywords

• AI models
• reward hacking
• task reliability
• METR
• evaluation methodology
• adversarial benchmarks

Sponsors

Prolific

Mentioned in this episode

Organizations: METR, OpenAI, ARC-AGI

Books & works: GPQA, HCAST, METR Time Horizons paper

Explore listener stats, chart rankings, contacts and more on the Machine Learning Street Talk (MLST) podcast page.