“How much should the ideal person cry wolf?” by KatjaGrace

About this episode

KatjaGrace discusses the implications of 'crying wolf' in the context of AI safety and rationality.

It is a fact about wolves and rationality that you should warn people about wolves quite a few times for every effective wolf attack. In particular, there is an asymmetry between the costs of having one's flock devoured and averting a non-eventuating wolf attack. If the carnage is a hundred times worse, then it's worth up to ninety-nine false alarms to stop it. The original fable was about a boy who would continually lie about wolves, and that is definitely poor form. But in modern parlance, ‘crying wolf’ seems to be used for just being openly alarmed about things that turn out ok—I don’t hear much implication of deceit. And in modern sensibilities, being seen to ‘cry wolf’—by even once raising an alarm that isn’t consummated with disaster—is something people seem to really fear. I think multiple people have asked me about whether AI safety people might have ‘cried wolf’ about some earlier GPT model. I’m not aware of anyone doing that, but the idea that they might have is so tantalizing that it bears investigating. Because if even a a few people somewhere did, it would be such a nice embarrassing blow to AI [...] --- First published: April 30th, 2026 Source…

People in this episode

Host: LessWrong

Guest: KatjaGrace

Topics covered

• rationality
• AI safety
• crying wolf
• false alarms
• philosophy of risk

Keywords

• cry wolf
• AI safety
• false alarms
• rationality
• wolf attacks
• GPT model
• alarm
• philosophy

Mentioned in this episode

Organizations: LessWrong

Books & works: How much should the ideal person cry wolf?

More episodes of LessWrong (30+ Karma)

• “The Uncertainty That Matters Isn’t Fundamental” by jimmy · June 13, 2026 · 22 min
• [Linkpost] “US government directive to suspend access to Fable 5 and Mythos 5” by Capybasilisk · June 13, 2026 · 0 min
• “Claude Fable 5 and Mythos 5: The System Card” by Zvi · June 12, 2026 · 60 min
• “Citations Needed: Magic Encyclopedias to Save the World” by Oliver Sourbut · June 12, 2026 · 10 min
• “Simulating Simulators” by kromem · June 12, 2026 · 27 min
• “Implications of Continual Learning for LLM Agents: Introduction” by RohanS, Rauno Arike, Owen Terry, Achu Menon, Zhijing Jin, Francis Rhys Ward, Seth Herd · June 12, 2026 · 14 min