“No Strong Orthogonality From Selection Pressure” by lumpenspace

About this episode

The episode discusses the relationship between intelligence and morality, arguing that intelligence does not guarantee human-compatible values.

TL;DR If everything goes according to plan, by the end of this post we should have separated three claims that are too often bundled together: Intelligence does not imply human morality. Weird minds are possible. 3.A reflective, recursively improving intelligence should be expected to remain bound to a semantically thin “terminal goal” that emerged during training. I accept the first two. I am arguing against the third. So: I am not making the case that sufficiently intelligent systems automatically turn out nice, human-compatible, or safe. Nor am I trying to prove that a paperclip maximizer is impossible somewhere in the vast reaches of mind-design space. Mind-design space is large; let a thousand theoretical paperclipper views. I hope to defend this smaller claim: intelligence is not a neutral engine you can just bolt onto an arbitrary payload. Larger claims I am not making A typical rebuttal to anti-orthogonalist perspectives is: The genie can know what you meant and still not care. Of course it can: an entity can perfectly map human morality without adopting it as a terminal value. Superintelligence does not imply Friendliness. I am not trying to smuggle Friendliness in…

People in this episode

Guest: lumpenspace

Topics covered

• intelligence
• morality
• superintelligence
• mind-design space
• anti-orthogonalism

Keywords

• intelligence
• morality
• superintelligence
• terminal goal
• anti-orthogonalism
• mind-design space
• Friendliness

More episodes of LessWrong (30+ Karma)

• “The Uncertainty That Matters Isn’t Fundamental” by jimmy · June 13, 2026 · 22 min
• [Linkpost] “US government directive to suspend access to Fable 5 and Mythos 5” by Capybasilisk · June 13, 2026 · 0 min
• “Claude Fable 5 and Mythos 5: The System Card” by Zvi · June 12, 2026 · 60 min
• “Citations Needed: Magic Encyclopedias to Save the World” by Oliver Sourbut · June 12, 2026 · 10 min
• “Simulating Simulators” by kromem · June 12, 2026 · 27 min
• “Implications of Continual Learning for LLM Agents: Introduction” by RohanS, Rauno Arike, Owen Terry, Achu Menon, Zhijing Jin, Francis Rhys Ward, Seth Herd · June 12, 2026 · 14 min