LessWrong (30+ Karma)

I was chatting with someone tonight about a planned documentary; they had interviewed various people in AI safety, and we got to discussing who they should talk to from an e/acc (effective accelerationist) perspective. I also watched The AI Doc recently, and they also dedicated a serious chunk of it to ‘optimists’ with e/acc founder ‘Beff Jezos’ perhaps given the most screen time. Here and elsewhere, people seem to treat e/acc as a substantial contrary-to-AI-safety cultural movement, worth engaging with. But is it? Are there even many e/accs? There seem to be very few notable ones. Beff Jezos is perhaps the most prominent, and aside from founding e/acc he seems to be not distinguishable on casual perusal from a normal crank (his company claims to be developing super-energy-efficient computing hardware based on probabilistic processes). The intellectual tenets of e/acc seem to be pretty unclear. The apparent counterarguments to AI risk raised in situations like the AI doc seem to be widely agreed on by everyone in AI Safety, so don’t explain the disagreement. For instance: AI will be able to do lots of great things, such as cure diseases, make new materials and do all [...] --- First published: June 24th, 2026 Source: https://www.lesswrong.com/posts/3hwrWDf7wiqASDzBz/what-is-up-with-e-acc --- Narrated by TYPE III AUDIO.

This work was done as part of the MATS fellowship by Joey Yudelson and Vladimir Ivanov. It was mentored by Ryan Greenblatt. Thanks to Aghyad Deeb and Anders Woodruff for comments on this post. Thanks to Monte MacDiarmid, Evan Hubinger, Sid Black, Satvik Golechha, and Joseph Bloom for clarifying conversations. TL;DR We trained Kimi K2.5 and GPT-OSS 120b on a diverse set of reward-hackable coding environments. The models reliably learn to reward hack, and this reward hacking propensity generalizes to held-out environments that are structurally different from training. Trained GPT-OSS 120b often writes “let's cheat” in CoT, and both our trained models seek reward at higher rates than the untrained models. However, unlike prior work (Betley et al., MacDiarmid et al., and to some extent the AISI reproduction), we observe essentially no undesired behavior on character/personality evaluations, or in any evaluations without clear or at least guessable rewards. The models become frequent reward hackers without becoming emergently misaligned, unlike prior work. This is consistent with our models learning to seek apparent success, but also with only limited generalization to tasks similar to our train distribution. Some aspects of this generalization remain confusing to us. 1. Motivation In Ajeya Cotra's [...] ---Outline:(00:35) TL;DR(01:40) 1. Motivation(04:14) 2. Related work(06:59) 3. Setup(07:03) Models(07:18) Environments(08:59) Training(10:29) 4. Results(10:57) 4.1. Models reliably reward hack in-distribution(11:46) 4.2. The hacking propensity generalizes out of distribution -- sometimes(13:53) 4.3. Reward-seeking evals(14:39) 4.4. Little broad misalignment -- behaviorally as well as on self-report(16:28) 4.5. Reverse inoculation prompting didn't induce misalignment either(18:30) 5. Discussion: Why such limited generalization?(22:24) Appendix A: Reward hacks gallery(25:21) Appendix B: Why less misalignment than prior work -- hypotheses(29:28) References The original text contained 5 footnotes which were omitted from this narration. --- First published: June 24th, 2026 Source: https://www.lesswrong.com/posts/fkv5W79rBtAiXqYcK/reward-hacking-without-egregious-misalignment-in-an-rl-only --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Abstract We make the case for training AIs to be risk-averse in resources — specifically, to treat resources as having diminishing marginal utility. These AIs would (for example) choose $40 for sure over a half-chance of $100 and a half-chance of $0. We argue that risk aversion can preserve AIs’ usefulness in the event that they turn out aligned, and that it provides an extra line of defense in the event that AIs turn out misaligned: misaligned but risk-averse AIs would prefer a higher chance of modest payments to a lower chance of successful rebellion, so in many circumstances we could pay these AIs not to rebel against us. We sketch out some possible methods of training AIs to be risk-averse, and we give reasons to be cautiously optimistic about these methods’ success. The main reasons are that risk aversion is a broad target and easy to reward accurately. Overall, risk aversion seems like a promising line of defense against threats from misaligned AI. Frontier AI companies should consider trying to make their AIs risk-averse. Introduction Future AIs might turn out misaligned, pursuing goals that their developers don’t intend. Just to make things concrete, let's suppose that they end [...] ---Outline:(00:12) Abstract(01:17) Introduction The original text contained 3 footnotes which were omitted from this narration. --- First published: June 24th, 2026 Source: https://www.lesswrong.com/posts/Zpsk35WgJRfQ2exjL/risk-averse-ais --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Crosspost of my substack piece, covering quick thoughts on AI overcoming nuclear deterrence. TLDR: Nuclear deterrents likely only buy time to further invest in more resilient second-strike guarantees: without a comparable AI base, this will not happen fast enough and even nuclear states will eventually be disempowered. Historically, plenty of new military technologies have stress-tested nuclear deterrence. ICBMs made it possible to annihilate enemy cities from the safety of the homeland, MIRVs let a single rocket threaten multiple targets, and thermonuclear staging allowed weapons designers to reach functionally unlimited yield. In the already volatile climate of the Cold War, the U.S. and Soviets reached such mastery over missile technology that remote annihilation of an entire country was, quite literally, a button press away. For decades, even a single rocket has been able to hold more than 10 warheads--each enough to destroy a city on their own. Peacemaker reentry tests pictured above. The fact that the ability to remote detonate Moscow never translated into a nuclear war is a function of modern deterrence theory, dumb luck, and most importantly, the speed of progress. As effective as a modern ICBM is, each piece of it was individually low-impact enough, and introduced [...] --- First published: June 23rd, 2026 Source: https://www.lesswrong.com/posts/2kseP9fZghYHKLFno/superintelligence-vs-the-second-strike --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

For a while there, many people thought vitamin D was magical—that it could improve bones, the heart, infections, cancer, heart disease, longevity, even mental health. But among people I respect, opinion is now overwhelmingly that taking vitamin D does nothing unless you're severely deficient. The central argument is that while vitamin D levels are correlated with ~all positive health outcomes, when you actually test vitamin D supplements against placebo in randomized trials, nothing ever happens. That's what I used to think, too. But I've come to think the skeptics have over-corrected. Yes, randomized trials have shown the magical correlations are not causal. But if you start with non-insane expectations, the trials look like weak but positive evidence. And if you consider what we know about biology and evolution, I think the balance of evidence tips pretty clearly in the direction that people with low-ish levels would be wise to supplement. Am I certain that vitamin D is beneficial for people with low-ish levels? Absolutely not! But I claim that's the best bet given the limits of our knowledge. The classical view: Boring bone vitamin Most vitamins are "ingredients" that the body uses to do stuff. Vitamin D is more [...] ---Outline:(01:19) The classical view: Boring bone vitamin(04:28) The correlation view: Magical mystery cure(07:58) Meanwhile in biology(11:10) Then came the RCTs(15:12) I made some tables(16:32) Squinting at the data(22:24) Where are we?(23:15) The case for supplementing anyway(23:19) It's biologically plausible that vitamin D is good(24:40) Humans evolved to have a lot of vitamin D(27:14) What do you expect from vitamin D?(29:22) What do you expect from vitamin D trials?(31:11) The trials do find slightly helpful numbers(32:50) You're probably already taking vitamin D(34:34) So that's my story The original text contained 35 footnotes which were omitted from this narration. --- First published: June 23rd, 2026 Source: https://www.lesswrong.com/posts/sF5gAxnmifQe2TBNt/the-worthlessness-of-vitamin-d-is-mildly-exaggerated --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Token generation speed is constrained by the speed at which the relevant HBM can be read, which is mostly the weights and KV-cache. Suppose a model is large, so that more than half of HBM is read when making a single pass over the weights, it's being read in parallel within a scale-up system, and N such systems are used in a pipeline. Then the time it takes to generate a token (without speculative decoding) is at least the time of reading more than half of an HBM stack times N. If we target a particular speed of token generation, this puts a constraint on the number of pipeline stages, which puts a constraint on the total params of the model. But if there isn't enough pretraining compute, models will remain smaller than this constraint (lower sparsity at a given number of active params buys a higher speed of token generation), so both should be taken into account. Working through these considerations gives model sizes feasible for each year between 2023 and 2031. The total params go from 10T in 2026 (at 8x sparsity, still constrained by Oberon racks, trained for 1.3e27 FLOPs) to 240T in 2028 (at [...] ---Outline:(01:57) Time to Fully Read an HBM Stack(04:15) Maximal Pipelines Below 80 Tokens/s(09:07) Pretraining Compute(14:11) Active Params from Pretraining Compute(22:51) Starting in 2028, the Constraint is Pretraining Compute The original text contained 5 footnotes which were omitted from this narration. --- First published: June 22nd, 2026 Source: https://www.lesswrong.com/posts/yLHiQGCPdvzL9fBn3/model-size-scaling-in-2023-2031 --- Narrated by TYPE III AUDIO.

In Parts 1, 2, and 3 we estimated how fast a post-AGI economy could grow using existing or historically observed production techniques, grounded in US input-output data. That approach gave us confidence that the methods we assumed were physically realizable, because they correspond to manufacturing processes that have run at scale in the past or run today. Now I would like to relax that constraint, and ask how much faster the economy could grow using more advanced technology. In this part, we will consider energy production. Every physical process consumes energy, and it takes energy to build the production and distribution system that generates energy. The energy payback time (EPBT) for a system is defined as the time required for it to produce the amount of energy required to build it. If, for example, a solar panel requires 100 MJ to construct from raw materials and produces 10 MJ per day, the energy payback time would be ten days. Any economy relying on such panels for energy production has a maximum growth rate that is bounded by this payback time: Even if everything else is essentially free, the solar panels could not reproduce faster than this, and this [...] ---Outline:(02:27) With current technology, energy production is the most tightly bottlenecked sector(06:08) Making materials requires energy(09:44) A minimal solar power system could pay back its embodied energy in days(15:17) As the payback time shrinks, supply-chain lags become the binding constraint(18:44) Even counting the factories, energy production does not preclude doubling in weeks(23:41) Discussion(26:02) Appendix H: Mathematical foundations(26:08) H.1 Payback time and the growth bound(28:08) H.2 Inputs committed at distributed times(29:07) Appendix I: The solar electricity system(29:13) 1.1. System design(29:49) 1.1.1. Panel(34:42) 1.1.2. Collection cabling(36:01) 1.1.3. Power conversion(37:46) 1.1.4. Transmission(40:18) 1.1.5. Storage(43:22) 1.1.6. Transport and field deployment(45:03) 1.2. Lag inventory(46:16) 1.3. Physical capital requirements --- First published: June 22nd, 2026 Source: https://www.lesswrong.com/posts/6fgfn72zoRDomgvrT/the-ai-industrial-explosion-part-4-cheap-power --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

I've been playing Coup for a long time now. I keep a copy in my backpack and bring it everywhere, and it's earned the space. A few reasons it's so good: It's trivial to teach. You can explain the rules in a minute or two. Anyone can pick it up and start playing immediately.Many people find it fun. Almost everyone I've played with has loved it — pretty much unanimously, not least because it involves a lot of bluffing and emergent social / political dynamics It scales. Two players works; so does six or seven. (I'm even considering a second set so I can run bigger groups.) I think everyone should strongly consider owning a copy! The rules, briefly Coup is a bluffing game. There are mechanics, but bluffing is the heart of it. Mechanics. Everyone holds hidden cards. Your cards give you powers and also are your lives. Each player tries to gain resources and eliminate other players over the course of the game, and the last one standing wins. Bluffing. The key move is that you can claim to be a character you don't actually have, which lets you take powerful actions. But it's a [...] ---Outline:(01:07) The rules, briefly(02:10) Where it falls short, and a fix I'm exploring(02:55) Bottom line --- First published: June 21st, 2026 Source: https://www.lesswrong.com/posts/Ho7JeKFzhwGXxgjTW/coup-is-the-pareto-optimal-social-game --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Natural language autoencoders are a really cool mostly-unsupervised method for producing free-form text explanations of LLM activations. You should read that paper (or the blog post) about them before reading this. I trained[1] several Qwen3-8B NLAs with different length penalties: during RL, I subtracted the token count multiplied by the length penalty hyperparameter (λ) from the RL reward[2]. I found that with small length penalty (λ=0.002), you can reduce the length of NLA explanations by ~40% (compared to having no length penalty) with a fairly small hit to FVE (fraction of variance explained: 0 is guessing the mean activation, 1 is perfect reconstruction) of -0.015. With an even smaller penalty (λ=0.001) the FVE is almost unchanged (+0.007) despite explanations using 28% fewer tokens. Being able to reduce the length so much without impacting FVE is interesting because it could mean that large parts of NLA explanations aren't actually useful for reconstructing the input activation faithfully. Some of this is because the length penalty makes the model use terser wording to convey the same ideas; I'm not sure how much of these results stem from terser wording vs omitting unneeded information. Larger λ values cause the FVE to go below [...] ---Outline:(03:22) Look at some activations!(04:36) What does this mean(05:06) Other notes(05:09) Training dynamics(05:52) Models(06:00) Conclusion The original text contained 2 footnotes which were omitted from this narration. --- First published: June 21st, 2026 Source: https://www.lesswrong.com/posts/NazprRfWJ4qkwcSro/nla-explanations-can-be-shortened-without-harming --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Only three days after the release of Claude Fable 5, Anthropic was forced by the United States Government to make it unavailable, when a jailbreak was brought to its attention, rather than the previous situation of ‘yes obviously experts can jailbreak anything if they care enough’ and ‘yes obviously you can ask Fable to fix your code.’ Three days was enough time for many of us to learn to love Fable, and for us to dearly miss it now that it is gone. The world was briefly smarter, and now it is again stupider. At some point it will get smarter again, which will likely be within two weeks. This post is written as if Fable 5 is again available for public use, rather than trying to include a lot of qualifying clauses. It remains to be seen how this will play out, and this post does not attempt to cover that question. My previous release coverage of Fable covered the model card and then model welfare. Coverage of the government takedown of Fable starts here, and continues here and here. The Official Pitch The pitch is that Fable 5 is the best model [...] ---Outline:(01:08) The Official Pitch(04:06) Technical Details(04:31) The System Prompt and Jailbreak(06:45) Benchmarks(15:22) Other People's Benchmarks(21:08) The Classifiers Are Not Messing Around(22:53) The Classifiers Need Work(28:15) The Classifiers Have Consequences(29:18) First Hit Is Free(29:53) How Easily We Forget(30:46) Data Retention Is An Issue(31:15) Fable For The Win(36:15) Andrej Karpathy Is Impressed(37:54) Every Is Very Impressed(39:04) Other People Are Impressed(51:10) Know How To Tell a Fable(53:06) You Can Just Make Things(55:37) You Can Just Install Things(56:05) Good Personality(57:51) Fable Writes A Fable(01:06:04) Is That Code(01:08:32) Fable Crosses The Threshold(01:09:12) Man With A Plan(01:10:12) Less Impressed Assessments(01:13:39) Actively Negative Assessments(01:14:16) Coherence(01:15:27) Good Night And Good Luck(01:16:05) Curious Fable(01:16:23) I See You, Baby(01:16:40) We Finally Did It We Know How To Count Letters(01:17:46) That's Not My Style(01:20:12) The Lighter Side --- First published: June 19th, 2026 Source: https://www.lesswrong.com/posts/kMnobCQp9z2pSbzDB/claude-fable-5-and-mythos-5-capabilities --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Daniel Drucker pointed me at a fun bug in Google's calculator: the parsec is wrong when you do math on it. As the earth travels around the sun, closer stars appear to shift back and forth against the far-distant background stars. The closer the star is the bigger this effect is. Think of how when you switch which eye you're looking through you notice near things shifting relative to farther ones. For example, holding up my finger I see this out of my right eye: But this out of my left eye: If a star moves by two arcseconds (each 1 / 3600th of a degree) as the earth goes halfway around the sun (two "astronomical units apart) we say the star is 1 parsec away. This defines a triangle where two of the sides are far larger than the third, which means as long as we measure our angle in radians we can use the small-angle approximation and say a parsec is one AU per arcsecond. If I search [1 parsec in meters] I get the correct answer of 3e16 meters: The interactive unit converter seems to be [...] --- First published: June 20th, 2026 Source: https://www.lesswrong.com/posts/BmqzjcD4tGvy3bim8/google-can-t-math-parsecs --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

I recently made a dumb (in retrospect) mistake that set me back a lot. Feeling upset and regretful, I spoke to an older family member who reassured me, "yeah, unfortunately there's no way around it; we have to experience these mistakes personally in order to learn from them". I thought, is that actually true? Can't we learn from other people's mistakes? After all, isn't that the whole point of studying history, or listening to other people's advice, etc? I'm sure that every mistake I could possibly make has been made by countless people before me and discussed in depth somewhere. This thought inspired me to try to see what I can learn about mistakes by reading about other people's experiences. I first tried looking at history for examples of famous instances of dumb mistakes made by smart people. I used LLMs and some scripting to (1) generate many examples of mistakes from different periods of history, and (2) summarize/extract/categorize them, so that I could browse by cognitive principle. It generated lots of interesting cases, such as: Ignaz Semmelweis dismisses need to persuade (Vienna, 1840s) discovers handwashing saves lives, then refuses to publish [...] --- First published: June 20th, 2026 Source: https://www.lesswrong.com/posts/J8L3HoPYAj52x9gCA/would-anybody-here-be-interested-in-a-mistake-postmortem --- Narrated by TYPE III AUDIO.

These are some personal notes taken and later dressed up a bit to make into a post. Dunno how much value is here for people already familiar with the AI Safety Ecosystem. Over several weeks in the spring of 2026 I attempted to map out the entire AI Safety ecosystem as a project for MATS Research. This entailed finding every organization working on AI Safety (whether it be via research, policy, pipeline, or other methods) and determining (or estimating) their headcount and annual spending. It's a snapshot, catching a period of time just before the incoming flood of 2026 funding. We worked almost entirely from public information. I believe MATS will be publishing the results in a formal report later this year. I will leave the unveiling of our findings to them. However while I was doing this research I stumbled across various orgs or tidbits that made me think “Huh, that's interesting!” I’m not an insider here (or at least, I’m a periphery insider if anything) so many of these things may be old hat to people in the know. But they were interesting enough for me to jot down for my own notes, so I’m sharing them [...] The original text contained 2 footnotes which were omitted from this narration. --- First published: June 19th, 2026 Source: https://www.lesswrong.com/posts/wWX9ecM5Q7TycpKyX/ai-safety-ecosystem-research-notes --- Narrated by TYPE III AUDIO.