Redwood Research Blog

Subtitle: If it transfers misalignment, we might get a misaligned model that's easier to incriminate. If it doesn’t, we might get a capable benign replacement model. Suppose we have a dangerous misaligned AI that can fool alignment audits, and distill it into a student model. Two things can happen: Misalignment doesn’t transfer to the student. If so, we get a fairly capable benign model, which we can use to perform tasks that we wouldn’t want a misaligned AI to perform. Misalignment transfers to the student. The student might also be worse than the teacher at hiding its misalignment (e.g., because it is less capable). If so, auditing the distilled model might give us indirect evidence of the teacher's misalignment. In a previous post we discussed the second possibility and proposed distillation for incrimination techniques: distillation methods that we hoped would transfer misalignment without transferring the ability to fool audits. In this post, we discuss the first possibility, and propose distillation for capabilities techniques: distillation methods that we hope will transfer capabilities without transferring misalignment. Thanks to Carlo Leonardo Attubato, Eric Gan, Aniket Chakravorty, Francis Rhys Ward, Anders Woodruff, Alex Mallen, Buck Shlegeris, Julian Stastny and [...] ---Outline:(02:07) Why distillation might transfer capabilities but not misalignment(03:10) Ideas for implementing distillation for capabilities(04:55) Distillation double bind: If distillation for incrimination fails, then distillation for capabilities is somewhat likely to succeed(05:38) Why distillation for capabilities and incrimination might both fail(05:44) Reason 1: Capabilities and misalignment might be too tightly linked(08:29) Reason 2: Context-dependent misalignment(09:51) Reason 3: If U became misaligned in the first place, why won't M become misaligned for that same reason?(10:34) How we can empirically study distillation techniques(11:35) Appendix: details on ideas for implementing distillation for capabilities(12:02) Basic ideas(14:17) Idea 1: Iteratively filter and distill(15:14) Idea 2: Inoculate against misalignment(16:08) Idea 3: Reduce cognitive slack(16:47) Idea 4: Distill and audit The original text contained 2 footnotes which were omitted from this narration. --- First published: June 18th, 2026 Source: https://blog.redwoodresearch.org/p/the-distillation-double-bind-distilling --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Subtitle: When is "increasing safety budget" a useful concept? I often use what I’ll call the “safety-usefulness tradeoff model”, which is: developers face a tradeoff between “safety” and “usefulness” of an AI deployment, and the developer has only limited willingness or ability to sacrifice usefulness for the sake of safety. This model assumes that developers choose whether to take safety-relevant actions based on their cost efficiency, i.e., the marginal safety gain relative to the cost. However, that is not necessarily true. In this post, I spell out different stories for how developers choose what safety-relevant actions to take, in order to clarify when this model is relevant and how strategies for reducing AI risk are affected when its assumptions don’t hold. The model suggests two ways a safety-concerned person can increase safety: Safety tech improvements: push out the Pareto frontier, so that any given level of usefulness reduction buys more safety than it would have previously. Safety budget increase: increase the extent to which the developer sacrifices usefulness for safety. On the cheaper end, this means implementing safety measures; on the more expensive end, it might mean refraining from training or deploying models whose risks [...] ---Outline:(04:17) Rushed reasonable developers(06:08) Limited political will(08:42) This model is unhelpful if developers don't trade efficiently between safety and usefulness(12:59) Overall thoughts(14:26) Appendix: Definitions of safety and usefulness in the rushed reasonable developer model --- First published: June 8th, 2026 Source: https://blog.redwoodresearch.org/p/efficient-tradeoffs-and-the-safety --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

We’d like to develop training techniques that work when applied to future misaligned AI systems. One strategy for studying proposed techniques is to test them on model organisms. However, model organisms built with common techniques are often fragile: we (and other researchers like Roger et al. and Ryd et al.) have observed them to stop misbehaving after untargeted training—training that doesn’t directly target the misbehavior. For example, we have observed that simple untargeted training methods like “train the model to talk like a pirate” is effective against many model organisms that we have created, including many replications of prior work like Hubinger et al., Greenblatt et al., and Ryd et al.. Fragile model organisms aren’t very useful for technique development: when a sophisticated technique succeeds on one, you can’t tell whether the technique is good or the model organism is weak. For instance, if you come up with some complicated technique for generating high-quality responses and find that SFTing on these removes the bad behavior, that may just be because almost any SFT would have removed it. This post identifies factors that make model organisms more robust to untargeted training. Our main findings are: Prompted model organisms [...] ---Outline:(03:11) Experimental setup(06:12) Main results(06:15) Result 1: Untargeted training removes misalignment from prompted model organisms without degrading other capabilities(12:32) Result 2: FWFT model organisms are more robust than LoRA model organisms(15:09) Result 3: Backdoor behavior and trigger affect robustness(16:37) Result 4: Password locking makes model organisms less robust(20:20) Conclusion(21:29) Factors that don't help much(21:54) 1: Training longer doesn't seem increase robustness past a certain point for our backdoors(22:33) 2: CoT-distilling doesn't increase robustness for our backdoors(23:07) 3: Using larger models doesn't substantially increase robustness for 2 of our backdoors(24:44) 4: There aren't clear trends for what types of backdoor triggers and behaviors are most robust(25:32) 5: SOAP Optimizer doesn't help much(26:35) 6: Weight Decay doesn't help(27:18) Appendix:(27:21) Appendix 1: Pirate SFT is better than QA SFT(27:45) Appendix 2: Are blue team methods wiping previous training?(29:30) Appendix 3: Backdoor list The original text contained 10 footnotes which were omitted from this narration. --- First published: May 28th, 2026 Source: https://blog.redwoodresearch.org/p/advice-for-making-robust-to-training --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Suppose we have a dangerous misaligned AI that can fool alignment audits, and distill it into a student model. Two things can happen: Misalignment fails to transfer to the student. If so, we get a fairly capable benign model. Misalignment transfers to the student. The student might also be worse than the teacher at hiding its misalignment (e.g., due to being less capable). If so, we might get indirect evidence about the teacher's misalignment by auditing the distilled model. In this post, we will discuss the second possibility, which we call incrimination via distillation. Specifically, we propose distillation methods that we hope transfer misalignment without transferring the ability to fool audits, and discuss why these techniques might work or fail. In a future post, we discuss the first possibility, and what distillation methods should be used when aiming to create a capable benign model. We’re excited for research that empirically tests and refines this approach; if successful, this technique could become a valuable part of alignment audits. How incrimination via distillation works Powerful misaligned AI models might not be auditable: they might pass alignment audits but still act on their misaligned drives when given the chance. [...] ---Outline:(01:32) How incrimination via distillation works(02:45) How we propose implementing incrimination via distillation(03:59) Auditability-preserving distillation(05:11) Misalignment-targeted distillation(06:34) Why incrimination via distillation might work(08:17) Why incrimination via distillation might not work(10:53) Conclusion The original text contained 2 footnotes which were omitted from this narration. --- First published: May 18th, 2026 Source: https://blog.redwoodresearch.org/p/incriminating-misaligned-ai-models --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Subtitle: My median guess: it's as good as a crystal ball that sees 2.5 months into the future. This post was drafted by Buck, and substantially edited by Anders. “I” refers to Buck. Thanks to Alex Mallen for comments. People who work inside AI companies get access to information that I only get later or never. Quantitatively, how big a deal is this access? Here's an operationalization of this. Consider the following two ways my knowledge could be augmented: I get a crystal ball that tells me all the information I would know n months in the future. I become an employee of a frontier AI company (like OpenAI or Anthropic), with access to all the private information I’d normally get from working at that company. How big would n have to be for me to be indifferent between these two options, from the perspective of learning things that are helpful for making AI go well? The answer is presumably different for me than for many readers, because I’m a reasonably well-connected researcher; I see published information and news from the rumor mill and I talk to researchers at frontier AI companies all the time. [...] ---Outline:(03:20) What do insiders know?(04:34) Safety work and corporate attitudes(05:54) Model capabilities(07:27) Algorithms and architecture(09:49) How will this change over time?(12:27) Conclusion The original text contained 4 footnotes which were omitted from this narration. --- First published: May 11th, 2026 Source: https://blog.redwoodresearch.org/p/how-useful-is-the-information-you --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Subtitle: Fitness-seeking is increasingly what misalignment looks like in practice—how should we respond? Current AIs routinely take unintended actions to score well on tasks: hardcoding test cases, training on the test set, downplaying issues, etc. This misalignment is still somewhat incoherent, but it increasingly resembles what I call “fitness-seeking“—a family of misaligned motivations centered on performing well in training and evaluations (e.g., reward-seeking). Fitness-seeking warrants substantial concern. In this piece, I lay out what I take to be the central mechanisms by which fitness-seeking motivations might lead to human disempowerment, and propose mitigations to them. While the analysis is inherently speculative, this kind of speculation seems worthwhile: AI control emerged from explicitly taking scheming motivations seriously and asking what interventions are implied, and my hope is that developing mitigations for fitness-seeking will benefit from similar forward-looking analysis. Fitness-seekers are, in many ways, notably safer than what I’ll call “classic schemers”. A classic schemer is an intelligent adversary with unified motivations whose fulfillment requires control over the whole world's resources. Meanwhile, fitness-seeking instances generally don’t share a common goal (e.g., a reward-seeker only cares about the reward for its current actions), and many fitness-seekers would be satisfied1 with modest-to-trivial [...] ---Outline:(03:50) Overview(11:32) The basic reasons fitness-seekers might be safer than classic schemers(16:01) Four mechanisms for risk and their mitigations(16:46) Potemkin work(22:13) Instability(27:49) Manipulation(31:25) Outcome enforcement(36:52) Cross-cutting mitigations(37:18) Deals(40:05) Control(44:45) Alignment(44:48) Preventing fitness-seeking from arising(48:51) Making any fitness-seeking motivations safer(51:32) How does online training change the picture?(55:09) Overall recommendations(59:12) Conclusion The original text contained 18 footnotes which were omitted from this narration. --- First published: May 1st, 2026 Source: https://blog.redwoodresearch.org/p/risk-from-fitness-seeking-ais-mechanisms --- Narrated by TYPE III AUDIO.

Subtitle: Eliciting long-term forecasts from myopic fitness-seekers. We’d like to use powerful AIs to answer questions that may take a long time to resolve. But if a model only cares about performing well in ways that are verifiable shortly after answering (e.g., a myopic fitness seeker), it may be difficult to get useful work from it on questions that resolve much later. In this post, I’ll describe a proposal for eliciting good long-horizon forecasts from these models. Instead of asking a model to directly predict a far-future outcome, we can recursively: Ask it to predict what it will predict at the next time step, Use its prediction at the next time step to provide intermediate rewards, Finally reward using ground truth at the last step. This lets us replace a single distant forecast with a chain of short-horizon forecasts, each verifiable shortly after answering. I call this proposal recursive forecasting. It does have limitations: for example, it requires that developers maintain control over the reward signal at least until the final step, which makes it most useful for forecasting events that resolve well before developers are disempowered (if they are). This post was primarily [...] ---Outline:(01:40) The default long-term forecasting behavior(04:08) Recursive forecasting(07:08) When is recursive forecasting helpful?(07:12) When we have access to (somewhat) robust ground truth rewards(09:44) When the AIs forecast doesnt substantially affect the resolution(11:52) When forecasts arent used as optimization targets(12:52) When we credibly inform the AI of the setup(13:54) Appendix A: Comparison to temporal difference learning(16:02) Appendix B: Error tolerance The original text contained 9 footnotes which were omitted from this narration. --- First published: April 28th, 2026 Source: https://blog.redwoodresearch.org/p/recursive-forecasting --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Subtitle: Third-party experts should assess defenses against tampering and theft — and publish high-level findings. AI companies should get third-party security experts to assess (and possibly also red-team/pen-test) their security against key threat models and then publish the high-level findings of this assessment: the extent to which they can defend against different threat actors for each threat model. They should also publish who did this assessment. The assessment could be commissioned by AI companies, or performed by a third-party institution that AI companies provide with relevant information/access. There are presumably lots of important details in doing this well, and I’m not a computer security expert, so I may be getting some of the details wrong. This is a relatively low-effort post in which I’m mostly trying to raise the salience of an idea. (I don’t make a detailed case or spell out all the details here.) Thanks to Fabien Roger and Buck Shlegeris for comments and discussion. I suspect the controversial part of this claim is that they should make the high-level findings public. Publishing a summary of which threat actors you’re robust to (for each relevant threat model) shouldn’t meaningfully degrade security against the threat actors we [...] The original text contained 4 footnotes which were omitted from this narration. --- First published: April 27th, 2026 Source: https://blog.redwoodresearch.org/p/ai-companies-should-publish-security --- Narrated by TYPE III AUDIO.