Doom Debates!

Asian Dad Energy is a former big tech developer with an Ivy League education. We have a lot in common. We’re both:* Software engineers;* New-(ish) YouTube creators covering AI;* Dads with a lot of existential angst.But while I dread AI extincting humanity, Asian Dad has other concerns. We debate the risk of the AI bubble popping, a “techno-feudalism” scenario, and whether AI will outperform a human CEO in the near future.Let’s find out where he gets off the Doom Train™.Timestamps00:00:00 — Cold Open00:00:54 — Introducing Asian Dad Energy00:03:36 — Asian Dad’s Career in Tech00:09:06 — Getting Laid Off: The Job Market Is a “Hellscape”00:12:29 — Software Engineering in 2026 vs. 202100:17:03 — Why Are They Investing Trillions in AI?00:24:37 — Debate: AI Is Different Than Other Tech Bubbles00:33:54 — Debate: AI Is Not Truly Intelligent00:41:12 — Intellidynamics: Judge AI by Its Outputs00:48:53 — Debate: Could AI Outperform a Human CEO?00:56:33 — Asian Dad: AI Can’t Do Genuine Innovation01:08:14 — What’s Your P(Doom)?™01:13:17 — “Are You Calling Me a Scammer?”01:14:41 — Asian Dad’s Mainline AI Doom Scenario01:20:53 — Will AI Wipe Out Humanity by 2050?01:29:29 — Why Is AI Doomerism “Getting Out of Control”?01:33:11 — Debate: AI Lacks Agency01:42:13 — Asian Dad Warns of “Techno-Feudalism” & Transhumanism01:48:12 — Wrap-UpLinksAsian Dad Energy (YouTube channel) — https://www.youtube.com/@AsianDadEnergyAsian Dad Energy, “AI Doomerism Is Getting Out of Control” — https://www.youtube.com/watch?v=8bnzaG9QRG4David Deutsch, The Fabric of Reality (Amazon) — https://www.amazon.com/Fabric-Reality-Parallel-Universes-Implications/dp/014027541XRobert Wright, The God Test (Amazon) — https://www.amazon.com/God-Test-Artificial-Intelligence-Reckoning/dp/1668061651OpenAI model disproves the Erdős unit distance conjecture — https://openai.com/index/model-disproves-discrete-geometry-conjecture/METR, Measuring AI Ability to Complete Long Tasks — https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Dr. Holly Elmore, executive director of PauseAI US, is done giving AI insiders the benefit of the doubt.In this spicy episode, Holly calls out a number of respected figures in the AI safety community for allegedly “selling out” the cause: Effective Altruists, Anthropic & OpenAI employees, Eliezer Yudkowsky, Zvi Mowshowitz—and even yours truly! Holly isn’t afraid to psychoanalyze her subjects, which I push back on, but I can’t deny she has a track record of pushing the Overton window and surfacing uncomfortable truths.Does Holly’s approach risk burning bridges with allies for little benefit, or does AI safety need a moral conscience willing to call out the rot nobody else will?Timestamps00:00:00 — Cold Open00:00:48 — Introducing Holly Elmore00:02:20 — Why Holly Moved to DC00:06:03 — What’s Your P(Doom)?™00:07:18 — How We Got Here: EA, Rationality, AI Alignment00:16:32 — “If Anyone Builds It” and LessWrong’s Missing Mood00:21:47 — Holly Supports the Government’s Fable Crackdown00:33:39 — Beef: Dean Ball (OpenAI)00:53:09 — Beef: Eliezer Yudkowsky00:56:14 — Beef: Zvi Mowshowitz01:11:50 — Beef: Liron Shapira!01:16:15 — Beef: Nirit Weiss-Blatt (Dr. Techlash) Being “Bad Faith”01:19:31 — Anthropic Took Over EA01:22:04 — Holly’s Basilisk: What If Building Superintelligence Becomes a Crime?01:32:35 — Beef: Amanda Askell, the “Mother of Claude”01:39:05 — Beef: Daniela Amodei, Co-Founder of Anthropic01:40:23 — Beef: Joe Carlsmith, Member of Technical Staff at Anthropic01:49:39 — Beef: Open Philanthropy / Coefficient Giving, Ben Goldhaber01:54:28 — Wrap-UpLinks* Support PauseAI US: https://www.pauseai-us.org/* Holly on Twitter/X: https://x.com/ilex_ulmus* Joe Carlsmith’s “Leaving Open Philanthropy, going to Anthropic” post: https://joecarlsmith.com/2025/11/03/leaving-open-philanthropy-going-to-anthropic/* Dean Ball joins OpenAI announcement (FAI post): https://www.thefai.org/posts/dean-ball-joins-openai-as-head-of-strategic-futures---Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Bestselling author Robert Wright has landed on a sobering conclusion: on our current trajectory, AI dystopia is likely within eight years.We get into his P(Doom), his case for slowing down, why racing China could backfire into authoritarianism here at home, and why he thinks evolution itself might have a purpose.Bob is the author of The Moral Animal and Nonzero, a Pulitzer finalist, and hosts the excellent Nonzero podcast.Get his new book, The God Test: Artificial Intelligence and Our Coming Cosmic Reckoning.Timestamps00:00:00 — Cold Open00:00:50 — Introducing Robert Wright00:03:21 — What Is The God Test?00:04:59 — AI as Evolutionary Force, Not Normal Technology00:07:34 — AI Heaven vs AI Hell00:09:09 — Bob's Background: From Hinton in 1983 to Today00:16:09 — What's Your P(Doom)?™00:20:09 — The Mainline Scenario: Dystopia in 8 Years00:22:04 — Destabilization and the Road to Authoritarianism00:25:35 — Why Racing China Backfires00:28:37 — Slowing Down AI + Would Bob Support PauseAI?00:32:38 — Intellidynamics and Why AIs Collude00:44:44 — Hinton's Warning and How LLMs Discover Meaning00:53:48 — Symbol Grounding and Killing Searle's Chinese Room01:02:03 — Let's Talk About God01:10:14 — Orthogonality, Power, and the Concentration Problem01:16:31 — Wrap-UpLinksGet Bob’s new book, “The God Test” — https://www.simonandschuster.com/books/The-God-Test/Robert-Wright/9781668061657Follow Bob on X — https://x.com/robertwrighter“If Anyone Builds It, Everyone Dies” by Eliezer Yudkowsky & Nate Soares (referenced) — https://ifanyonebuildsit.comJoin the Doom Debates Discord — https://discord.gg/g2X8h5UCrUDoom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

The US government banned Claude Fable like a clown show. But I’m happy about it. Here’s my appearance on the AI in the AM livestream (recorded Tue, June 16) with Nathan Labenz and Prakash Narayanan, where I explain why this was good for society. The government’s action shows that we can actually pause AI, a precedent that matters way more than the boneheaded rationale behind Fable’s ban.We then get into a mini-AI doom debate where Prakash defends the AI optimist’s worldview. I push back on how much I expect superintelligence to reshape the planet.Timestamps00:00:00 — Why I’m Happy the Government Banned Fable00:11:18 — Why We Lose Control of AI00:20:27 — The Icarus Graph: Heaven, Then Hell00:22:33 — Get Ready to Pause00:25:51 — What Exactly Should We Ban?00:27:42 — Wrapping UpLinksThe Cognitive Revolution podcast with Nathan Labenz —https://www.cognitiverevolution.ai/Nathan Labenz on X — https://x.com/labenzPrakash Narayanan on X — https://x.com/8teapi AI in the AM on X — https://x.com/AI_in_the_AM TranscriptWhy I’m Happy the Government Banned FableNathan Labenz 00:00:08On that note, perhaps a fitting transition to our first guest of the day. Liron Shapira is the host of Doom Debates, a YouTube channel where he has found quite a bit of success in bringing people from really all walks of life on to discuss how scared we should be of AI and just how high our P(Doom)s should be.So Liron, I’m excited to get your take on recent events. Obviously, never a dull moment in the AI space. The thing that caught my eye most from you over the last few days was when you said basically, “I want the government to be willing to take action on AI issues so badly that even if this whole Fable ban turns out to be totally boneheaded and unjustified in some local sense, you support it, or you’re at least happy to see it because it breaks the ice and now nobody can ever tell you again that it’s impossible to imagine the government doing anything.” So give me the double click into that story, and then I’m sure we’ll have plenty of interesting directions to go.Liron Shapira 00:01:17Hey, Nate. Great to be here. That’s exactly right. I’m a simple man. I see AI getting paused. I feel good about breaking the Overton window. The government can do it. It’s that easy, guys. This is a precedent. Overall, I’m happy. You can talk about the nuances. It was done like a clown show. It was done for bad motives. It doesn’t really consider China or a treaty or anything. There’s a lot of problems, but I’m really happy about smashing the Overton window, where now tech folks don’t think that they’re in a bubble or they’re untouchable. It happened, guys. And we can only go from here.Prakash Narayanan 00:01:53I actually agree with you, because I think it was a little bit delusional for tech to feel that it wasn’t gonna get touched. And the government just has so many small and large ways to effectuate its power. So it was not that surprising to me that they kind of went through left field and went with the export control rather than anything else.But it also strikes me that as they exercise this, we start also to go into kind of what we wanted to avoid. It’s a little bit of a small tyranny. I think several people on the timeline have commented, Dean Ball has commented, this kind of unstructured regulation looks kind of selective and vengeful almost, and it starts putting you in this zone where tech people start to mistrust the government because you also see — I think there’s a lot of narratives on the timeline which are being leaked, “sources close to,” “sources familiar with,” and as they get leaked, it’s not very certain whether those things actually happened. Would someone actually attest to that in front of Congress? Very unclear.And we’ve also seen this kind of behavior from the administration in other affairs as well, where you have multiple conflicting narratives. It’s happening with the Iran war right now, where it’s not even clear to Congress what the deal is. You have different people saying the deal is a different thing.Prakash 00:03:46So where do you think that puts us?Prakash 00:03:49It’s great that it’s happening. I understand you feel it’s great that it’s happening to AI right now, but does that put us in a position where it’s detrimental to the body public at large?Liron 00:04:03Well, I think your analysis — you’re weaving together a few factors, but I think the elephant in the room, I hate to get political because when it comes to President Trump, he’s a mixed bag for me. I don’t have Trump derangement syndrome. I don’t love everything he does. I don’t hate everything he does. But I think the common thread with Trump is it’s just a mess. It’s not disciplined. And I think we’re definitely seeing that on display right now. I would argue we’re seeing that on display in the Iran war.Previous administrations, there was just more pressure to have logical consistency, some kind of narrative. And this is another one of those cases where you see people in his administration saying all these justifications why something happened, but then the next day it’s, “Oh, it happened for this reason.” “Oh, Dario did this, he wasn’t responsive to us. That’s why we’re doing it.” And then Anthropic’s like, “Oh, no, he was responsive to us.”And it’s still not clear exactly what did Fable do that was so dangerous, because Anthropic is like, “Oh, this jailbreak is nothing special.” And the Trump administration’s like, “Oh, well, our secret source,” Amazon or whatever, “they’re telling us that it is dangerous.” So I hate that it’s a clown show. I hate that this is how humanity’s operating. I’ll take the win that it’s a pause, but I also think it’s probably time for a new administration.Nathan 00:05:19One thing I think you should share with folks who may not know you — I think, like many people that are worried about big picture AI safety issues, your background is one of being a, I would say, techno-optimist libertarian for the most part, right?Liron 00:05:38Totally.Nathan 00:05:39So how much are you personally missing Fable, and what was your kind of initial impression of it that I know is superseded by your big picture analysis? But how did your first few days of Fable unfold and how bad are your Fable withdrawals?Liron 00:05:56My Fable withdrawal is not too bad because I’m not pushing it to the limit. I’m not doing hard research. My main use case for frontier AI is Claude Code on a regular production app that has thousands of daily users, but it’s not super hard to serve. So my experience using Claude Code is actually that Opus 4.8 or even going back to Opus 4.6, if I had to go back to Opus 4.6 fast mode, I’d actually still be pretty happy. That’s my threshold of where I’m happy.So when Fable came out, they said it’s more powerful but it’s slower. But typically, when I give Opus 4.8 a big project, I am pretty happy with the results. I don’t mind giving it a few comments and letting it work again. If anything, I would actually prefer more speed to more quality for my use case at this time. But then again, I’m not on the frontier of computer security. I’m not trying to do anything super novel. So yeah, I’m not experiencing withdrawal right now.Nathan 00:06:50Interesting. Did you notice — I mean, for me, it, I wouldn’t say it was so much coding, but in the more general purpose knowledge work type stuff that I do, I just found its outputs to be notably better, and I was kind of like, “Oh, this is gonna change how I work. I’m not gonna need to be so precious about my language anymore. I need to actually recalibrate how I think about authorship or shared authorship with AI.” Did you experience any of those kind of feelings?Liron 00:07:21I can’t say I did firsthand, but I’ve been reading a lot of other people’s accounts, and my vague impression of what’s going on with Fable is, obviously the time horizon is blasting forward. If you talked to me a few months ago, I would’ve said the Achilles heel of all of these AI models is robustness. They’ll do a bunch of work and they’ll mostly get it right, but they’ll make a few mistakes, and the mistakes pile up because the error correction isn’t robust, and that’s the source of the time horizon problem. They pile up too many errors, too many lethal mutations or whatever.It seems like Fable subjectively has more of this meta process where it can go back of itself and be like, “Oh, no, let me fix this, let me fix this.” So its time horizon is longer, and this also relates to some buzz I’ve been hearing from random people who are on the frontier, which I don’t quite include myself on, but I’ve heard this from Anthropic employees and I think Gary Tan was early to this — this idea that you have multiple roles. You have a company of agents, and they can reflect on each other, and one of them’s whole job is quality control, and one of them is the project manager. I think this idea of the swarm of agents might get us past the robustness hurdle and unlock a new time horizon.Nathan 00:08:35And how scary is that for you relative to your kind of big picture fears? I mean, you can give the briefcase for doom if we don’t change course better than I can. But I’m also interested in how far you perceive us to be now on that curve.Prakash 00:08:58Mm-hmm, mm-hmm.Nathan 00:08:58And also what in the system card, or the sort of interesting new behaviors that we’ve observed from the Mythos Fable models, are kind of most catching your attention or you think are underappreciated by people who are not so focused on watching these leading indicators as you are.Liron 00:09:23I’m glad you’re asking about the end state because everybody’s got their head two inches in front of their nose. Everybody’s like, “Oh my God, more tokens. What’s gonna happen next? Are AIs going to replace this role?” And it’s like, yeah, these are interesting questions for the next 12 months. But to me, it’s pretty clear that we can predict some properties of the end state. The end state will involve the human brain being worthless at thinking. I just don’t think we’re good at thinking on any dimension.So yes, we still have a lead. Even though AI can do all this stuff, it can’t do everything better than us. I think we can pretty robustly predict that this is going to stop. Will it take more than a year? It might. So we can talk about, do we need another paradigm? But if we fast-forward 10 or 20 years, it’s probably more than enough time to make this happen.I just think AI will have all the skills, and it’ll be overdetermined. And I also think that the universe is malleable to AI. So I think not only can AI write code, I think that AI will be able to manipulate atoms really well. That’s right, I went there. I think nanotechnology is physically possible.People tend to just not dream big enough in terms of what this universe allows when you have a grown-up intelligence operating it. This universe is not really meant for humans to operate. We just kind of punch above our league because we try really hard. But a good mental model of AI is it can just make a blueprint for where it wants the atoms to go, and then they’ll mostly go there — minus some heat exhaust.Prakash 00:10:49I actually completely agree with you, but I wonder why I end up taking the optimistic viewpoint of that, that that capability will be used to expand humanity’s frontier, and why you have the exact opposite viewpoint that it is a doom scenario.Why We Lose Control of AILiron 00:11:18I would turn it around on you. What makes you so optimistic? Because I think the key moment for me is I just think we’re going to lose control. We’re gonna have this AI which is incredibly powerful, but the linkage between what it wants to do and what we would want it to do, I think at some point we’re just going to break the link. There’s too much power, and there’s no natural persistent link.Getting the link to hold as it’s scaling up and going through all these transformations and copying itself — this is what people aren’t imagining. Making new versions of itself from scratch. So all these transformations are happening. All these successor AIs are happening.Prakash 00:11:53Mm-hmm.Liron 00:11:53And you think the human species is back here still holding some kind of leash? I just don’t really see it.Prakash 00:11:58So again, to push back — why is that even important that humans maintain control? Because let me give you an example.Liron 00:12:09Yeah.Prakash 00:12:09The average — let’s say about 50% of humanity is probably still doing kind of subsistence or near-subsistence farming, or small shopkeepers, et cetera, across the world. How much control does the average farmer in Uganda have? Does that person have control? And are we judging — what relationship does that person, who maybe got a smartphone two years ago, what relationship does that have to control? What aspect of control does that particular human have, and how does this relate to that particular human?Liron 00:12:50Mm-hmm. Yeah, yeah.Prakash 00:12:50What aspect of control does that particular human have, and how does this relate to that particular human?Liron 00:12:55So you’re leaning on the convenient fact that when there’s a farmer in Uganda, their territory is far away. It’s causally distant. So somebody in the United States, we don’t have a button available to us. Even though we’re a rich country and we have more control over the world per person, we have more power, we don’t have a convenient button we can press that will go harvest the atoms over in Uganda and then deposit $1,000 in our bank account. We don’t have that button.It is ultimately a matter of intelligently configuring atoms to get there. It’s kind of like a code base. Since I got Claude Code, my code base is a lot cleaner because I’m running all of these cleaning processes. I’m investing more in the code because it’s cheap for me. Similarly with Uganda — that’s kind of just unused resources from the AI perspective. Once we get enough power, it’s, “Okay, yeah, let me harvest all the land in Uganda. Who’s gonna stop me?” And then it only comes down to me caring for the Ugandan farmer. Okay, I’m not gonna kill him. But the AI might not care.Prakash 00:14:11That does not make sense at all because they are linked to the market economy. And in fact, what we do end up doing in the United States is we press a button, and that’s called a buy button, and that buy button ends up harvesting the cotton in Uganda. That’s literally what we do. We go to Amazon and you press the Buy button, and as a consequence of pressing that Buy button, and through the linkages of all of these entities, which are all like small mini AGIs — all these companies in the middle.Hundreds of these companies. The logistics firm, the firm supplying the fertilizer, the firm doing the milling. All of these firms in the middle, and then we get the product in our backyard. So I basically resist this idea that that button doesn’t exist. That button exists. We push it every day. It’s the Amazon button.Liron 00:14:40Well, this is kind of the common argument of Ricardo’s law. It’s like, oh, we’re trading with them, right? So why won’t the AIs trade with us? But we don’t have the button called Harvest the Atoms, because that would be a better button if we had it.Prakash 00:14:55No, I’m pushing back here because you’re saying that we don’t have control, and you’re saying that these AIs will have that control, and we’re — right now, humanity does not have that control. I’m saying humanity already has that control on that particular farmer in Uganda through the market economy. And —Liron 00:15:15Oh.Prakash 00:15:15This does not really differ whether you replace the market economy with a buyer who happens to be an AI. It’s basically the same, right?Liron 00:15:24I think you may not be realizing how many preconditions are necessary in order for a market to be the relationship that emerges. Because we as humans, like you said, we’re used to being in this kind of market trading relationship with other humans.Prakash 00:15:37Yeah, yeah.Liron 00:15:37But you might also note we’re not in a market trading relationship with the other animals. Cows — we don’t really let them buy turf. We kind of decide where the cow’s gonna live. So I think you might end up with that kind of relationship with AI versus human in terms of the power differential.Prakash 00:15:54So this is a question of control. It’s a question of, do humans lose control to AI?And I was pointing out that that particular human in Uganda does not have much control anyway, and there’s no real control loss in that sense. So I just don’t see how — 50% of the global economy consists of basically farmers who are trading, and I just don’t see how much control they have to lose, and I don’t see why this matters to them at all.Liron 00:16:25Well, yeah.Prakash 00:16:25I think it matters to, let’s say, the leadership at a big tech firm. Yes. It definitely matters to them because they have control over the tech sphere. I don’t see how this matters to most of the global economy or most humans in general, actually.Liron 00:16:41Well, so you’re basically asking what’s the discontinuity or the disanalogy between the farmer today and that specific farmer after superintelligence.Prakash 00:16:49Yes.Liron 00:16:49Very simply — the benefit of harvesting their resources or destroying their resources will exceed the cost. Right now, why don’t I personally take the farm — besides caring about... Why doesn’t the US take over Uganda? It’s just not high value to us to take over that land. We think that our intellectual property and what we can build here in the US is already much more interesting, and also, we believe in moral rights. That’s a factor too — sovereignty.But from the AI’s perspective, you can also think about side effects. Let’s say the AI wants to turn the planet into a factory to produce probes so it can participate in the land grab across the galaxy, across the universe. Okay, so if you’re building probes, Eliezer Yudkowsky pointed out you wanna run the Earth really hot. That’s your preferred temperature, and the oceans will probably boil away. As long as you can radiate away the heat, the temperature is good. But that’s going to be a higher temperature than humans can live in open air. So suddenly you can’t just be chilling as a human. It really is up to the AI to decide who gets to live where.Nathan 00:17:53So there’s a couple parts of that story that I think people just have extremely different intuitions about. One — and even including in the camp of people who are very worried that we’re gonna end up in a bad outcome — one is how long does it take? There’s sort of the fast takeoff foom scenario where we all drop dead suddenly, and then there’s the kind of gradual disempowerment vision where this maybe happens over not a super long period of time, but through a bunch of locally sensible decisions.Yeah, I mean, I guess we should let AI run our company now because it is gonna probably do a better job than we are, kind of played out everywhere. Next thing you know, it really is kind of AIs running the show and we’re maybe sort of okay with it at first, but there’s a lot of potential for drift over time.Of course, there’s also the question of should we be optimistic that we’ve kind of put Claude or whatever into the benevolent basin, that it might just kind of stay in? Or are there a lot of ways that it’s gonna inevitably roll out of the benevolent basin into some other even lower point in its loss landscape that we’re not gonna be so happy with?We don’t have time probably for your full arguments on all those points, but maybe at least tell us where you are on a couple of those big questions, and then use that to sketch out —Nathan 00:19:27We now have Anthropic and OpenAI saying that they’re at least open-minded to some sort of coordinated slowdown, not like the one they just got handed to them by the Trump administration. They don’t want that kind of haphazard pause button. But they’re into something, or they’re at least open to something. And that of course leaves us with the question of, well, what should that look like? What exactly are we pausing? What exactly are we banning?And there I really don’t know too much about your position in terms of concretely what should be stopped, what should be slowed, what should have limits, what limits should maybe escalate or even be reduced. I know there’s been proposals for reducing compute limits over time. And do we end up in your vision figuring this out and getting to an AI win, or is there no AI win in your mind possible and we just kind of have to steer some other direction for humanity’s future? I know there’s a lot there. You can use the rest of the time if you need it.The Icarus Graph: Heaven, Then HellLiron 00:20:28Yeah, okay, sure. So my worldview, my outlook right now, it’s what I call the Icarus graph. I feel like nobody gets this. Everybody’s like, “No, I think the world is good. It’s gonna go this way.” And some people are like, “No, we’re terrible. Enshittification. It’s gonna go this way.” And I’m like, no, no, it’s Icarus. We’re gonna fly closer and closer to the sun, it’s gonna be great, and then we’re gonna do a 180 degree turn and plummet down to hell. So basically we get a taste of heaven, and then we get hell.So you have to ask me then, okay, so where on the Icarus graph do we stop? And it’s a brutal question because every day I’m enjoying the flight as much as the next person. Yeah, give me the next Claude. Make my code faster, great. Let it help my business run better. Make me better AI videos.So there’s no natural point in terms of when it feels right to stop. I just think it’s important to stop before capabilities get to a runaway point, and we’ve been kind of frog-boiled to be, “Oh, each model comes out and we’re doing great.” If we could stop the clock now, would I turn back the clock? Would I lose Fable? Would I lose Opus? No, I’d keep it all. I still think we’re playing shuffleboard, we’re playing Icarus. So far so good. Should we bet again? Should we keep betting until we lose? It’s a crazy tough question.I think the Eliezer Yudkowsky — yeah, the turkey graph kind of... Yeah, yeah, exactly. Although, the only difference with the turkey graph is each day of the turkey’s life is actually better. Not only is it living longer, it’s actually living better and better. So the turkey is really happy with its life.So I think the Eliezer Yudkowsky MIRI position, which I agree with, is just, we don’t know when to stop, so let’s get ready to stop. At the very least, let’s get ready. I would probably stop today. I would stop and I would be bummed. I saw a food influencer say this about how she eats chocolate basically. “Yep, I just ate this chocolate and now I’m bummed.” That’s what you gotta do. Don’t reach for another chocolate. Just sit there and be like, “This is the prudent place to stop right now until we have any idea of some kind of theoretical method by which we understand what a superintelligence wants to do and what an equilibrium state of a superintelligence looks like.”That’s actually something MIRI was trying to study, identifying equilibriums that are plausible for superintelligences. There’s actually a rich vein of theory there that’s highly neglected today. Let’s do some theory there. Maybe then we can unpause. I think that’s gotta be the best plan. And so I think the number one leverage point here is just repeating, “Get ready to pause.” And like you said, OpenAI and Anthropic, they said it. They said, “Let’s try to get ready to pause.” So I would love to see more people saying it because it really has to be a giant groundswell.Get Ready to PausePrakash 00:23:01I mean, if you take a step back and you look at, over the next two years, about a trillion, one and a half trillion dollars of CapEx being spent, and then you calculate the return on equity required in order to make that CapEx worthwhile or to repay that, it basically puts you in a position where if you pause, you go into a depression.So you have to have — you’re gonna have to deliver the economic growth that this one and a half, two trillion dollars has been spent on. And that indicates that you’re gonna have to grow in that 2027, 2028, 2029 period. You’re gonna need to see that growth or else we go into depression. And when you offer the choice to the public or policymakers, “Hey, you can pause because of this vague and unspecified claim of disaster in the future” —Or we can — and in addition to that, a guarantee of a depression, because those GPUs cannot be used for anything else besides compute. And that compute that is useful is AI compute. And you give them a choice between a depression state, your 10 largest companies basically getting their stock price slashed by 90%, and this vague and unspecified claim of disaster.I think at this point, maybe it would have made sense — and you guys were on this way, way before, and not enough policymakers were listening. But at this point, it is a very hard sell to make this case to policymakers.Liron 00:24:46Well, I agree with you and I disagree. I agree with you because, like I said before, yes, we are all going to be bummed. Even forget about the economics — the scientific breakthroughs, the medical breakthroughs. Nate was talking about his son. There’s no decelerationists in the hospital ward. I totally agree, and I’m sure the next time I get sick or an injury or whatever, I’m gonna want the latest frontier model.So I’m going to be bummed. The only thing I’ll say is when you then mix in the economic element and you say, “Oh, we’re gonna have a depression,” I actually think that is unnecessary. You don’t really need to mix that in because, first of all, I don’t think we’re literally going to be in a depression. Realistically we’ll be fine. It’s just money that’s already been spent. We still have plenty of resources.But the other factor is, I actually think that all of these valuations and all of the build-out that’s happened so far is actually just perfectly congruent with using the AI we have today for the kind of applications we’ve already discovered today. That should actually be fine to create more trillions of dollars of value. I don’t even think that the market caps reflect the actual singularity. So I think we’ve actually got a glide path if we wanna pause.Nathan 00:25:51Do you think that we could localize the pause a little bit more to, for example, some sort of operationalizable ban on recursive self-improvement? I know you did an episode recently with someone who — was it Steven Casper — who was advocating for a pause on all research. And you were, I think, kind of like, “Oh, I’m not sure if we should try to pause theoretical work on control or interpretability or what have you.”And you’re also saying we can just run the inference. So I think that’s a key thing to keep front and center — my understanding of your proposal does not call for a shutdown of ChatGPT as it exists today.How narrow — if there’s a Pareto curve on the pause frontier, how much value of a pause can we get with how narrow of a pause in your mind?Liron 00:26:54Vaguely I would say frontier AI capabilities. But it’s like, if you’d asked me three years ago, I’d have been like, “Oh, might as well pause now. It’s prudent to pause now.” But then we kind of gambled and won. So from my perspective, we keep playing shuffleboard. We keep doing Icarus. We keep going higher and winning, kind of, but we’re also getting closer and closer to the point of no return.So even though it feels like we’re winning now, we’re also killing our ability to pause ‘cause we’re so close to the point of no return — the last breakthrough where after that the AI takes over the research, and then we’re really screwed. So basically I think roughly a good policy is, okay, no more frontier capabilities upgrades for a while. It’s just too dangerous, and I know that concept is hard to communicate to people when every day life is getting more awesome. I think we’re in a screwed situation, but that’s just what I think is prudent.Wrapping UpNathan 00:27:42Would you recommend any recent Doom Debates for folks who wanna go deeper on your ideas? Where do you think this was kind of best hashed out, either recent or perhaps near future on the feed?Liron 00:27:55So we’ve just been discussing policy, and that’s really not even my wheelhouse, so there’s no really strong policy Doom Debates. But if you just wanna know the lay of the land from my perspective in terms of the next paradigm that’s coming and why I think it’s gonna break the trend and it’s not just gonna be another nice LLM — it’s going to be a reinforcement learning monster that takes over the world — check out my recent episode with Dr. Steven Burns. It’s one of the most popular episodes because he’s an extremely deep thinker, and he really lays out the coming future.Nathan 00:28:24Perfect. Thank you. Liron, keep fighting the good fight, man. I appreciate your tireless energy on this, and while I’m not maybe quite as high on the P(Doom) scale as you are, I certainly am not a dismisser. And I think you’ve done a valuable job for me personally in terms of pushing me to be more consistently candid about just how scary I think the situation actually is, even though obviously there’s a lot of upside to everything that’s been created as well. So I appreciate that from you, and I’ll continue to follow the Doom Debates for the latest, and I encourage other people to check it out as well.Liron 00:29:07Thanks, Nate. Thank you, Prakash. Always a pleasure, guys.Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe