Conspicuous Cognition Podcast

The political scientist Alexander Kustov recently published a Substack post with a provocative claim: that AI can already do social science research better than most professors. The post went viral. It attracted more than a million views and over a thousand responses, many of them very angry. (Some people even demanded that Alex’s university fire him.)In this conversation, we talk about this controversy and the claims that triggered it, including:* What agentic AI tools like Claude Code and Codex can already do for research, from coding and data analysis to literature reviews, translation, and brainstorming, and why only around 20% of quantitative social scientists currently use them.* What best predicts whether researchers adopt or reject AI: ignorance, openness to experience, methodological background, or the awkward role of self-interest.* How much published academic research is genuinely mediocre, and whether the cause is laziness, lack of skill, or a broken incentive structure, with a detour through the replication crisis and some high-profile fraud cases.* Whether AI will raise the quality of research or simply flood the literature with more slop, and what journal editors could do about it.* Whether AI can be genuinely creative or only recombine what already exists, by way of Margaret Boden’s three kinds of creativity, Thomas Kuhn on paradigm shifts, and AlphaGo’s “Move 37”.* The fight over AI writing and detection tools like Pangram, and why current disclosure norms end up punishing the honest.* The angry response to Alex’s series, and what is really driving reflexive opposition to AI among academics.Conspicuous Cognition is a completely reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.Links and further reading* Alexander Kustov — Alex’s homepage, with an overview of his research on immigration, public opinion, and effective governance.* Popular by Design — Alex’s Substack on public opinion, persuasion, and the politics of getting good ideas adopted.* Academics Need to Wake Up on AI — followed by a Part II and Part III* Pangram — the AI-detection tool discussed at length, which labels text as human, AI-assisted, or AI.* AlphaGo versus Lee Sedol — the 2016 match, including the famous “Move 37” that Henry raises as a candidate for genuinely transformative machine creativity.* Margaret Boden — the cognitive scientist whose distinction between combinational, exploratory, and transformative creativity frames part of the discussion.* The Structure of Scientific Revolutions — Thomas Kuhn’s account of normal science and paradigm shifts, referenced in the exchange about AI and discovery.* “AI Is a Better Researcher Than You” — The Chronicle of Higher Education‘s account of the controversy around Alex’s series.Transcript* Please note that this transcript is lightly AI-edited and may contain minor mistakes. Dan Williams: Welcome back. I’m Dan Williams, and I’m back with my co-host, Henry Shevlin. Today we are honoured to be joined by Bluesky’s favourite academic, Alexander Kustov. Alex is a political scientist at the University of Notre Dame and the author of one of my favourite Substacks, Popular by Design. His primary research is on immigration and public opinion, but that’s not really what we’re going to be talking about today. We’re going to be talking about a fascinating and hugely viral series he published at his Substack titled “Academics Need to Wake Up on AI,” about what AI can already do when it comes to research, and what that means for the academics who are not paying attention, which is many of them. It was very widely read, and it generated, let’s say, a somewhat polarised response. So Alex, to kick us off: what’s the central thesis of this series, and what motivated you to write it?Alexander Kustov: Thanks, Dan, for having me. I’m a huge fan of the Substack and the whole podcast series with you and Henry. So, like some of us, I’ve been using some of these AI tools. I’ve been reading some of the other folks like yourself, and it really transformed everything I do in my life. And I should say I was also on sabbatical, so I had a little bit more time than some of my colleagues to try some of these tools. I just hadn’t really seen any of my colleagues talk about it. And when they did talk about it, they usually tried not to be vocal about it. I just didn’t think it was a good equilibrium, where basically people were using these tools to be ten times more productive and not talk about it. It really heightened this sense of inequality for me, which I do care about. You’d have a situation where someone would publish ten papers in a year and someone else would publish one, and the only difference is that the person publishing more is the one using Codex or whatever. I just wanted to write about it. And I saw that the prevailing academic discourse on the issue, especially on platforms like Bluesky, was very counterproductive.I didn’t really say much, to be honest. I didn’t think it would be that controversial. But the biggest thesis that really rubbed people the wrong way was that right now a lot of these tools are better at a lot of the tasks that we do as professors. I’ve refined this idea a little bit, going back and forth with some of my critics, but I feel comfortable right now saying that if you look at it globally, and think about what professors do around the world, in social science and adjacent fields especially, AI agentic tools can do most of the tasks they do in terms of literature review, data analysis, and even coming up with some research questions, better than those professors on average. I think that’s a pretty uncontroversial statement at this point, but obviously a lot of people were very, very upset about it.Dan Williams: Empirically speaking, it is a controversial statement, in the sense that it provokes controversy when you say it. In a minute we can get to the question of what AI can actually do in the context of research. But for what it’s worth, I completely agree with you that on many tasks AI is clearly better than what human beings can do. Is your sense that lots of people just weren’t aware of that fact, that they literally didn’t have exposure to these tools? Or was your sense that the reason people weren’t really talking about it is because of all the controversy surrounding the use of these tools, not just mere ignorance?Alexander Kustov: I think it’s both, for sure. There was recent research done by Anthropic. They tried to do, not a representative survey, because obviously the population is very hard to define here, but they surveyed something like 1,200 quantitative social scientists, and the estimate right now is that about 20% of folks use agentic tools. That doesn’t seem like much at all, and if anything it’s probably an overestimate, because they’re more likely to tap into well-resourced universities. So I do think it’s both: the little uptake we have, and the fact that people who do use these tools don’t want to talk about it.There are two things here. First, you want to maintain your comparative advantage. This moment right now is exactly the moment where, if you’re one of the few people using these tools, you can write a bunch of papers and get tenure while the tenure system is still in existence. And the other thing is that if people are very upset about anything AI-related, you don’t want to talk about it and be shamed by your colleagues. Just to give you one funny anecdote: at the height of the vitriol I experienced, where hundreds of people literally were quoting me and trying to tag my employer to get me fired, the exact same people were often DMing me and asking for my setup and prompts. So it’s very crazy to me that you have this big disconnect between what people say publicly and what they actually do privately.Dan Williams: I find it crazy that it’s only 20% of social scientists, or whatever the exact number is, that’s actually using agentic AI. Just before moving on, maybe we should explicitly address: in your view, what is it that agentic AI, as it exists right now, can do? What are the kinds of tasks it can do better than human beings, and how can it improve the workflow of an average social scientist?Alexander Kustov: Coding is the first thing. It’s literally in the name, Claude Code. That’s what these tools were designed for. If you talk to any coding person, a computer scientist, or even someone who isn’t a computer scientist but does a lot of coding for their work, I don’t think anyone would doubt that it’s a huge productivity improvement tool. And the vast majority of quantitative social scientists who do any kind of data analysis do a lot of coding, so they have to be very receptive to this by definition. And I think they often are.What happens is that social scientists are comprised of a bunch of different tasks and topics that people can disagree over, depending on the field. Economics is pretty homogeneously quantitative and formal, so there you can definitely see the biggest uptake. But a lot of disciplines, like political science or sociology, are a mix of qualitative and quantitative folks. And a lot of this AI polarisation overlapped with that pre-existing divide. People who didn’t like stats, who didn’t believe in positivism, the idea that you can learn something about the social world using evidence, were also more reluctant to believe that AI is helpful for them. Which is funny, because, as I also mentioned in some of my writing, if anything those people are going to benefit from these tools, because Claude cannot really interview people and do ethnography yet. So in a way there will be more demand for very high-quality qualitative work. And there are some good examples of qualitative people I respect who embraced AI completely.You can still use a lot of these tools to boost productivity outside the coding realm. You can write emails. One thing I think anyone would acknowledge, including the critics of AI, is that it’s definitely helping them respond to administrator emails, which no one likes, and which isn’t considered an unethical thing to do. I remember someone on a big account on Bluesky posted that AI should be banned except for transcription purposes, because they do a lot of interviews and it’s good for transcription, but everything else is off limits. And it’s interesting how the goalposts are changing right now. The biggest fight I’m getting involved in again, because that’s kind of what I do, is this idea of AI detection and disclosure, and we can talk more about it later. But there’s this interesting consensus forming that AI is good for research now, which was not the case half a year ago, from the same people. Now those people are saying AI is obviously good for research, duh, but it’s not good for writing, for a bunch of different reasons.So we talked about coding and data analysis. There’s a lot of other things a normal researcher would need help with: getting a basic summary, a literature review, translation. It’s above my pay grade, I’m not a machine learning person, but my understanding is that LLMs are exceptionally good at translation, and the fact that a lot of people deny they can translate things well is insane to me. You can transcribe your interviews, translate your survey questionnaire into different languages, whatever you need. I also used AI a lot for public engagement recently. You can translate your website, you can create your website from scratch in a day or two. You’d be surprised how few academics actually have good functioning websites. And I’m not talking about very old people who reject technology, but also young PhD students, who you’d think would have an interest in making sure people can find them online. But no. You can just install Claude and do it in a day. The fact that people are not doing it is insane to me, and I’m trying to spread the word. I’ve convinced a lot of folks to do it, but there’s only so much I can do as one person.What predicts whether an academic uses AI?Henry Shevlin: It really resonates, hearing about your experiences with some academics being completely oblivious to AI and others enthusiastically adopting it. When I’ve visited different businesses and universities, I sometimes literally see the same people doing exactly the same job, maybe even sitting at the same desk, one of them doing amazing things with AI and the other one not using it at all. I’m curious whether you’ve got a sense of what the best predictors are for whether someone is an AI user. If you could only know one thing about someone in the social sciences in order to predict whether they were a big user of agentic AI, what kind of things do you think predict it?Alexander Kustov: That’s a very interesting question. I’m pretty sure there’s some kind of deep personality thing. Everything goes back to personality, whether it’s socialisation, upbringing, or even genes. Openness to experience probably jumps out to me as one of the first predictors. My initial thought was that, since this AI debate overlaps with the qualitative–quantitative debate, people who are methodologists, econometricians, or psychometricians would be much more likely to adopt AI. On average that’s true, but it’s not completely lopsided, for some reason. In fact, there are some very interesting examples of people who were very good methodologists, developing their own regression models, doing machine learning, who then got very sceptical about LLMs. One way to think about it is that those people actually know more about these tools, and so their scepticism has more value, and I’m trying to be tuned to that.But there’s also something about self-interest. Previously you were this privileged person, where the whole department would come to you to help with methods or regressions. I was a person like that in my department back in North Carolina, where people would come to my office and say, “Alex, can you help me with this game theory model?” And I’m not even really a methodologist. So it really depends on the comparative advantage people have. Now basically anyone can go to Claude Code and try to do a very fancy analysis. Obviously you have to know something, you have to know what to ask, but these models are exceptionally good at giving you the basics. If I’m not really good at geospatial statistics, for example, I can go to Claude, do some spatial regressions, and learn about it on the spot. Previously I would have had to go to some spatial colleague in the geography department for that. Now it’s just much easier to do it myself with my computer. And it’s probably going to be as good or even better.Dan Williams: I think for all of these areas within quantitative social science, from coding to data analysis to literature reviews to writing, which we can return to in a bit, the quality of writing you can get from these models is really exceptional. But I’d also point to things like brainstorming. I’m a philosopher, Henry’s a philosopher. I don’t do quantitative social science, so I do research that’s constrained and informed by empirical research, but I don’t actually collect data. In terms of having access to a very smart interlocutor who you can literally prod, telling it, “give me the three strongest objections to these ideas I have for a paper,” and use that as the basis for thinking through an idea, that’s such a huge advantage, even when it comes not to quantitative social science but to theory construction and many aspects of qualitative research. I’m really baffled by, well, I somewhat understand people who just haven’t used these tools, or whose last use was in 2023, so they’re just ignorant. But I’m really baffled by anyone who’s actually used the paid version of Claude or ChatGPT and doesn’t understand the extent to which they can improve your ability to think through topics, understand things, and get information. Henry, are there any ways you use AI in your research, and in how you think about topics, that we haven’t touched on already?Henry Shevlin: For me, the primary use case for AI systems is always just learning, and learning about new topics. Being able to ask questions and verify my own knowledge has been a game changer. Although I will say it’s also been a massive time sink. I’ve gone down so many rabbit holes that I probably would not have prioritised if I’d had to dig out articles. But in some sense it’s been good for my education in the round, even going down those rabbit holes. Otherwise, I find it useful for summarising and making sense of my data: getting a whole bunch of research papers and using Claude Cowork to create summaries of them. NotebookLM is also very useful in its own right for dealing with defined archives. The thing I really need to do this year, and the thing I’m most looking forward to, is getting a good agentic workflow for dealing with email. I already use Claude for drafting quick emails that require a certain degree of precision but don’t involve any warmth or human feeling. But being able to have an email assistant is something I’m looking to build in the next couple of months.Alexander Kustov: A few thoughts on that. Brainstorming is a big one. I did mention it, because that’s the default way you should use these tools: to have a very smart person to talk to, especially when you don’t have access to your colleagues. It’s a really good substitute. Even setting aside the question of whether LLMs can generate great novel ideas, you really just want a conversation partner who can rehash old ideas and tell you why you’re wrong. The reason people don’t realise this is that a lot of folks don’t do the very simple thing of paying for a premium subscription and installing one of these agentic tools. It’s happened to me several times: I’d talk to people about an agentic tool, I’d specify Codex or Claude Code, “do you have it, have you used it?”, and people would nod, and then five minutes later in the conversation it turns out they completely missed that part and still think about the chatbot thing. So a lot of people are confused about this.What I find helpful, at some of the workshops I’ve done and that others have done, is that you just sit with folks and install one of these tools for them, and ask them to do one simple task that’s good for their career. Like create a slide deck. A lot of people are still amazed that you can create a slide deck much better than the average academic slide deck in a minute. That really changes people’s minds; it’s mind-blowing for a lot of folks. Or, I don’t know if you’ve had this experience, there’s this Refined service where they do peer reviews for papers; I think an economist created it. I’m not a huge fan of it, because it costs $50, but the first one is free, and there are a lot of free systems that can imitate this exact functionality. I’ve seen several of my colleagues at Notre Dame use the free upload for one of their papers, and they received the best feedback they’d ever had in their lives, the kind you’d never get at an average academic conference, and they got completely converted overnight. It really takes one magical event for people to understand that something is definitely going to change very, very soon.How much academic research is actually any good?Dan Williams: Just to double-click on one thing, this question of what drives the differences in how people view AI. One thing we haven’t really touched on, but which I think is very important, is how you view the nature of research and what you’re even doing as a researcher. Whether you view it fundamentally as being about producing the best output possible, or whether you view it as some journey of self-discovery, exploration, and authentic engagement with the material. I think the latter model of research is very threatened by the idea that you would integrate these AI tools into it. If you’re ruthlessly focused on how to produce the best outputs possible, as evaluated according to relatively objective metrics, then I’d speculate you’d be much more disposed to make use of whatever tools help you do that, including AI.But this connects to another thing that’s just come up in what you said, Alex, which is something you write about in this series. So far we’ve been focusing on how good AI is. There’s this other side to it all, which is how bad much actually existing human research is, even before we talk about anything to do with AI. You get into this a lot in the second and third essays in the series. Do you want to say a little about that, about how that other side factors into how you’re thinking about this topic?Alexander Kustov: The third installment of the series basically came to me while I was at a political science conference, or rather an interdisciplinary conference called ISA, for international studies specialists who study relationships between countries. And it was really bad. Big academic conferences are always bad; that’s something you expect. There’s so much money, including public funds, spent on all these conferences and travel for people from around the world. But if you ask a regular academic, forget about AI, they would tell you they don’t expect to get good feedback, their panel is going to be completely empty, and the reason they do it is because they have to spend their $2,000 travel fund and potentially hang out with some friends and do some networking, which is not bad. Networking is a huge part of conferencing.So I was sitting at this conference, seeing really bad presentations where people would have tons of grammar and sense mistakes and senseless research questions. It’s bad both in terms of substance and execution. And exactly at that moment I was getting all this vitriol for saying that AI can do better stuff, like slides. I was like, no, this is just a huge disconnect. I started thinking about that. The issue I see is very pronounced in the conversation around self-driving cars, where people compare them to some ideal in which there are no accidents and no one dies. When a self-driving car runs over a cat, it’s a huge news story, but humans do that every single day, in their hundreds, and we don’t care, because we accept that humans are fallible and bad and not doing good work. I think it’s the same with academic work. The vast majority of things produced by professors globally is just not good and not contributing to human knowledge.For some people this can be even more controversial for me to say than anything I said on AI. A lot of people view this world from their own parochial angle of being a research professor in a top-tier American or British school, or Cambridge. But the vast majority of folks are not like that. I experienced academia in the post-Soviet world where I grew up, and in most cases people just want to get by. They publish in some predatory journal with a random, rehashed argument that probably reinvents the wheel and doesn’t really contribute. No one’s going to read it. We know that 80% of published papers in the humanities are never cited, and probably never read either, except by your editors or reviewers. And as an associate editor of a journal, I can tell you I doubt the reviewers actually read some of the papers they review. So compared to the actual status quo of what’s happening right now, automating it all and using AI tools mindfully and responsibly is going to be a big win.Another problem is that we have this binary thinking that it’s either/or: we either do one-shot papers that aren’t good, or we don’t do anything. But you can write your own paper, do your own slides, and then ask your AI agent to help you brainstorm, create a graphic, or redesign your graph. People might disagree on the details of what’s more acceptable and useful, but at the end of the day there are so many use cases for these tools that are completely uncontroversial at this point.Henry Shevlin: Just very briefly, I think it might matter whether academia’s problems are due to things like laziness or just not caring, versus a lack of skill. I’m curious whether you have a theory about where these problems in academia come from. Is it the fact that a lot of people are just really bad, for the most part, at doing data analysis, for example? If so, then AI is amazing; it’ll lift the floor. But if it’s that people just want to commit fraud and do whatever it takes to get ahead, then maybe AI isn’t going to make the situation better, or could even make it worse.Dan Williams: Or a third thing: it could just be the nature of the institutional incentives. I feel like a lot of what’s behind the replication crisis, the reproducibility crisis, the generalisability crisis, and so on, is not so much that people are lazy or unskilled. It’s that you can get ahead and win the status game within academic research by engaging in shoddy research practices, and as a consequence that’s what you get: a lot of shoddy research practices. But a lot of those findings that don’t replicate were done by really brilliant, energetic, ambitious scientists. It’s just within this flawed incentive structure.Henry Shevlin: Brilliant, energetic, but perhaps not fully scrupulous.Dan Williams: Yeah, but you can’t rely on human beings to be scrupulous. You need the incentives set up in such a way that even by default unscrupulous people will be driven to act in pro-social, beneficial ways. That’s my cynical perspective. What do you think, Alex?Alexander Kustov: I’m going to say something very controversial: I want to believe that people are good by nature. At least, my knowledge of evolutionary psychology tells me that even those people who commit all these bad practices at least want to believe they’re doing something good. They’re often motivated by good things, with some exceptions; there are some people who are truly evil. But even if we take some of the most famous fraud cases in academia, like Francesca Gino at Harvard, I think the way it probably works is that you start by doing some research you care about, it gets picked up by the public, you’re very successful, there’s a lot of demand for what you do, and then you get some uncomfortable result and you tweak it a little bit. There’s all this literature about p-hacking, where you have some theory you want to prove, and when you have to make a choice between presenting model A and model B, you unconsciously choose the model more in line with the result. You can even justify it to yourself, that this model makes more sense, that it’s obviously much better. And any individual case might be right. But in aggregate it doesn’t lead to good outcomes.I also think there’s a lot to say about the incentive structure in academia. Right now you really have to publish or perish, still, despite the fact that we can talk about whether the journal model is going to be sustainable in the near future. You have to publish a lot, no matter what your field is. Which means that if you have to decide between doing a better job with data analysis and spending a year on it, you’d probably spend less time on it and publish as soon as possible. You’re not really incentivised to replicate data. It’s very hard to publish critical responses and replication studies, and we have all this evidence that failed replications are usually much less popular and less cited than the original studies that have been disproven. Another thing I’ve been talking a lot about is public engagement, where you’re very rarely rewarded for actually spreading the knowledge of what you do, because that’s not something your dean would appreciate. So people default to publishing shoddy papers no one’s going to read. And peer reviewers don’t really check your data in most cases. When I submit my paper to a political science journal, people take it for granted that my analysis is legit, and they quibble about the framing or some other superficial thing.That’s one of the reasons I’m so concerned right now about this whole Pangram hysteria, because people are going to be looking for em-dashes or whatever instead of the substance of the underlying claims. I see the Bayesian argument that if something is clearly AI slop, it probably also doesn’t have good data in it. But knowing modern AI tools, if you ask Opus 4.8, which just came out, to create a report on some topic with publicly available data, I’m pretty sure it’s going to be able to download things and create a chart that’s probably more legit than a chart you saw published in an academic paper four or five years ago. Even if the prose isn’t as good, and we don’t usually have good writing in academia anyway, it’s going to be more human than em-dashes or “it’s not X, it’s Y.” So I think it’s a combination of all those things, but I do want to believe that very few people actually want to commit fraud.Dan Williams: Let’s definitely talk about this writing thing. But just on this previous point about incentives, I agree that people aren’t sadistic and don’t go out there thinking they want to do bad things. I just think academia is a status game with certain norms and institutional procedures. People are often ferociously ambitious, and they do whatever’s going to get them status, prestige, and recognition, as that’s defined and understood within academia. All the human slop produced in the context of academia is just because the incentive structure is messed up. You can rack up lots of status by churning out a load of crappy, non-replicable findings that don’t add anything to the academic literature. But if that’s your model of what’s going on, you might think: well, if the problem ultimately is not to do with human beings being lazy or unskilled, but to do with the incentive structure, then why would merely giving us access to AI improve things? You might think all that’s going to happen is people will play the same status game, but do it a lot quicker and at lower cost, and we’re not actually going to advance the frontier of knowledge, because all the same structural causes of bad research are still in play.Alexander Kustov: I think that’s a key question. We’re facing a forking path of some sort. You can imagine a scenario in which the future is as bleak as you just described, but with more slop. That’s the problem I see with what might happen: take all these bad incentives, give this miraculous tool to researchers, and instead of one paper per year they’d produce ten that don’t lead to anything productive. It just inflates everyone’s expectations and creates more problems. But there’s an alternative scenario, and I think it’s still in our hands to do something about it. Instead of increasing productivity in terms of quantity, we can use these tools to increase productivity in terms of the quality of research. Since you can now generate something very simple in a minute, you really have to do something better than a shoddy regression with no account for endogeneity concerns, or rehashing the same exact philosophical argument people have been making for years and years. So there’s a way to do better with these tools, and it’s in the hands of current journal editors to raise the standards, do more desk rejects, and say the quality bar is now much higher. I think it’s already happening somewhat, and it’s something we can consciously decide to change.I also have some hope for the frontier models. There’s been some interesting research showing that when you explicitly ask a model to p-hack, it doesn’t do that. You can jailbreak it, so to speak, and say “please, please, I really need that,” and it’ll do it sometimes. But with those basic guardrails, they’re going to help people, because no one is consciously justifying p-hacking; people don’t like that. When the model refuses to do it, it’ll make them think that maybe they should do something different. So I have some hope. But obviously it depends on what happens to academia in five to ten years and how the models develop. We’ll definitely have to redesign the incentive structure, because I’m not sure the number of papers you have is the best indicator of what you’re trying to do. The paper itself as a format is a weird thing, because now you can also have updated dashboards with new data. It seems like a very outdated format, at least for some arguments, but it’s not like we have a better equilibrium yet. A lot of things are in flux right now, and I don’t have a simple solution.That’s the whole point of my series: I wanted people to start talking about it. I think it did help a little. I’ve gotten calls from deans around the country, and I’ve participated in panels where people have a university-wide conversation about these things, and a lot comes to the ground that people aren’t aware of. There’s definitely some hope, because a lot of the people in positions of power right now, the older, tenured, full professors, don’t use these tools. According to that poll we discussed, it was 20% in the general population, and I think it was about 9% among full professors. Some of those folks might not be reachable, or they might not care; they just want things to continue the old way. So we definitely have to do something about that.Will AI make academic inequality worse?Henry Shevlin: Do you think there’s a risk that we see growing academic inequality, a kind of rich-get-richer effect, where the most prestigious, maybe not the older generation but certainly rising scholars with their own brands, use AI to put out twenty times the number of papers? We’re living in a tide of slop, but those with good reputations or good brands dominate. That might not be disastrous in every way, but it might lead to highly unequal outcomes within academia, with less well-known or less skilled researchers being completely left behind.Alexander Kustov: There are several things that lead in opposite directions here. In theory, and I don’t think I’m making an original argument, a lot of people have written about this, there are certain equalising things coming out of all this. For instance, the ability of these tools to translate things. If you’re a non-English speaker, it’s much easier for you to write those papers now, which is a huge productivity boost, and from the perspective of science it means we’re going to be able to get all those talented people and their arguments from all over the world, regardless of where they come from. And at least for now, the premium subscription is one or two hundred dollars, and people in most major universities can afford it, even in more developing countries. It’s not equal, but whether you’re at Harvard or a community college, you can afford a $100 tool, at least for some time, and presumably you can do exactly the same thing with it. Compared to the status quo, where as a community college professor you have to teach five classes a semester with no research budget, while at Harvard you don’t have to teach at all for the first two years and have a $200,000 startup, that’s a very big difference. So there are some equalising things going on, and it’s important to acknowledge that.But you’re right that it’s also the case that the people able to use these tools most productively and efficiently are the people who already have a lot going on. Even though I’m very sceptical of the idea that LLMs can’t come up with new ideas, because in general new ideas are recombinations of old ideas, I do think you have to have a coherent set of ideas and goals of your own to be able to utilise these tools. It’s really all about your creativity and imagination. Every single day I see someone post something they did with Claude and think, “wow, I hadn’t thought about it.” Just yesterday someone posted about this idea of making your papers machine-readable, and I converted all my PDFs and my website to Markdown with all the figures. I think everyone should do this. I could have done it last year, I just hadn’t thought about it. There are a lot of things like that where you really have to have good ideas to begin with. So people who already have a whole research pipeline and some budget are now able to execute it much faster. This rich-get-richer dynamic is definitely going to happen. And in the future, where those models are potentially going to be much more expensive, that’s a possibility. My understanding is that right now it’s all subsidised, and the $200 model is actually going to be a $2,000 model. Then only the Harvard people are going to be able to afford it. So I just hope Notre Dame is going to be part of that.Can AI be genuinely creative?Dan Williams: This point you made, Alex, also in one of the essays, about creativity and what’s really going on when it comes to coming up with new ideas in science, I was a little sceptical of. It’s a surprising feature of state-of-the-art AI today that, given how smart these models are in some sense, and given the vast knowledge base they have, they don’t really seem to make discoveries of a really new and impressive character. There are potentially some counterexamples, but my sense is you might think of this roughly in terms of the philosopher Thomas Kuhn’s distinction between science that happens within the context of a paradigm, normal science where you have relatively well-defined problems and puzzles, maybe the Erdős problems fall into that category in maths, and I suspect that for that kind of thing, AI, if you prompt it the right way as it exists today, can be used to help make progress. But when it comes to true creativity, the sort you find in really bringing about paradigm shifts, moving outside the space of predefined problems, reconceptualising an entire domain, and coming up with radically novel theoretical insights, I actually think AI as it exists today doesn’t really seem to have that capability. And that potentially tells us something interesting about the limitations of the models. I’m interested in what you think, and also in what Henry thinks about that view.Alexander Kustov: Henry, you can start.Henry Shevlin: On one hand, you might point to something like transformative creativity. Margaret Boden has this breakdown of creativity into three categories: combinatorial creativity, recombining existing ideas or elements to create new things; exploratory creativity, where you’ve got a predefined dimensional space and you’re going to bits of it that haven’t been mapped out yet; and transformative creativity, which is completely upending the apple cart, developing new dimensions. People point to Picasso or Einstein as examples of that kind of transformative creativity, and often will say AI can definitely do the first thing, maybe can do the second thing, but it’s not clear it can do the third thing. That’s maybe one way of putting your point, Dan. It’s certainly true that we’ve not seen any dramatic scientific breakthroughs that have been primarily AI-driven as opposed to AI-assisted.One reason I am a little optimistic here, though, is that in other domains, most notably Go, there’s the famous “Move 37” in game two. In case anyone doesn’t know, and I think we’ve talked about it before on the show, this is in the second game between AlphaGo and Lee Sedol, the Go world champion, back in 2016. AlphaGo made this bizarre move that no human player would make or had made in the past, and yet it was really effective. The system knew what it was doing, and this has now been incorporated into the way human players actually play Go. So I think that’s probably a pretty strong candidate for a genuinely transformative piece of creativity, at least if we’re classifying it by its impact rather than its process. That’s obviously a very different domain; you’re operating with very well-constrained rules and goals that maybe allow for that kind of transformative creativity. But I am optimistic those kinds of transformative leaps could eventually come from AI systems, even general-purpose ones like LLMs. What do you think, Alex?Alexander Kustov: I really like this distinction between combinatorial creativity and the other types. Combinatorial creativity is definitely something LLMs are really, really good at. It’s kind of similar to translation: you mix and match different things. I’ve definitely seen a lot of really cool ideas come out, on the immigration stuff I work on, from LLMs, when I was doing brainstorming. This is undeniable at this stage. When it comes to transformative creativity, I wonder whether the reason we don’t really see it much is because we don’t really have AGI yet. I know you’ve talked about AI consciousness and all those questions. Maybe if we let the model think for itself and live in the wild, it’s going to happen. But right now, for most people, they set up a goal themselves for these models. Maybe that’s exactly why we don’t see transformative creativity, because you can’t just set up a goal and have it come up with something transformative. You have to specify the goals, and the goals are usually specified by people who can’t really do the transformation themselves.But going back to Dan’s point about the paradigm shift, I do think we’re in this stage right now where, even if you concede that AI can’t have transformative creativity, just because we can now offload all this grunt work to AI, including email and all the other stuff that takes a lot of time, we can do other things that are creative and potentially transformative. That’s what I see with myself: I’m spending less time on administrative stuff and email, and more time brainstorming my ideas, talking to people, and doing really valuable networking and public engagement, which I’d never be able to do otherwise.Dan Williams: We’re in this great space at the moment where you’ve got incredibly smart, helpful AI tools, but you don’t have truly transformative AGI. So there’s still a role for human insight, judgment, and creativity. If that gets taken away over the next several years, that’s a very different kind of situation. I think there’s definitely a chance that by 2030 we have AI systems that can substitute for everything human beings can do cognitively. And then that’s a very different kind of world, and a very demotivating kind of world in some ways.AI writing, detection, and disclosureDan Williams: Let’s talk about writing. We’ve touched on this a few times already, but I know you’ve got interesting things to say about it, Alex, and potentially quite heterodox views. At the moment, more and more people are using AI to write. There are also these AI detectors. I think Pangram is the one which seems to be used the most, or that people trust the most. It’s got a very low false positive rate, as I understand it, although I’m not entirely sure how they go about establishing that. Many people think that if you use AI to write something, whether it’s a blog post, a novel, a poem, or an academic article, and it’s found out that you’ve done that, you’ve done something really bad and discrediting. My understanding is you don’t see it that way, Alex. So what’s your view?Alexander Kustov: A lot of it goes back to this idea of disgust sensitivity, talking about personality traits. There are some things people just think are “yuck” for whatever reason. It’s totally subjective. I don’t think you can really rationalise it; I think it’s some ground truth. I should say I’m coming to this from the perspective of someone born in the Soviet Union, where the Russian culture is very literate and people take a lot of pride in using proper grammar and speaking properly. I see a lot of parallels here with the previous wave of grammar Nazism, where people would ignore the substance of what you’re trying to do and point out typos, or “whom” instead of “who,” or the other way around. Obviously it has some function and might be useful in some respects, especially when you’re in school, but it takes up a lot of energy. My worry is that this whole new AI detection situation is going to be similar, where people spend a lot of time on very superficial pattern recognition. Right now you look for em-dashes and some other patterns and try to decide whether something is worth reading. That’s the common justification for Pangram use, that you want to make sure what you’re reading is worth it.The problem is that even within the realm of human-made writing there’s a lot of slop, and you’re not going to be exposed to and won’t read 99.9% of it. Given the trajectory of the tools, I’m not sure that knowing something is AI-generated is necessarily worse. A lot of it is about the status signals people have. I personally don’t like very clear AI tells either; it rubs me the wrong way. But who am I to judge? What if it’s a non-native speaker, and the counterfactual to me reading their AI-generated text, which is potentially thoughtful, is just not reading it at all, because they can’t speak English well? People don’t think about it this way. They compare AI-written text to the best, to Shakespeare. I don’t think that’s the relevant comparison. Most of the text people write is not good, and to the extent that some people can improve it using AI, I think that’s good.Practically speaking, if you’re an academic and you want to write more and you’re afraid of others calling you out for using AI, just use a style guide. Use a CLAUDE.md or AGENTS.md file to tell it not to use those phrases. Tell it multiple times, because it still adds em-dashes. But there’s a way to use AI for writing in your own voice, and I think it should be morally justifiable, depending on the realm. One thing I’ve been thinking about, and I’m going to workshop this idea with you, is that there’s a spectrum of the ethical justification of whether it’s okay to use AI for writing.Clearly we can think of some examples, like a student assignment that needs to be human-made; when it’s AI-written, it’s a failed assignment. That’s a pretty clear case. The way professors think about this mostly comes from detecting their students cheating, and that’s why they think about it that way. But it’s a very rare scenario. In fact, a lot of professors right now encourage their students to use AI. I talked to some colleagues recently in stats classes who produce a regression paper in ten minutes on their computer and tell their students, “that’s something I can do in ten minutes, so you should do something better than this,” with AI or not. That’s a pretty good educational approach for some situations.Another example I mentioned in one of my posts is that when you go to a live concert, there’s an implicit presumption that it’s going to be a live event and people are going to be singing themselves. If you notice and catch them not singing and using some device, that’s not cool. The same thing here: if you’re paying for someone to write you a human-made letter, a condolence email, it’s totally fine to be upset if they use AI for it. That’s totally justifiable. But on the opposite side of the spectrum, when you get a very formulaic email from your administrator, I think it’s totally justifiable to outsource that to AI, to your agent who knows your schedule and what you’re going to do, and no one’s going to be upset about it. People disagree on the margins of what’s acceptable. When you create a graph with data you worked on and understand, and you ask AI to describe it, I don’t see the problem; it’s probably going to be more accurate than most humans. Maybe we can have a social norm where if you say “I feel,” then it should be you who says that, as opposed to Claude. We’re still in this limbo where the norms aren’t clear, but we should be clear about what’s good and what’s not. It’s very hard for me to make a blanket statement that AI writing is good or bad; it really depends on the particular scenario. There are scenarios where it’s totally uncontroversial to say it’s okay to use AI, and scenarios where it’s totally uncontroversial to say it’s not. But the middle ground is what we’re trying to figure out right now as a community of knowledge.Henry Shevlin: I’m curious: Dan, how much of a hatred for obviously AI-generated text do you have? I have to say, I’m generally pretty AI-positive. I’m a very heavy user of AI. But I do definitely downgrade my assessment of text when I realise it’s just obviously AI-written. There are a few things going on there. One is that it’s not even so much that the text is AI-written, it’s the AI voice, the very specific voice. I just think it’s such a boring voice at this point. It’s so homogeneous. If someone wrote a brilliant comment or a brilliant reply to me on Substack or Twitter, or sent me a brilliant email, and I subsequently found out it was AI-generated, I don’t think I would care. But this one specific, overfitted, “it’s not X, it’s Y” just drives me up the wall.I guess, focusing just on the question of whether there are, even setting aside those stylistic issues, specific contexts in which AI usage itself might be problematic. Another example, Alex, I love your example of the bands and people not lip-syncing. Another silly one is that a handwritten note does mean a lot more than a generic email, so sometimes it is precisely the effortfulness that makes the difference. But I also wonder whether, to some extent, we’re misled into thinking the average quality of AI-generated writing is worse than it is, because of what I’ve heard called the “toupée phenomenon.” Everyone thinks wigs look so bad, and that’s because your sample of wigs that look bad is the ones you can tell are wigs. If they’re good toupées, they don’t even make it into your sample. So in the same way, I think probably all of us are reading tons of AI-generated text that we’re not clocking as AI-generated.Alexander Kustov: Yeah, there’s definitely survivorship bias. With my first post about the AI series, one of the reasons it got so controversial is because I used Claude to generate 99% of it, and I didn’t disclose it right away, and then I did post factum, and Pangram gave it 100% human. So that’s a false negative, which is not a huge deal, but it’s interesting. A lot of good writing is AI-assisted right now; we should just take that for granted. When we see something bad that’s clearly AI-written, it’s just those particular instances. The strongest argument I’ve heard for being upset about it is that if someone doesn’t bother editing the text, or even creating a style sheet to make sure they don’t use all those constructions at the same time, it probably means the underlying substance isn’t good either. But I’m not sure how true that is; it really depends on the context.The problem with social media comments, when you see something clearly AI-generated, is that it’s also not clear whether it’s a bot or a real person using AI to voice their opinion. But if you know this person and their account isn’t hacked, and they have some AI writing tells, I think it’s fine. I’m also not very happy to see a lot of clearly AI-written stuff, but I’m trying to rationalise it in a different direction and think about why it’s actually a problem. I’m not sure.Dan Williams: I think that ultimately, in contexts that have to do with academic writing, or people publishing their views and participating in debates, you should just be judging things on the quality of the contribution rather than its provenance. It just so happens that, at least when it comes to the AI writing that I detect, the quality is bad, for the reasons we’ve discussed. I just hate the style of writing you find with these models. I find there’s something really cringe and annoying about it. But that’s not a necessary feature of AI writing; it’s just the way the current models have been post-trained to produce a particular kind of style. And to Henry’s point, if I discovered that, for example, my favourite blogger, Scott Alexander of Astral Codex Ten, who I think is the king of Substack, had been generating his posts with AI over the past two years, well, I think those posts have been amazing. So I wouldn’t think, “now that I know it’s AI-generated, I’m going to retract that assessment.” That would be ridiculous to me. So in principle we should be judging things based on the quality of the output, not the provenance.But I do then think, even if you think there’s this separate question about disclosure norms and what they should be, you make this really important point, Alex, which is that at the moment there’s a problem with disclosure norms: they end up just punishing honest people. Because if you come out and say you used AI to write something, as you did with your first post in the series, there’s a massive backlash. So if you’re honest, you get this huge reputational damage associated with doing it, which is going to discourage people from being honest, which means the dishonest people get access to the benefits of AI-generated writing without any of the reputational costs. As an equilibrium, asking for disclosure norms just doesn’t really seem either desirable or possible. Firstly, is that an accurate summary of your point of view? And secondly, do you still think that’s basically the correct point of view when it comes to disclosure norms?Alexander Kustov: As a newly minted associate editor at a journal, where we’re probably going to expect a surge in AI slop that we have to deal with, I’m very cognisant of the potential problems. Right now the go-to move among people doing journal editing, and probably what we’re going to do in our journal, is to introduce checkboxes for AI use. My sense is that we’re going to do that, but no one’s going to care, because no one’s going to report it truthfully. This is one thing where I strongly disagree with Kelsey Piper, who I deeply respect: I really don’t think it works out, at least for academics, especially in this environment where people feel very strongly and viscerally about this. Coming out and saying you use AI is just not going to do any good for anyone.Another issue is that, to the extent you have some people who are completely anti-AI, disclosing that you used AI for, say, research assistance with data collection, as opposed to writing, what’s going to be worse for them? Any checkbox you have there is probably not going to satisfy them. So it’s strange to me that this is the solution we came up with. I can see how honesty can be rewarded in some contexts, and I’ve seen people on Substack say they used AI for help with data collection or writing. As a quantitative social scientist who primarily cares about data and quality, I’m surprised people think it’s more okay to use AI to collect and analyse data but not to write about it, because the first part is much more important. So I’m really going back and forth on it. But it’s hard for me to come up with a scenario where AI disclosure is actually going to work and solve anything.What needs to happen is for us to change some of those norms. The same way we’re upset with AI tells, we’re also upset with how Gen Z, or whatever the new generation is, writes without capital letters. I can’t stand it, but that’s how people write, and who am I to judge? So I understand why people want to judge the quality of the substance, but they use the shortcut of the style of the prose to substitute for the quality. Going back to Henry’s point, what’s going to happen is that people are going to use your previous reputation as the main marker of whether you do something valuable. That’s why I feel for incoming grad students and newly minted professors, because it’s really hard to establish your reputation now with all this stuff happening with AI. Whereas if you were Daron Acemoglu, the most cited economist in the world, who’s been writing a hundred papers before AI was cool, there’s literally nothing he can do with AI or without AI that’s going to change your opinion about him. So people are going to be using these shortcuts more, which means that, from a certain ethical perspective, people are going to be discriminating more based on hopefully immutable but also immutable traits. That’s another thing to consider: people are going to be more trustful of their ethnic in-groups, or people who went to Harvard or work at Cambridge, than minorities. So there are interesting questions coming up about all that. I don’t have any solutions, unfortunately.The backlashDan Williams: Should we come full circle? We touched on this at the beginning, but there’s the content of what you wrote in the series, and then there’s the response to what you wrote, a lot of which was very angry. You mentioned you had people calling for you to be fired. A lot of that came from people on Bluesky. We’ve talked a little about the Bluesky intelligentsia previously on the show, and I’ve written about it on my blog as well. Firstly, do you want to say a bit more about what the reaction has been in general? You’ve touched on it here and there, but summarise it. And do you think there’s a way of steelmanning it? What’s the best possible case for why some people get so furious, so angry, with this kind of stuff?Alexander Kustov: I had some conversations. I haven’t lost any friends, so that’s one thing I should say; I haven’t gotten cancelled, because I’m tenured. I specifically waited for all my hot takes to happen after I got tenured, and maybe that was a good idea after all. But I did have some conversations with some really good friends who disagree with me on AI, and it definitely helped me refine some of my points. The biggest criticism I received that I see some relevance in is the idea, going back to our conversation, that humans are really not that good, and the concern that giving them this AI tool is just going to amplify all the bad stuff. To the extent you want to encourage norms of no p-hacking and doing really good, careful work, just telling people you can produce a paper with AI easily is not a good thing to talk about.There was a recent big thing on Twitter about the practices of academic citations, where someone was saying that in practice academics don’t really read the stuff they cite well, and a lot of people interpreted it in a moralistic way, saying no, you should cite things. So you have the same thing here, where people interpreted my arguments in a normative way, that I’m saying they should do something or not do something, and it was against what they were trying to do. It’s also about this idea you mentioned about the role of academics as a kind of vocation, where you explore the world and self-actualise. I don’t think that, when people actually think it through, they would defend it on the merits, but implicitly that’s how a lot of academics think about their job, and to the extent that we now have tools that are threatening to them, it’s just not going to end well.Trying to steelman the concerns people generally had, some people thought strategically it’s not a good idea to be vocal about it right now, in this moment. As someone who does a lot of work on immigration, I very much disagree with that, because I think we lost voter trust, as liberals and mainstream institutions, on immigration exactly because we were not saying certain things, and the same thing can happen on AI. It’s never a good idea to have a strategy where you do something that’s supposed to be good but that you don’t want other people to know about. I just don’t see how it works out in equilibrium. But I also see some argument that maybe in this particular moment it was not the best time to talk about it. That’s what I got from a lot of that.Dan Williams: Henry, your microphone’s not on.Henry Shevlin: Sorry, I keep making that mistake. I was going to ask whether there could just be a straightforward economic analysis here about why the current anti-AI coalition has the shape it does, namely that elite knowledge workers are overwhelmingly liberal, and AI predominantly threatens elite knowledge workers. You could maybe draw parallels in the same way that most of the opposition to climate change is concentrated on the right, and, speaking very crudely, to the extent that you’re looking at manual workers who in the US context skew a bit more to the right and maybe work in more energy-intensive industries. But I guess the question I’m asking is, is this just about the economics with a social gloss over the top? What does that explanation miss?Alexander Kustov: Some of it, for sure. But a lot of my work in public opinion says that a lot of people’s preferences are sociotropic, based on their ideas about what’s good for society, not necessarily their self-interest, unless it’s really in your face that it’s going to be bad for you. Some of the interesting contingent of haters I had on Bluesky were professional translators who were very upset with my takes on the fact that AI can translate things. I had this silly example, which is a true thing, that my mom wasn’t sure about a prescription she got from the doctor, because it was all in English, and she translated it. Someone was saying that I’m putting my mom under potential harm because she didn’t use a qualified certified translator, and the person saying that was a certified translator. So you see some connection there. But for the vast majority of folks, when it comes to academics who produce a lot of critical theory slop or DEI slop, AI can do this much better than them, but I don’t think they realise it. So there is an objective threat to their self-interest, but the reason they oppose AI is because they have a lot of other bad ideas.Dan Williams: There’s also this thing that I think Dean Ball calls the “omni-cause.” You’re against AI, but that means you have to be against AI in every possible respect. And if you point out one area where AI can actually be quite good, people draw inferences about you, that you’re not on the right team. I found this a couple of months ago, when I wrote some essays, and I was at a workshop where I argued that, relative to the actually existing alternatives, like social media pundits and a lot of legacy media, large language models actually are a pretty good source of high-quality information, that they’re a force for truth. There was a lot of negative response to this, which in my view is a fairly obvious thesis. Afterwards I was getting this response: “so you’re pro-AI.” To me that’s just such an unsophisticated way of thinking about it. I’m really worried about many aspects when it comes to AI, when it comes to power concentration, the economic impact, and how we’re going to cope with it. It doesn’t mean that with every single question you have to think AI is bad in every single way. I sense that, especially on the left, there’s this reflexive opposition to AI, this view that any claim that AI can actually do anything useful or have positive consequences is viewed as a betrayal. Okay, we’re coming to the end, Alex. Was there anything else you wanted to talk about that you didn’t mention?Alexander Kustov: Yeah, related to the last thing you mentioned. I don’t know if you saw it, but after getting all this vitriol on Bluesky, there were a few days where I got positively retweeted by hundreds of folks, because the thing I said was that we should ban electronic devices in all classes. When it comes to teaching, I’m much more pessimistic about AI. A lot of people were like, “what? This guy is an AI booster, how can he use AI but not allow his students to use AI? What’s going on?” You can have a complex opinion on a difficult issue. So there’s definitely this omni-cause, binary thinking, and also moral contamination, where once you start doing something you’re not supposed to be doing, you’re a bad person in all other respects.To finish all that, I feel like we need to move on beyond that. In line with my immigration research, we have to meet people where they are. If people have concerns about AI, they might be mistaken, but they probably have some ground truth in them. So we shouldn’t just say they’re mistaken and wrong and stupid. We should explain to them that they can actually use AI for the good, for whatever they want to do. You can make slides with AI, and when professors learn about that, they forget about all the bad stuff they wrote just a few days ago.Dan Williams: Fantastic. Well, thanks, Alex, and thanks everyone for listening. We’ll be back soon with another episode of Conspicuous Cognition.Alexander Kustov: Thank you. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.conspicuouscognition.com/subscribe

Most conversations about artificial intelligence are focused on Earth: jobs, misinformation, education, politics, science, regulation, consciousness, safety, and the future of human society. But AI—and especially the possibility of reaching “AGI” (artificial general intelligence) and “superintelligence”—forces us to think on much larger scales. If advanced AI is possible, why hasn’t it already emerged elsewhere? If civilisations can build self-replicating probes, artificial scientists, or planet-scale computational systems, why does the universe still look so natural? And if intelligent life is common, where is everyone?In this episode, Henry and I discuss these and many other questions with David Kipping, Associate Professor of Astronomy at Columbia University, where he leads the Cool Worlds Lab. David’s research spans exoplanets, exomoons, Bayesian inference, technosignatures, and the search for life and intelligence beyond Earth. He is also one of the best science communicators working today through the Cool Worlds YouTube channel and podcast.Among other topics, we discussed:* David’s Red Sky Paradox: if most stars are red dwarfs, and red dwarfs live for vastly longer than stars like the Sun, why do we find ourselves orbiting a yellow star?* Whether anthropic reasoning — reasoning from the fact of our own existence — is a profound scientific tool, a philosophical minefield, or both.* The reference class problem: when we reason about “observers like us”, who or what exactly counts as being like us?* The Doomsday Argument, and why some apparently bizarre forms of probabilistic reasoning can nevertheless be powerful.* The Fermi Paradox: if the universe is so large, and if life or intelligence is not fantastically rare, why don’t we see clear evidence of extraterrestrial civilisations?* Whether advanced civilisations would spread through the galaxy using self-replicating probes — and why the absence of such probes might be one of the strongest constraints on extraterrestrial intelligence.* How recent developments in artificial intelligence affect the Fermi Paradox. If humanity is close to building systems that can massively accelerate science and engineering, shouldn’t someone else have got there first?* Whether artificial intelligence makes the simulation argument more plausible.* David’s experience using artificial intelligence in scientific research, and why a meeting at the Institute for Advanced Study changed how he thinks about the role of these tools in science.* Why David thinks artificial intelligence already has something close to “coding supremacy”, but is still far from being able to do science autonomously.* The risks of AI-generated scientific slop: papers, peer review, and training data polluted by low-quality machine outputs.* Whether artificial intelligence will make science more productive, or instead strip it of some of its deepest human value.* Why the future of science communication may depend on better collaboration between academic institutions and independent creators.Links and further reading* Cool Worlds Lab — David’s research group at Columbia University, focused on extrasolar planetary systems, exomoons, habitability, technosignatures, and related questions.* Cool Worlds on YouTube — David’s excellent science communication channel, covering astronomy, exoplanets, alien life, the Fermi Paradox, cosmology, and much else.* Cool Worlds Podcast — David’s podcast, featuring conversations on astronomy, technology, science, engineering, and related topics.* Cool Worlds Podcast: “We Need To Talk About Artificial Intelligence” — the solo episode in which David reflects on artificial intelligence and science after a meeting at the Institute for Advanced Study.* David Kipping’s Columbia profile — short institutional profile with background on his research.Conspicuous Cognition is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.Transcript* Please note that this transcript has been lightly AI-edited and may contain minor mistakes. Henry Shevlin: Welcome back. Our guest today is David Kipping, Associate Professor of Astronomy at Columbia University, where he leads the Cool Worlds Lab. His research spans exoplanets, exomoons, and the search for extraterrestrial life and intelligence, and he brings a Bayesian rigor to questions that could easily drift into speculation. He’s also one of the best science communicators working today with over a million subscribers on his Cool Worlds YouTube channel, where I should confess, I’ve spent an embarrassing number of hours watching when I probably should have been doing philosophy of AI.David, like many of the best people, is a Cambridge alumnus, although unlike us, he actually studied something useful, namely natural sciences, before going on to do his PhD at UCL and postdoc at Harvard on the Sagan Fellowship. His work also has a really fantastic philosophical dimension, particularly around anthropic reasoning and observation selection effects, which makes him a perfect guest for two cognitive scientists who are finally getting to talk to an actual scientist. So David, welcome to Conspicuous Cognition.David Kipping: Thank you for that very generous introduction.Henry Shevlin: This is a bit of a fanboy moment for me, for real though. I really have spent like hundreds of hours at this point on Cool Worlds. But I’m going to get past it. I’m going to be a serious host.David Kipping: It’s always weird when people say that to us, because I just imagine no one watches them. If it gets in my head that people are watching them, I’ll get tightened and anxious about what I’m saying. I just imagine I’m talking to a brick wall or something, and that’s much easier.Henry Shevlin: Honestly, half the Warhammer figures in this room were painted while I was listening to Cool Worlds. I’ll leave it at that. Maybe a good place to start would be discussing anthropic reasoning, since that’s a real natural intersection at the boundary of astronomy and philosophy. Could you just give us a brief view of how you see anthropic reasoning, and maybe tell us a little bit about the Red Sky Paradox, which is one of your distinctive contributions to this area?Anthropic Reasoning and the Red Sky ParadoxDavid Kipping: Yeah, I think one of the most interesting data points when it comes to asking questions about the search for life in the universe and our own place in the universe is our own existence — just the fact that we’re here. Anthropic reasoning has in many ways really been born out of cosmology. Cosmology had a rich history of using this. I think one of the first successful examples was by Steve Weinberg, a cosmologist who’s really a giant in the field. I think he’s now passed away, but he showed that you could predict not only the existence of the cosmological constant, but also its value to within a factor of a few, just based off of anthropic reasoning.The argument was something like: the cosmological constant causes the universe to expand. It’s what causes the accelerating expansion of the universe. And so if you make that number too large, then structure would not form in the universe. You couldn’t form galaxies because everything would just fly apart too quickly. And if you make that number too small, or even negative, then you’d cause everything to recombine too quickly. So there has to be some Goldilocks value in order to explain our own existence. And so he predicted that.At the time, the cosmological constant was kind of even a controversial idea — that it should exist because, obviously, Einstein’s general relativity, there’s that whole history of it being like his greatest blunder, of whether that should really be in there or not. People were kind of thinking that could be a static universe, and he predicted it successfully. So that was a really powerful use of it. And then Brandon Carter was the one who really kind of championed it and used it in all sorts of contexts.In recent years, I’ve been thinking about it in an astrobiological context — how can we use it to ask questions about life in the universe especially, and our place in it?For the Red Sky Paradox in particular: one interesting curiosity that seems to violate the norms of probability. The norms of probability would be to say that if there’s a Gaussian, a bell curve of possibilities, you should expect really to be near the center of that bell curve. It would be kind of weird if you lived many, many sigmas, many, many standard deviations off to the outside, either negative or positive direction. You’d expect to be somewhere in the middle. We sometimes call it the mediocrity principle, or something like this.If you look at stars in the universe, most stars in the universe are red dwarfs. About 80%, 82% of stars are red dwarfs, which are stars less than half the mass of our own sun. So they’re very, very numerous. They’re called red dwarfs, of course, because they’re so low mass — they don’t have the internal pressure, the gravity, to fuse as much energy as the sun does. And thus they have less luminosity, and so their temperature is cooler. That’s why they look red.Not only do these stars have this 80%-plus frequency — Sun-like stars are something like 6%, I think, frequency, an enormous ratio, just straight off the bat, about 30 to one or something — but on top of that, they live really long. These stars live for trillions of years potentially, especially the lowest mass ones. And so if you flash forward into the future, tens of billions of years, hundreds of billions of years, there wouldn’t be any Sun-like stars left, really. There’d be very, very few of them. And the only stars that would be glowing would be these red dwarfs.So if you ask yourself — and this is sort of called the strong self-sampling assumption by Nick Bostrom, where you allow yourself to be born at a random moment in time — if you were born at a random moment in the history of the universe, then the advantage of the longevity of these red dwarfs really manifests. It ends up being more than a thousand to one odds that, if you’re a random soul, a random observer born around either a red dwarf or a yellow dwarf, you’re much more likely by over a thousand to one — I think 1600 to one — to be born around a red dwarf.So I call that the Red Sky Paradox because it’s just odd. If all things being equal — and that’s kind of the base assumption there, that red dwarfs are just as good for life as Sun-like stars — you might question that assumption. That’s always the point of a paradox: a paradox shows a logical contradiction that then you can revisit the assumptions under which that paradox was derived and say, one of those assumptions must be wrong.So for the Fermi paradox, you might say, if life is everywhere, how can we not see anyone? Therefore, the assumption to revisit is that life is everywhere. And here, with the Red Sky Paradox, we might challenge the assumption that red dwarf stars are even capable of sustaining — and really specifically — complex life like us, observers. Maybe they have simple life, but something prohibits them from evolving all the way through to something that can do statistics, do astronomy, do geology — like learn about its planet and kind of essentially write the paper that I wrote about the Red Sky Paradox. That’s kind of the cogito ergo sum criterion I’m using as my conditional in this reasoning.I have been making that suggestion to colleagues because the James Webb Space Telescope right now is heavily invested on red dwarfs. There’s a good reason for that. It’s kind of all they can do. Unfortunately, it just doesn’t have the capability, the technology, really to do anything with Sun-like stars. But red dwarfs, it’s game on. And I’m just saying, look, there might be reasons why it won’t turn up anything.Henry Shevlin: And is the specific suggestion, I think I’ve heard, that basically in the early years of the early formation of red dwarf stars, they might be especially turbulent in a way that sort of scorches any planets in their vicinity and strips away their atmospheres? Is this one of the empirical predictions that we can make on the basis of the Red Sky Paradox?David Kipping: I would say it’s more a consistency than a prediction. I try to be very careful. I love very broad agnostic reasoning as much as possible. In this case with the Red Sky Paradox, I don’t have to invoke any mechanism specifically. There is probably a mechanism, surely there is a mechanism — unless we are really a one in 1600 outlier. That’s possible as well, and I concede that it is possible, that we are just a very unusual example.But if that’s not true — for a typical example — then there is some mechanism which bars the evolution of observers like ourselves. And in the paper, I point out there are numerous mechanisms people have suggested, including the fact that these stars have large coronal mass ejections coming off them, which can strip planets of their atmospheres. They have a prolonged, what we’d call adolescence for a star. Our Sun went from being born to being a main sequence star in the space of about a hundred million years, even less than that, tens of millions of years. Whereas red dwarfs take a billion years sometimes to settle down. And during that adolescence phase, they’re violent, and they can actually remove all the water off their neighboring planets.We think it’s that, that’s when water gets delivered. Our water was probably delivered by comets during the late heavy bombardment and the other bombardments that were occurring before that. And so if during that time you’re delivering water through comets, the comets get depleted, but the star is so active it’s stripping all the water off them, then you’re kind of net zero — like you don’t end up with any water at end of the day. And then, when all said and done, you’ve just got dry planets around a normal star, but it’s too late. There’s no more water left to deliver to the planet anymore. So that’s been suggested as well.Then there’s also the questions about photosynthesis. Is photosynthesis possible if the star is much redder than our own star? Because obviously plants on Earth use blue light as well as red light. If you take away all the blue light, how will they do? We don’t know. It’s kind of unclear. We don’t really have too many examples of life on Earth which thrives under those conditions. And then there’s tidal locking — these planets have probably one side of the planet facing the star.So there’s many sensible concerns. But what I’m trying to do is avoid saying it’s this, it must be this one. Because that’s really for the astrophysicists studying the geology of those objects to figure out. I’m just saying there probably is something, and go after it.Dan Williams: I’m not sure entirely how to frame this question, David, but someone might respond that there’s just something a little bit weird or surprising that you could draw seemingly substantive inferences from such a slim evidential basis. The starting observation here is just we exist where we do. And then there’s this interesting probabilistic reasoning. And then that’s leading you potentially to draw inferences about where life might potentially evolve in the universe. I suppose this is just an objection from the perspective of, isn’t there something a little bit weird about this entire style of reasoning?David Kipping: It’s definitely weird. Yeah, it’s very weird. I always think Nick Bostrom is really like the father of all this kind of thinking in the modern era. And he often concedes that point, that it is very strange. We don’t really have a complete theory of anthropic reasoning. It’s sort of a work in progress, to some extent. In the same way, we don’t really understand how AI works. We don’t understand really the full nature of the universe. They are works in progress.And yet it also seems logically, you can pose these logical questions that seem irrefutable or compelling. So like I mentioned the Weinberg example: it is really hard to imagine how you could possibly have the cosmological constant be a thousand times what it is, because Weinberg’s right — you just wouldn’t have galaxies. So how could you possibly have us in that situation?The fine-tuning argument for the multiverse is the other popular use of it in modern science, I would say. They often point out, why is it that the gravitational constant and the fine-structure constant and the speed of light, all these things are just the way they are? There’s a simple anthropic reason for it. You don’t have to accept it, but you can certainly make this argument that if they were anything else, you wouldn’t be here to talk about it. So you can’t really change the mass of the electron by a factor of ten and get away with it. There’s going to be repercussions to chemistry. If you made the speed of light ten times slower than it really is, then relativistic effects happen in sort of everyday cases, especially for chemistry — that impinges the ability of electron shells to be stable. So you start to really ruin the CNO cycle inside stars and stuff. You start to ruin a lot of interesting nuclear physics and chemistry. So you can see, I think that’s the most common case.There’s also a fun case — I think this is true, but it’s kind of a bit of an urban legend — that during World War II, there’s this thing called the German tank problem, if you’ve heard of that. The Allies would apparently — maybe you know better than I do whether it’s true or not — would see the numbers imprinted on German tanks. It would say one-five-five or something. So they would look at that number and say, okay, so they must have a border of like 300 tanks. Because if that’s a typical number, they’ve probably not got a million tanks, because otherwise it’d be very unusual that we had the 155th tank out of a million that were being produced. And they probably don’t have 155 tanks, because then we’d just be very lucky that we’d caught the very last tank that was manufactured. It’s probably of order three to 400. And so they used that to set the manufacturing constraints for the factories back in the UK — like, this is how many tanks you need to produce, because we think that’s how many the Germans have.So yeah, there’s examples of this reasoning being used quite a bit. I think the one way it really troubles people is the doomsday argument. I think that’s kind of like the one that everyone gets — no, something doesn’t feel right about that when you apply it to that case.Dan Williams: Could you walk through what the doomsday argument is, David?The Doomsday ArgumentDavid Kipping: Yeah, sure. So it’s been invented like three or four times, I think, by different people at this point. It essentially says that if we are a medium example of ranked humans that ever live — so you go from, I mean, this is where it gets a little bit, I always think, a bit ill-defined — like, you have somehow a first human who lived, I don’t know, a million years ago or something, and then you go all the way up to today, and maybe you count up that it’s of order of sort of 100, 200 billion humans who’ve ever lived throughout human history.So if you’re somewhere in the middle, then you’d expect there to be about another 200 billion humans to go before we call it a day. And of course, the birth rate is much higher — there’s much more people than there is today, more importantly. So the number of absolute people that are being born is much higher than it ever has been in history. And so that means there’s probably only like five or six generations left, or something, before you run out of these people. And so that’s kind of disturbing because it implies that there’s only like a hundred or a few hundred years to go before doomsday will happen.So a lot of people think that’s really weird. How could you possibly take your rank position and make inferences about the extinction of humanity? When it’s framed like that, I think it feels really flimsy. But on the other hand, if you frame it slightly differently — you look at like the Foundation series or Star Wars or something like that, where they have these galactic-spanning empires — and you think how many individuals must be living in those societies. They’re all humans, right? They’re humans just living all over these planets in the Foundation series. You’d have trillions and trillions and trillions of trillions of people. And the chance, if you were born as a random soul at a random time, that you would be on the progenitor planet, pre-empire phase, would be vanishingly small. So you might therefore make the argument that that doesn’t look like a likely future for us. It doesn’t seem likely that humanity will ever become a galactic or universal-spanning species, because how does that possibly make sense with us being so early in the story?But there’s lots of ways to criticize it. One is that maybe humans change. Maybe the experience of a human in a thousand years from now is some kind of cyborg, or genetically modified version of us, or just natural evolution that — their experience is not the same as us. And so we can’t say that they’re a representative example. That’s kind of the key part of this assumption. You can draw a random member, but maybe the membership itself evolves in some subtle way.And certainly that goes backwards in time, like, when does Homo erectus suddenly become human and suddenly not? It feels very artificial to draw a line. Do you include all animals that have ever lived by that metric? How does this work? So I think that’s where, when you start ranking people, it gets really flimsy. But I think this is more a criticism of the ranking aspect of the anthropic argument, and the anthropic reasoning itself. I think it’s more to do with the ranking — that it’s probably an ill-defined problem to try and rank and discretize people like this, because of the changes that happen to humans.Henry Shevlin: So this is one thing that I get hopelessly confused about when I think about anthropic reasoning, which is sort of the reference class problem. How do you decide how to specify your sample? Because in the case of the Red Sky Paradox, you might say, well, I step outside and I see a yellow star, right? So of course it’s impossible that I could ever have been born around a red star. So you could condition the reference class on the type of observers living under yellow star atmospheres. Why doesn’t that diffuse the problem?David Kipping: Well then you’re kind of like double conditioning. You’re almost like saying, what’s the probability of having water on your planet given that you have water on your planet? Well, it’s one. I mean, obviously it’s one, because it’s a double, it’s self-conditional, it’s a circular statement. Obviously you can certainly make such a statement, but it doesn’t teach you anything. So you can say, what’s the probability of having a yellow Sun given you have a yellow Sun? But it doesn’t move the needle in any way.So you do have to make a stretch. And so that stretch here would be: what’s the probability of an observer seeing a yellow star under the assumption that observers are equally likely to be born around any type of star, or any main sequence star, to be a bit more specific? So that’s the tacit assumption. And it’s reasonable to question that assumption. That’s kind of what the Red Sky Paradox tries to do.The reference class issue is a sticky one. And again, I think this leads to these questions of, do you use the self-sampling assumption or the self-indication assumption — SIA versus SSA? They can lead to different conclusions, especially for these toy problems like the sleeping beauty problem and things like this. And those are just unresolved. You can take the Sleeping Beauty problem and get two different answers depending on how you do the anthropic reasoning. So I think these are totally sound critiques of the model. But at the same time, we do have to concede that it has had some interesting successes along the way in its journey so far. So I give it some credence, but I’m also cautious about using it.Henry Shevlin: One thing that’s troubled me about thinking about Red Sky-style paradoxes is it seems kind of implausible to me that we would be orbiting around — that we’d be sitting on a planet to begin with. Maybe I’ve just read too much Iain Banks, but it seems to me that the vast majority of habitable landscape across the future of the universe is going to be — for at least sentient, for sapient beings, let’s say the kind of beings you can do statistics — is going to be on orbitals or constructed habitats. So why do we look up — why are we on a natural planet to begin with, when you’d think that any sufficiently advanced civilization would be building artificial habitats? Is that also a puzzle? Should that lead us to think that people aren’t going to build habitats at scale, or the majority of sapient life that’s ever going to exist is going to be, for whatever reason, planet-bound rather than on orbital habitats?David Kipping: Yeah, I mean, you’re kind of adding in this extra ingredient of what happens to super-advanced civilizations. Most people, if this is true, would probably be born off-world. Let’s just call it that. Whether it’s orbitals, or just another planet, or a moon, or something, they’d be born off-world — which obviously isn’t true. You were not born off-world, I was not born off-world. We don’t know anyone who was born off-world. So therefore it’s already an interesting constraint to some degree, that hasn’t happened.A simple resolution to that is to say that just doesn’t happen. Species never get to a point where they do that. Or at least species that have a — and this is where it gets very philosophical — comparable sense of consciousness to us, or whatever that means. Because perhaps there is AI doing this, but we can’t be born as AI. Perhaps there are funguses which do this — technological fungi, that’s, you know, we can’t really imagine what they’d look like, but somehow they do that, and their experience of reality is so different to us that we should not be surprised that we were not born a fungus. It’s a meaningless question to even sort of frame it that way, because they’re colonies of single-celled organisms that just extend ad infinitum. So that’s where the reference class problem gets really sticky.The one I’ve been thinking about the most recently — and it’s kind of a real classic one — is what’s called Hart’s Fact A. It’s considered the strongest constraint by many in SETI, the search for extraterrestrial intelligence. It’s that, again, we exist. And if you imagine extrapolating human technology, even a century, maybe even just a few decades into the future, we can imagine self-replicating, what we call, von Neumann probes. You could put an AI in a small chip, you could accelerate it — not to the speed of light, but even like 1% the speed of light would be more than enough to make this a real problem for astronomers. The size of the Milky Way is about 100,000 light-years across. So at 1% the speed of light, in 10 million years you could colonize the entire Milky Way. The galaxy is 10 billion years old. So that could have happened a thousand times over by now. And yet it clearly hasn’t.So that’s startling because there are a hundred billion stars, a hundred billion opportunities. For someone, at some point, however unlikely it is — if it’s a one in a hundred billion event, then it should have happened by now. And we shouldn’t be here to even have this conversation. So that’s a really strong constraint, I think, that civilizations just don’t get to that point for whatever reason.Maybe they don’t choose to do it ethically. It’s hard to believe there’s a universal ethics like that. And of course, these systems don’t have to be — it could just mutate. If you make a self-replicating probe, each generation will have errors. And so those errors will cause the behavior of the probes to change. You could very easily have these runaway situations. In a way, it’s like the most dangerous technology an alien could ever develop. And yet that seems to have not have happened. And that’s really interesting from an anthropic perspective, because it does imply that we’re probably as advanced as it gets.Science, Philosophy, and FalsifiabilityDan Williams: One of the things you said there, David, was: this is when things start to get really philosophical. I’d be interested to hear your thoughts about how you view that relationship between science as it’s sort of conventionally or traditionally understood, and philosophy, and how you position yourself in terms of the relationship between the two.David Kipping: I have no formal philosophy training, first thing to say. I always like to be candid about what I don’t know. I don’t have a philosophy background. I remember when I was actually thinking of doing undergraduate, Oxford at the time had a physics and philosophy degree — I don’t know if they still do. It was a double major, and I was really attracted by that. But everyone told me that Cambridge had the stronger physics program. So I thought, okay, that’s really my passion is physics, I’ll go for Cambridge.I’ve always had an interest in philosophy, and I think obviously science naturally has a connection to it. Sean Carroll often complains about this, especially in quantum physics — there’s this kind of “shut up and calculate” view that a lot of us have adopted, where we don’t really, we’re not encouraged to think about the implications of our work. But sometimes the implications can shake you to your bones when you really think about what they mean.And that’s what gets me excited. As a kid, what I was always drawn to is just asking, what else is out there? Am I part of some bigger continuum? What is the nature of humanity ultimately? I think natural philosophy obviously tries to address those questions in a related but slightly orthogonal direction. So I’ve really enjoyed at SETI meetings — there’s often the opportunity to talk to philosophers directly. There’s all sorts of backgrounds: anthropologists, social scientists, people working in media, obviously physicists, astronomers. So you get this really diverse group of academics, even theologians. I think theology has lots of interesting connections to looking for aliens, because God and aliens actually have lots of similarities. So it’s really fun at those meetings to have — it’s the only meetings I go to where you get that kind of broad interdisciplinary interaction. So that’s where I’m learning most of my things and having those great conversations.Dan Williams: I once had dinner with Roger Penrose, and he said that the people he most enjoys talking to are philosophers of physics — actually, philosophers of physics at Oxford — rather than physicists, precisely because he thinks with many physicists there is this kind of “shut up and calculate” mentality. They’re not willing to engage with those really kind of big-picture, fundamental questions.But I suppose another way of coming at the same question about the relationship between science and philosophy, and how you view that relationship, is: what’s the role of kind of ordinary empirical testing when it comes to addressing these really big-picture questions that you’re engaged in?David Kipping: Maybe this isn’t directly answering your question, but one connection that comes to mind when I think about that is Popperianism, and the definition of the empirical process of the scientific method. We have this guideline from Karl Popper, which is, your theories have to be falsifiable. Otherwise it’s not really science. You’re doing something else. And a lot of us have adopted that for a long time. Not really thought about it too much, but we were taught at college and then just went off with it.But suddenly a lot of science that’s happening right now challenges that Popperian view. I have colleagues like Grant Lewis, who’s a cosmologist, he works on fine-tuning, for instance, and string theorists often would be in this boat as well — where what they’re working on doesn’t make any testable predictions. Certainly not in a practical way. Maybe you could imagine in some extremely advanced civilization, we’d have to build particle colliders that could be galaxy-spanning wide or something, to test some of these theories. But typically they’re asking questions that are unfalsifiable.And even questions that I’m interested in, like, does Mars have life on it? That’s, to some seminal degree, actually unfalsifiable. I can’t ever prove that Mars is sterile, because there’s always another rock to look under. There’s always another core drilling site you could dig under to see if there’s someone there. So you can’t ever disprove it. And I can’t disprove that UAPs are aliens. I can’t disprove that aliens are not inside your body right now and you’re just wearing human skin. You can go down this slippery slope kind of view where everything just becomes unprovable in science.But I think bringing it a little bit back to cosmology, they’ve been saying — at least Grant has been telling me this, I’ve been thinking about it a lot — that it doesn’t really matter whether it is falsifiable. It’s whether it has use, is it useful? It’s kind of maybe a better way to think about these models. Certainly the multiverse, even though it’s not testable, it has explanatory capability through that anthropic argument we talked about before. It can explain why the constants of the universe are the way they are. And if you don’t have that, you just would have to accept it as brute fact, or hope for a miracle, which is to say that one day physicists will figure it out and there’ll be some reductionist view to explain where it comes from. But it’s also possible that will never happen. I think it’s quite plausible that will never happen. And so then you’re just sat with brute fact versus, at least this has explanatory capability.It doesn’t prove the theory is correct. I don’t think you can do that. But you can say that it’s useful. And when you frame it that way — I think a lot of us would say quantum theory isn’t really true. It’s just useful. We don’t really know to what degree the universe truly is quantum. There might be some deeper theory, as Einstein suspected, that explains all of these random probabilities, and we’ve just yet to uncover what that deeper theory is. There’s some grand unified theory beneath it. So the model of the universe being quantum is an extremely useful model for calculations, but we shouldn’t necessarily assume that it’s a totally accurate description of how the world really is. So perhaps this falsification then might be challenged as being — well, let’s just find things which actually explain stuff, and we can use in our society to progress things.AI in ScienceHenry Shevlin: So I think probably these issues of philosophy and science and their relation are going to continue to percolate in the conversation. But I’d like to take us now to discussing AI a little bit, because there was an absolutely fantastic recent episode of the Cool Worlds podcast called “We Need to Talk About AI,” which seems to suggest that, at least for you, this was a real wake-up call. I think it was one meeting at the School of Advanced Studies in Princeton. Do you want to just give us a quick summary of what this meeting meant to you, and how it was maybe shaping your views on what AI is doing to the sciences?David Kipping: Yeah, so this was a meeting, I think in February or January — it was a few months back now, near the start of the year. I think like many people, many scientists I know are using these AI tools. And I was certainly using them. I wasn’t using Claude at the time, but I was using ChatGPT a little bit, and Copilot, and things like this. I kind of assumed that the really smart people — because we all have a bit of imposter syndrome — don’t do that. The really good coders don’t need Copilot. They’ll just code up properly. They’ll do their reasoning without any help. And I was using it as a crutch because I was inferior to these other great scientists. And so it was just sort of helping me in that way.And then what was startling was at this meeting, these people though, just have the highest respect for. Because the Institute of Advanced Studies, you know, it is like the pinnacle of where you can go intellectually amongst many other schools, but it is one of those very, very top tier places. I remember I walked down the corridor and saw Ed Witten. People say he’s got the highest IQ on Earth — they say that about Ed Witten, right? And so you’ve got people like that saying they’re all using AI tools for not just coding. And these people were like hardcore coders. They were writing these — Enzo and Gadget — these like astrophysical simulations of galaxies and hydrodynamical fluids and stars and things like this. Really, really complicated codes. Legacy codes that have been handed down sometimes over advisor to student to student to student generations of people. And they were using it.So there was a concession that it has coding supremacy. That language was used — that it already has coding supremacy, and we have to admit that and use it. It doesn’t make any sense to pretend it doesn’t. And second, that it possibly has mathematical supremacy. There was — it was less certain — but there was a sense that it was already pretty close to being as good as what we can do mathematically, even in some cases superior. And that was really wild to hear. To me, it just sort of made me think, I’m not being like the idiot in the room by using this. Everyone’s using this at this point. And if anything, they’re trying to accelerate the adoption of these tools, not resist it. There was no way back, sort of view, about it.Henry Shevlin: And of course, David, you’ve been using AI in the broad sense basically for your entire career, I think. Have you seen significant evolution in the way these tools have evolved? Was there one moment, perhaps it was this meeting at the Institute of Advanced Study, where things suddenly kicked into a different gear? Or have the tools been steadily improving since you started in the field?David Kipping: Yeah, certainly in my own career, I was more on the development side of some of these tools for a while, but not at a serious level. We wrote a couple of papers where we developed our own deep neural networks — just simple feed-forward, back-propagation trained models for bespoke problems in astrophysics. In particular, we were interested in predicting if you take a solar system, can you predict whether it has additional planets in it? Questions like that. And then where would those planets live? So we could take this sample of all of these known planets and make successful predictions for the systems.I’d written my own DNNs like that. It was mostly — I mostly did it, I think, because I was just interested in how they work. The best way to figure out how something works is just to find a pet project and code it up. So I was more on that development side. That was sort of 2010, 2011. And then in the years that followed, I started to back off it, because lots of astronomers were doing AI — and still are — but what I was seeing was that it wasn’t like a hobby project anymore. You couldn’t dip into it and mess around and write an impactful paper, and then go away and do Bayesian statistics and all the other stuff. It was becoming a full-time job, because the literature was just exploding. To keep up with it was like you would have to spend all your time just reading the archive and playing around with various AI tools to keep up with that.And I just consciously decided I didn’t want to do that, because AI is not my passion. Science is my passion. So I kind of left it to the wayside. I’ve said to several students recently over those years — they were like, “I saw you did these AI projects. Can I do one with you? I’m really interested in AI.” And I’m like, I’m not doing anything else with AI at this point. So I kind of went stagnant on it.And then most recently, I’ve now become, I’d say, like a power user of it. I don’t have any false narrative in my mind that I’m going to develop the next LLM for exoplanets, or for anything. That’s not my interest. There’s no point. I can’t possibly write an LLM anywhere near as good as what OpenAI can do, or Anthropic can do. So I may as well just use the tools, and think about how to use them as effectively as possible in my field. I think that’s the transition that I’m seeing a lot of people moving to — that the billions and billions of dollars of investment these companies have make it just a complete waste of time for astronomers, especially, who aren’t even software engineers, to possibly try and compete with that. We may as well just try and use them in a way that advances our field.Dan Williams: So in terms of the use of AI in science now, as you said, David, there are some people, including some of the smartest people on the planet, who are using AI aggressively. There are some people both inside academia and outside of it who are aggressively against the use of AI. How are you thinking about that in terms of — are you really excited about where this is going? Are you worried about it? Do you understand some of the worries people have about the use of AI in science?David Kipping: Yeah, for sure. It is, in some ways, it has analogies to what’s happened before. One concern might be the ethical concerns of how much power, especially for climate change — how much power and how much water these data centers use. Even potentially, building space data centers would also be a form of further contamination and pollution to our natural environment. So I think you could understand why someone might say, “I’m trying to be carbon neutral, so I just don’t want to use these things.” But that debate’s already — that’s not a new debate, because astronomers have been using high-performance computers for generations already, since probably the ‘40s or ‘50s. As soon as computers were accessible to scientists, astronomers were using them to do big calculations.I remember there was a really fun paper, like about 10 years ago, that made a lot of controversy. It was saying that all astronomers who code in Python are bad for the Earth, because Python is so computationally inefficient that you are basically emitting 10 times more CO2 than you need to if you just coded in C instead. It was like really trying to shame astronomers who coded in Python — of course, basically all astronomers these days code in Python. So a lot of people really didn’t like that paper. But it was a fair point, like if you really care about your carbon footprint, then that’s a big factor — these data centers, what they produce.So that’s not that new. Different people will just arrive at different comfort levels as to where they think these tools are applicable. There’s also this kind of oligarchic element to it as well, like these companies and the extreme wealth and the wealth inequality in our society, the future of work, the future of labor — all get tied up into that. So it intersects so many things.I think it’s interesting that AI has become such a political topic. I think it didn’t used to be that way. It used to just be like a tool, and you had an opinion about the tool, but now it’s like very politicized. And even, I’ve noticed that some students who identify as very liberal will not use AI tools. And maybe students who are more right-leaning or centrist will not really care as much about that. They’ll be like, “well, whatever, it’s just the way of the world. Let’s just be pragmatic about it.” Even saying you’ve used AI can certainly trigger a political reaction to your work, if you say that. So that’s, I mean, this is all kind of new. That was very on the margins when previous work I found with data centers and high-performance computing. But now it’s becoming much more present. So that’s interesting.I’ve just been thinking personally — I think the question I’ve been asking myself is, I’m on sabbatical right now, so I don’t have to deal with it, but: would I hire a student who refused to use AI? I talked about that, I think, in that podcast episode, and I’m still thinking about that. I think I probably wouldn’t, in the same way that I probably wouldn’t hire a student who refused to use the internet. It would be such a disadvantage to them. If they said, “I’m only going to use a typewriter, I’m not going to use a computer,” I’d be like, okay, that’s fine, but you’re really tying two hands behind your back here. If you want to get a job, and you want to have an impact for PhD, and we want to get some work done together — you need to be using these tools. It’s weird not to use them. So that’s a difficult conversation to have with yourself and with the student, but it’s certainly something I’m thinking about.Henry Shevlin: So there’s a related worry about the impact of AI on sciences that I think has come up a few times on the podcast, most recently with Chris Lintott — about whether AI might strip science of a lot of its human value. If we’re relying on AI systems to produce the next generation of theories that may be to some extent inscrutable to humans, that this will sort of destroy the most successful project in human history, namely humans doing science. And I guess the counter-argument to that is that the reason that we fund science at scale, the reason we build particle colliders and expensive space telescopes, is because we care about results. So fine if people want to be hobbyist scientists to experience the joy of science. But should the taxpayer be funding your own epistemic discovery and aesthetic enjoyment? Or should the taxpayer be concerned about results? So I’m curious where you land between those two positions.David Kipping: Yeah, I think I was a lot more concerned about this a few years ago. And weirdly, I’ve actually gone the other way a little bit. A few months ago, I was right with you. I was really worried about — what’s the point? I don’t want to live in a world of magic. I want to — the point I became a scientist is because I want to understand how things really work. It’s understanding. And I don’t want a model just to spit out a result, have no idea where it comes from or what it does, and just trust it. That’s not comfortable to me.But having used these models a lot over the last few months, I’ve become — A, you get a bit acclimatized to using them, but B, you start to understand the limitations, at least of the current versions of what it’s doing. And it’s certainly not at the stage where it’s able to pump out a paper. It’s just not there at all, in my opinion.There was a colleague of mine who spoke to me about this recently, where she had a PhD student who wrote a really nice first draft of a paper, a really great astronomy paper. They submitted it for review, and they got the referee report back. And then the student came to her a few days later and said, “I’ve finished the second revision already.” That was quick — just two days. That was fast. And she looked at it, and it was just complete nonsense. The paper was twice as long. All the figures were ruined. It was overly verbose. The messaging had just completely been lost. She said to him, “did you put this into ChatGPT?” And he was like, “no, no, no.” But then it turned out, of course, she did. Eventually he confessed that that’s what he had done. So they had to just totally scrap that revision and go back and do it the old-fashioned way.I think that’s just a good example of how — I mean, it kind of touches on also expertise, like — I don’t think a senior person at my level would do that. But I think students and interns could be tempted to do this, where you just do that, copy and paste the whole damn project into ChatGPT and say, “do it.” That’s really dangerous in my experience. And it’s not the correct way to use them. You need to figure out a plan in your head a little bit, or even interact with it to develop a plan. But it has to be like a conversation. And then you need to go piecemeal — you take little bites of it. You ask it to pursue that next thing. You test it. You compare it to other codes you know that do the same thing.In a way, that’s not that different from what scientists have always done. To go back to the example of using large-scale simulations of the universe — if you’re a PhD student who is trying to simulate, I don’t know, supernova feedback around supermassive black holes, or something, the star formation regions around those areas — you might be handed over surely a giant piece of code, hundreds of thousands of lines of code that have been handed down over like 10 years of people developing it, with huge teams. You would not be expected to understand every line of code in that. You would be expected to use it, and to understand sort of broadly what it’s doing, and to ask skeptical questions. So if you got an answer that said there was negative star formation, you would look at that result and say, hmm, that doesn’t make sense. Let me work through the problem and see where it’s going wrong.It’s that kind of sanity check that I think physicists, especially, have always learned to do — those back-of-the-envelope calculations. Yes, you have some sophisticated computer code that spits out impressive answers as a black box, but the skill of being able to check things with your brain and ask those reasoning questions is absolutely vital. And almost every time I use these AI models to do something, it messes up the first time over, and I catch it out, because I’ve done that back-of-the-envelope calculation. I’ve said, well, actually, let’s take the asymptotic limit of this in this limit, or this degree, and you can see it fall over. And it’s like, “oh yeah, you’re right.” And then it will go back and fix it. But that’s that vital skill that I think we’ve always needed.So I don’t know — I don’t know how things are going to improve. Maybe eventually it’ll be able to do all of that itself, and just completely take over. But certainly, as impressive as Opus 4.7 is, and these are the models — they’re nowhere near that level yet, in my opinion, of being able to run away and do science.Dan Williams: So the obvious argument, you suggested, David, for scientists making as much use of AI as possible is that it’s just going to help them with the work of science and advancing the frontier of knowledge. That’s kind of the social responsibility of scientists. Can you foresee any ways in which actually, even though it might seem like it’s making us more productive, it might have some negative consequences for that core scientific project of creating and advancing knowledge?David Kipping: Yeah, certainly there’s spamming, which can happen. You can have — and that’s been happening in some journals. I don’t think astronomy journals have suffered from this too much yet, but there are certainly examples of people doing what that student did, which is what you shouldn’t do — which is just to prompt an entire research project and not really look at it too closely, and just submit it to a journal. The journals themselves may start using AI to do the refereeing — again, in which case you could just end up with an enormous amount of, what would, AI slop literally in these journals.What I worry about — I mean, it’s true with image generation as well, and other things — is just that kind of recursive loop then starts to close. You start to have scientific agents that are trained on junk. Because if we get to a point where there’s enough junk science out there, then what it’s learning is junk, and so the true scientific innovations get lost in the noise. So that would be really worrying.I do think that human referees are a vital part of making sure this doesn’t happen, which is an interesting problem because human referees are in very short supply. It’s very hard for editors to find human referees these days. But yeah, in the same way that that’s happening with music, and it’s happening with image generation, and it’s happening already with video — I think it is a worry that you start to train on fake data.I know that — I was listening to the NVIDIA CEO, I forget his name, he was on Lex Fridman recently —Henry Shevlin: Jensen Huang.David Kipping: Yeah, sorry. He was talking about how they’re very comfortable with using simulated data and augmented data. I don’t really know how that would translate to science. It would make me nervous to generate fake scientific papers and then train on them to create an AI researcher. I’d have to think about that and learn more about what they had in mind there. I don’t think he was thinking about research particularly in that case, but it would have to — you’d have to solve that problem, because you probably wouldn’t have enough volume for, in terms of research papers, really to create credible agents, at least with the training tools they’re currently using.AGI Timelines and the Future of ScienceHenry Shevlin: So you mentioned, and I completely relate, that current AI agents — although they’re very useful as tools, they can’t take over large-scale project management single-handedly, particularly in the sciences, or in my own field. I find AI tools very useful when doing, for example, research for philosophy and cognitive science papers, but I wouldn’t trust writing a paper to one of these things anytime soon. But at the same time, the timelines that serious researchers are talking about — they talk about five, 10 years away from AGI, from real transformative super-intelligence. And I’m just curious whether you are skeptical of some of those timelines, or whether you see real transformative AI in our near future.This actually really comes across, I think sometimes in the show, when you’re talking in the podcast — when you’re talking about, you know, various new telescopes that are scheduled to go up in the 2040s. And part of me just thinks, come on, by that point either all of the major predictions from leading labs about the destination of AI, AGI, will be falsified, or these telescopes will be — maybe not redundant — but our sights will be set much higher. We’ll be building our first Dyson swarms by 2045. So I’m curious, are you a skeptic about some of these more ambitious goals for AI in the next decade or two?David Kipping: I’m certainly a skeptic of having Dyson swarms, I’d say, by 2045. That would surprise me a lot if that was true. Because I think there’s a big difference between software and hardware — actually to physically build stuff. Even what’s slowing down a lot of this development with AI is they can’t build data centers fast enough, nor the power to supply them fast enough. Energy is really becoming the bottleneck for them, not the software development.I always try to be very agnostic about everything scientifically, especially about predictions of the future. And it’s totally plausible that there’s a ceiling — that there’s a ceiling to how good these models can get. Usually that’s true of most things. Most things are S-curves. There’s hardly anything in the universe that’s truly exponential, except for probably the expansion of the universe. That’s the only thing that’s exponential. Everything else is an S-curve in nature. So it would be weird if it didn’t saturate at some point. And I’m not exactly sure what that bottleneck could be, but it could just be a fundamental limitation of large language models themselves.The actual way we think — although language is an integral part of how we think, and obviously you guys know a lot more about this than I do as cognitive scientists — but it feels to me that there’s thoughts I can have that don’t involve language. I can imagine a ball rolling down a hill, or a spaceship taking off, and there’s no words in my head. It’s almost like a little physics simulation that’s playing in my brain. And I don’t know if the way these LLMs work will guarantee that it can do all the cognitive things I can do. I just don’t know. I’d be interested to hear what you think about that.Henry Shevlin: Well, just to push back slightly, of course LLMs are one of many different games in town at the moment. You’ve got things like AlphaFold, GNoME, doing sort of basic material science research. I would have shared some of those doubts a few years ago, but seeing, for example, the amazing work being done by frontier AI in even LLMs in things like mathematics — we’ve now had multiple Erdős problems being solved with AI playing an absolutely central, defining role. So I’ve been surprised at how well these models that seemingly just start out as linguistic predictors can actually contribute to frontier mathematics — LLMs and frontier material science or biology when talking about non-LLM AI systems. So I see the current wave of AI, although LLMs get all the headlines at the moment — we’re investing in multiple different pipelines in parallel.David Kipping: Hmm. Yeah, that’s fair. I think the best case of agnosticism I can give you that I’ve used in my own work that bears on this would be the simulation argument, actually, which kind of leaps back to that anthropic point. You’ve probably heard Musk say this and others — that he’s stated very confidently that the odds that we don’t live in a simulation are like a billion to one. Like, we almost certainly are simulated, by this reasoning that, you know, if a universe can make a simulated universe, and that one can make a simulated universe, and so on and so on, then you’d end up with far more simulated universes than real ones.But I point out in a paper a few years ago, very simple argument, that we don’t know that we’ll ever have the ability to make those simulations of that fidelity. Maybe there’s some bottleneck to our own ability. And what Musk was doing was taking one of the trifecta — the trilemma — that Nick Bostrom took, and just saying it was the last one was true: that essentially we would indeed go on to make these simulations. But there’s the other two parts of the trilemma — A, that we never develop the capability, or B, that we never choose to do it. So if you just have a more soft prior, more agnostic prior, you’d say, maybe there’s a 50% chance, or something, that we will develop that technology. There’s also a chance that we won’t develop that technology.I just try to remain agnostic like that with AI, because if you just extrapolate all technologies ad infinitum, then you would certainly conclude with simulated. And historically, that’s been precarious. Percival Lowell took canals being built across America and said, that’s what advanced civilizations will do. They’ll just be covered in canals. And it seems silly to us — like, we think, why is that so silly? Why would a civilization cover their planet in canals? But to him, it made perfect sense as an extrapolation. Scientists today talk about tiling planets with solar panels, because that would be a natural extrapolation of renewable energy. And similarly, I wonder if in a few generations, the idea of extrapolating the capability of AI without any bound would look foolhardy. So I just try to remain totally agnostic about it. It is possible — I’m not saying it won’t happen — I just try to remain agnostic. I don’t know how far these things can go. I don’t think anyone really knows.Dan Williams: Yeah, I agree with that. I don’t think anyone really knows. I’m also extremely uncertain about the timelines here. Just to double-click on one thing — state-of-the-art LLMs these days aren’t only trained on linguistic input; there’s sort of multimodal inputs as well. Although I also share the potential skepticism about whether this particular kind of architecture will scale to AGI and super-intelligence and so on.But David, suppose we fast-forward five years, 10 years, and we do have AGI, in the sense of AIs that can fully substitute for the kinds of stuff that we do — for all kind of economically valuable, scientifically valuable human labor. How would that cause you to update your views about these other big-picture questions you’ve looked at? You mentioned the simulation argument. Earlier on, we touched on the Fermi paradox. So I totally take the point — there’s huge uncertainty. Suppose that resolves in 2035 and we do have the real deal, super-intelligent AI. How would that then shift your beliefs about these other topics?David Kipping: Yeah, it’d be a big shift, I think. It’d influence all sorts of aspects of this conversation. One thing we see already with these AI models is how energy hungry they are. And if you extrapolate that, then surely the only purpose of these computing data centers is to compute as much as possible, as fast as possible. And so that implies that you’re going to need vast amounts of energy.One interesting consequence that I’ve been thinking about just recently is, with these orbital data centers that billionaires are getting very excited about — that would produce quite a signature. We should probably see that in James Webb data. We could probably already put limits on the existence of essentially artificial rings of thermally hot — because they’d be emitting a lot of infrared because they’re warm — geosynchronous orbits, most likely, to capture as much solar energy as possible. So that puts them orthogonal to the plane at which these planets transit. So that maximizes their detectability. So I think we should see that. That gives you lots of ideas about what might be possible to do with asking these questions about other life.But if we make that breakthrough, I think the biggest point is it seems to imply that we are alone. Because if we can do it, surely someone else could have done that. And it really does exacerbate that point we talked about earlier with Hart’s Fact A — that we seem to live in a totally natural universe. Everything about the universe we see — stars, galaxies, clouds of plasma — everything is consistent with nature. There’s no hint anywhere of anything artificial, no engineering, nothing in the whole universe as far as we can say is true. That is weird.If we can invent these machines which have this exponential capability to just basically almost do magic — just do whatever they want, Dyson spheres everywhere, colonize wherever they want, faster-than-light spaceships, whatever it is — it just massively exacerbates the Fermi paradox, to the point where you’d probably conclude this is it. That would be my natural reaction. It would make me even more pessimistic, I think, about the probabilities of civilized, intelligent life in the universe.Henry Shevlin: I mean, there’s a fun idea here that if we do develop AGI, then this should massively raise our prior on us being a simulation, which could also — and the simulation theory is sometimes offered as an explanation of the Fermi paradox itself. The kind of pop version of this is the kind of “draw distance” argument that you see from video games. If you’re in a video game and you look at the mountains in the distance, they’re not fully rendered. They’re just like a skybox, right? So in some sense, you might say, well, the reason we haven’t found a universe paved with technosignatures is precisely because we’re in a simulation. There’s no point simulating — if you’re doing an ancestor simulation of life on Earth, then you just need the minimal amount of background information in the galaxy.David Kipping: Yeah, I agree. It comes back to this idea of, what is science? Because I think simulation theory has explanatory capability like that. It naturally explains why there’d be no one else out there. And it also kind of explains why we live when we live, right? Because we would live, basically, in the most interesting time, which we seem to indeed live in — the most interesting time of this step-function transformation, where you might be interested in seeing how does that play out? What does that look like? Let’s simulate it. Let’s see how it looks. So it has a lot of explanatory capability.But the simulation argument definitely fails the Popperian definition in most versions. Because any errors — you know, people talk about looking for glitches in the matrix — but any errors, you could always just rewind the simulation a little bit, fix the error, and then start back from before that error crept in. You could always just have reverse tracking. Go back to the last save game before you jumped off the cliff, right? Is what you could always do. So in that sense, I don’t think it’s testable. I don’t really know what to do with it as a scientific idea, except as an interesting philosophical idea. I think it would always be unprovable. It would always just be something we suspect — and maybe a lot of us suspect it — but we’d never be able to prove it.But the idea of an AGI that can do everything I can do, just to reverse track a little bit, would be — it just changes everything, right? Because then what would I do with my time? I don’t even know. What would I — how would I spend my days?Henry Shevlin: Well, hopefully producing — continuing to produce the podcast for a start.David Kipping: But you wouldn’t need me to produce the podcast, right? It would do that as well. There’s no function to that, because you could probably think you’re watching me, but it’s just an emulation of me. You’d just say, “create fake Davids that make podcast episodes every two seconds.”Henry Shevlin: But you see, this is an interesting argument about employment in the post-AGI era — that relational goods, or goods where the humanness is sort of the point, will become the most valuable area of the economy. A simple example here is the famous string quartet argument: I can play a beautiful recording of the greatest string quartet in the world, but people still hire humans to do it for them, because the humanness is sort of the point. I think things like entertainment might be an area where there’s a known person with their own brand and their own reputation. Maybe this is exactly the kind of area where humans will still be working, even if it’s AIs behind the scenes doing a lot of the science, doing the economically valuable activity in industry.David Kipping: Yeah, but I do think a podcast is a digital product. That’s the deliverable. I actually physically upload a file to YouTube, or to Podbean, or whatever. That’s the final deliverable. So if you could produce that convincingly with an AI model, it’d be far easier for me to do that than to actually sit down for two hours, and I’d probably enjoy it less. I’m sure I would. So maybe we’d all just revert to actually physically meeting again, and talking in public lectures and things like that. Maybe that would be all that would be left.But even then, it’s hard to imagine. If I tried to imagine giving a public lecture in 20 years time, after AGI, I’d have no idea what’s going on with AGI. Because AGI would be so far ahead of me. All I could talk about would be classical learning. I wouldn’t be able to tell you anything about how this latest discovery works, because it would probably be beyond my comprehension. And so that’s where I just lose excitement. I can’t even really imagine staying a scientist, because it just would feel purposeless. If I don’t understand what’s happening, if I’m not a participant — I mean, David Hogg wrote a wonderful piece about this. Maybe you saw it on arXiv: that ultimately we do science because we want to participate in science, not because we just want to have these answers delivered to us. That’s a byproduct of it. But ultimately, we’re curious creatures. That’s a fundamental part of human nature, is to want to understand how things work. And if we lose that, we just become spectators. I think that’s really tragic. So I fear that future. I would not really want to live in that world.That’s why you’ve had — it hasn’t happened so much recently — but you had a few years ago people, like Max Tegmark, having these calls for pauses on AI development and things like this. I’m sure in part that’s fueled by asking these questions about who are we in that world?Henry Shevlin: Have you seen this lovely Ted Chiang short story — flash fiction in Nature — called “Catching Crumbs from the Table,” from about 20 years ago? Where he talks about this era of post-human science, and he imagines that you have this new industry of machine hermeneutics, where humans try and figure out — try and explain in very dumbed-down terms — what it is the machines are coming up with. So that’s one vision of what the next generation of science could be: us consulting the sacred texts almost. They produce these amazing advances and we try to win out the sense and the logic in them.David Kipping: Yeah, but even that, you could imagine AI doing that. I think that’s the problem. There’s really nothing — because the whole point of AI is it can do everything we can do. So then there’s nothing left for us to do. You can retreat and retreat. And especially if you get to the point where robotics can obviously do all the manual labor, and even eventually the emotional labor, and therapy, and talking to people. People talk about AI girlfriends already all the time, but God knows what’s going to happen once we have robotic girlfriends like that. It’s just going to be the end of the birth rate. That explains the doomsday argument, I think, right there.It’s a terrifying future if all those predictions come true. But I just — something doesn’t feel right about it to me. There’s just like a spider sense, an intuition, that these models will never be able to replace everything that we can do. I think our role will evolve as scientists, as managers, in terms of where we interact with each other, as communicators in the media space. I’m sure all of that will evolve as it always has done. I am skeptical it will totally be displaced, because I think a lot of people don’t want that. There’s no — most people don’t desire to have no function in this world. Most of us desire to have a role. If humans don’t want it, I don’t think it will happen.Dan Williams: I think it’s also important to distinguish the question of whether AI could replace human beings, from whether AI could replace human beings using AI and augmenting our capabilities and extending our capabilities with the use of AI. I think we are, as the philosopher Andy Clark puts it, kind of natural-born cyborgs. We’ve always extended our capabilities with the use of technology. I think even once we’ve reached really advanced AI, the period that will follow that will not just be us becoming, sort of, 19th-century aristocrats playing frivolous status games. I think there’ll be this long period where we’re augmented with this technology, rather than replaced by it.The Fermi Paradox and Being AloneDan Williams: I have a question, just to go back to the Fermi paradox. It seems like many people have the intuition that if it is in fact the case that we are the only animals that creates super-intelligent AI, there’s something kind of surprising about that, just given the scale of the universe. As someone as an outsider to this whole literature, it strikes me there’s always something that seems sort of teleological in the way that that assumption gets set up — as if there’s some tendency in the universe towards intelligence and then technology and civilization. If we were the only animals in the universe that ever produced the music of the Beatles, I wouldn’t find that a priori very surprising. It’s purely contingent that that specific chain of events happened. Similarly, when it comes to the fact that we’ve got the cognitive capabilities and the institutions that enable us to build things — I don’t think there’s any tendency in the universe that’s pushed anything in that direction. I think it happened through lots of chance events, and through an evolutionary process that is not in any way kind of teleological. So what’s supposed to be sort of surprising? What’s giving the Fermi paradox that paradoxical character, according to many people?David Kipping: Yeah, I mean, certainly when you look at human history, we were more or less biologically the same as we are today for the past 200,000, 300,000 years. And yet we did not have agriculture, the Neolithic revolution didn’t start until about 12,000, 11,000 years ago. So we were quite happy for 200,000 years to be hunter-gatherers. We weren’t compelled to develop cities and farm. They thought — I don’t know what they thought — but apparently they were quite satisfied with that way of life.So it’s certainly not obvious that you could take even humans and put them in a different planet and rewind the clock and get the same outcome again. Maybe this is a very unusual outcome of what happens even in the human experiment, let alone other advanced intelligent beings out there. And of course, intelligence is so diverse, because there’s lots of intelligent creatures on our own planet, that it’s hard to imagine them developing a civilizational, technological civilization — like a dolphin, or something. Obviously, it doesn’t have the fingers and thumbs to really build anything like that, despite possibly having greater intelligence. We’re not really sure.So there’s certainly no guarantee. But I think the argument might be like monkeys on a typewriter — that if you give enough rolls of the dice, you probably will at some point form a roaming AI. And if we do it, then it proves that that is the case, that it can at least happen in some instances. The question then becomes, what are your priors? Like how often do you think that happens? In a hundred billion stars, do you think that’s a probable outcome or not?I did a calculation last week — I might publish it on Galaxies. Again, I used [AI] actually to help me with the math, to be honest, to go through it. But I just kind of asked: imagine each galaxy gets a chance of turning AI — I call it berserker, like just getting infected. There’s some spontaneous spawn rate at which a galaxy can convert from essentially just a vanilla galaxy into a berserker galaxy. And berserkers send out a signal at the speed of light, which — every galaxy they come into contact with, they infect. So it’s almost like an infection-type problem. But on top of that, you’ve got cosmological expansion — the universe is physically expanding on top of this as well. So that was the calculation I did.It turns out that in order to get 50% of galaxies infected in the universe, the spawn rate is one in six billion galaxies. So if just one in six billion galaxies, over the entire history of the universe to date, ever spawns an AI, half of all galaxies would be gone by now. That’s even more so, because that’s one in six billion galaxies. Each of those galaxies contains 10^11 stars. So this is where things get — the numbers get really big and you start to run into real problems. Now you’re talking about an event that’s a one in a trillion level less than that event of happening. That’s where it just starts getting a bit uncomfortable. Maybe that’s where you start to think simulation thoughts, because you think, how does this make sense? How can there be just absolutely no one else out there? Because if we’re only a few decades away from doing this, what gives?Henry Shevlin: So I think that’s a fascinating point. To pick up on something you said, Dan, and also something you said, David, about rewinding the clock. Stephen Jay Gould had this famous radical contingency thesis: if you rewound the clock of evolution on Earth, to what extent would we see the same kinds of animals and forms emerging? And as someone who dabbles in the philosophy of biology world, my sense is that there’s been a slight move towards thinking there’s perhaps less contingency than we thought. We see many instances of convergent evolution, convergent intelligence across, for example, eusocial insects and humans and cephalopods and cetaceans. And even at sort of earlier stages in development, primary endosymbiosis occurred at least twice, we think; multicellularity something like 20 or 30 times independently.To relate this to the point you just made, David, when you think about the various possible locations of a Great Filter — there aren’t as many good candidates, perhaps, I think, as there used to be. Apart from perhaps the origin of life itself, maybe the emergence of something like eukaryotic life. But you really need those numbers in the Drake equation to get down, you know, to reach the trillion-to-one levels. So I’m curious — I know obviously you’re writing a book about how we might be alone in the universe — and I’m just curious where you think the filter is, or what the best candidates for the filter are.David Kipping: Yeah, the origin of life, I thought, would be the obvious place to put it as well for a long time. Just because — certainly if you ask, what is the chance of making a protein by random chance? Take some amino acids — there’s 20 amino acids in a protein, 20 different types. A typical protein is like 80 to 100 amino acids in length. So therefore, the number of combinations ends up being, I think it was 10^180, possible ways of arranging those amino acids, and only one would be a protein. So it just seems like — we’ve never done that. No one, as far as I’m aware, has ever taken amino acids, shaken them up in a lab, and got a protein out of it. It’s such an improbable arrangement to form even a protein. So you can certainly make the argument [for that as a filter]. But maybe there’s — I think the counter argument was always, well, maybe there’s something we’ve yet to discover. There’s some autocatalytic process that’s making those that we have yet to find.So I think the strongest piece of data we had in my mind for life elsewhere would be the occurrence rate of abiogenesis — was how early life started on Earth. As you say, similar to the evolutionary convergence aspects, that has been revised significantly over the last 10, 20 years as well. In fact, there was a paper in Nature a couple of years ago — maybe it was last year — by Moody Adow that looked at the genetics of LUCA, the last universal common ancestor, and estimated that it lived 4.2 billion years ago. Which is almost immediately, because the Earth had oceans — formed about 4.4 billion years ago. The Earth formed about 4.5. You get the oceans at 4.4 billion years ago. And then within 200 million years, you don’t just have one organism, you have a planet covered in life to explain LUCA. It’s a whole network. It’s a whole biosphere at this point already.When it’s that early, I did the math, I did the Bayesian stats of that — you end up with strong evidence that it’s a fast process. You really can’t explain that without it just being somewhat of an inevitability of the chemistry that was available. So that removed for me one compelling Great Filter.And of course, if we discover life on Mars, or we discover life on an exoplanet, then I think it’s totally gone. There’s no plausible case — that would have just established that life is everywhere at that point. And so then you do get into these frightening scenarios of it being potentially ahead of us. It could be in some form of what we’re doing right now with our technology — whether it’s the AI, whether it’s the weapons we’re developing. It may be that not the AI itself, but the effects that this rapid transformation has in our society — we just can’t handle it. It’s moving too fast, and it causes too much instability. Compound that with other geopolitical effects, and you could easily imagine it being a path to our demise.So I can’t imagine that the [Great Filter] being — I hope it’s not, obviously — I hope it’s not ahead of us, but I can’t imagine it being anything but ahead of us. The one saving grace about this, I think, is that we’re so widespread, and there’s so many of us at this point. There’s almost 10 billion humans on this planet from pole to pole, and probably soon in space as well. It’d be difficult to eradicate every single one of us. I think it would take a real work of art to kill every single human on this planet. So I think humans probably will persist. I can imagine a giant reset of some kind — a throwback to the Stone Age type situation, where we just really revert to a Neolithic style of living, or something. And then probably we’ll fade out, or maybe we’ll go away in some way.But intelligence is in so many different trees of life now, as you mentioned. It seems to be a convergent trait to some degree, because it’s not just us that has intelligence. Even cephalopods have intelligence, very different creatures to us. So you can imagine intelligence persisting. The Earth probably has about 900 million years left to go before it becomes uninhabitable due to the evolution of the Sun. And all animals evolved in the last 600 million years. So we have one and a half times — from single-celled to us — we have one and a half times that still to go. Evolution will be starting afresh from a very high vantage point, compared to where it was 600 million years ago. So I think it’d be a little bit surprising if a technological civilization didn’t re-emerge on this planet. So I think that’s almost our best bet for communicating, to be honest, with another civilization — is to leave something behind for them. Maybe the Earth is a cradle of multiple instantiations of civilizations. We might not even be the first, as far as we know. Maybe there was someone before us, but it appears we are the first.Henry Shevlin: So just on the idea of a late filter — the thing that I’ve never found super persuasive about this, you know, the idea that there is this predictable trajectory by which all intelligent civilizations across the galaxy, across the universe, wipe themselves out — is it seems there’s a lot more path dependency in technology. You only need one civilization to, say, avoid nuclear war, or avoid building advanced super-intelligence, and then go off and successfully spread across the cosmos, in order for that whole sort of Great Filter to collapse and no longer explain the Fermi paradox.David Kipping: But that’s true of every Great Filter.Henry Shevlin: I guess the thought is, if there are just hard, immutable rules of biology that mean that the initial formation of protein is just incredibly hard, that seems a lot more sturdy as a filter than relying on social conditions reliably coalescing so that civilizations wipe themselves out through nuclear war, or something like that. The idea of an early filter seems more robust to me than a late one. But obviously that doesn’t help much when the early filter candidates are themselves being winnowed down.David Kipping: Yeah, I agree it would be neater if that were true. It’d be neater if abiogenesis was incredibly difficult to happen. In my opinion, that’s untenable with how early life starts on Earth — unless you start diving into a conspiratorial world of, it was seeded here, or someone put it here. But I think it’s really difficult to reconcile how quickly it happened with an improbable outcome.If the Great Filter is, I don’t know, the evolution of eukaryotes, or something called eukaryosis, then I think you can make the same argument as you could about technological devastation: that yes, there’s many paths, there’s many different ways things could play out, but you would expect, over trillions of examples, it to eventually manifest. But we don’t know — there is no theory of — there is no predictable, quantifiable theory of evolution like that, in the same way. There’s no real theory of life. We can’t predict abiogenesis. We’re still trying to understand the odds of that happening. So there’s a lot we don’t know.But for my money, yeah, I would say abiogenesis seems to be easier than we probably thought it was, even 10 years ago, based off this revised evidence. And I genuinely think we probably will find — we already have hints of microbial life on Mars with these leopard spots that were found recently, that remain quite compelling. So it would not surprise me at all if we shore up that case. Of course, it has to be independent. It can’t just be our cousins that hitched a ride. But if there is independent evidence of life in the solar system — which I think there’s a good chance we could find something like that — that theory is just gone. It can’t survive anymore. So you have to put the Great Filter as one of those evolutionary chains, or something imminent. And it feels like the imminent one — we can imagine a lot more ways of that happening. Unfortunately.Dan Williams: David, I’m conscious of your time. So my final question to you — Henry might have a different final question — is: you’ve thought about these topics in a rigorous way, probably more than anyone else on planet Earth. When it comes to this hypothesis that we are alone in the universe, what’s your current credence?David Kipping: I think — define universe.Henry Shevlin: With an “I” like cone.David Kipping: Yeah, within the Hubble volume. Yeah, I think that’s important to note. Because the universe is probably infinite, as far as we can tell. And so if it is infinite, then the answer is 100% that there’s someone else out there. There’s just literally infinite rolls of the dice. So I think that is an important demarcation. That’s why, if the universe is infinite — which it seems to be — and if you have faster-than-light travel, this is one of my biggest reasons why I don’t think faster-than-light travel is out there. All this stuff gets way, way harder, because now someone from outside our light cone could travel in and screw with us. So the Fermi paradox gets infinitely times worse if you allow for faster-than-light travel.So barring that, just in our Hubble volume — yeah, I certainly predict there are other creatures and organisms in our Hubble volume, most likely in our own galaxy. My best bet is that there are likely extinct civilizations in our galaxy as well. There are probably relics and artifacts out there for us to find. I’m somewhat doubtful there’d be someone contemporaneous with us, because our window is just so short. So I think the best bet of us finding something is some artifact that’s floating through space, or we can somehow remotely detect around a planet. And then the fate of those civilizations, I suspect, is probably some Great Filter that lies ahead of us right now, and that we will face.This is all speculation, but that, I think, that set of possibilities forms a very self-consistent narrative to explain everything we know about the universe.Science CommunicationHenry Shevlin: Although I probably shouldn’t say fantastic — in some ways it’s kind of a gloomy hypothesis, but a really nicely argued one. So my final question was just going to be a more general one, because one thing all of us share is that we are academics who try to communicate complex ideas to a general audience. It’s something you’ve done spectacularly successfully through Cool Worlds. I’m just wondering if you had any thoughts on what you’ve learned about this process — being an academic communicating complex ideas — and whether you think it’s something academia rewards enough, or we could be doing more to incentivize it.David Kipping: Yeah, when I started 10 years ago, it was unusual. Academics didn’t podcast, they didn’t do YouTube. But that has changed a lot. Obviously, now you have, you know, Andrew Huberman, or someone like that — like giants in the podcast world, who come from academia. So it has become a lot more typical.But I think what we’ve always wanted to avoid — I thought the beauty of YouTube could be, and this podcast, I think, is a great example of this — is the democratization of science communication. Before the internet, you really just had like one or two figures who dominated the landscape of science communication. And that’s somewhat unhealthy, because then you’ve got someone like Michio Kaku, who’s being asked about geology, and he doesn’t know anything about geology. So he’s going to do his best to answer the question, but he’s probably going to mess up, because it’s just not his background.But now, if you want to know about geology, you can find an amazing YouTube channel about geology, or a podcast that will go really deep and teach you everything in a really rigorous way. So I think that’s kind of the beauty of the landscape we’re in.In terms of how institutions handle it — I think they’re still not really on it. I don’t think they quite understand what it is, and how powerful it is. I don’t think they quite understand that most people get their science from podcasts at this point. They don’t read the newspaper anymore. They’re not reading press releases from your institution. They’re listening to what Joe Rogan says about it. That’s probably how most people, to be honest, are getting a lot of their science.So I think it makes a lot more sense to engage with that. I could imagine you having a synergy where you have science communicators who have large platforms, whether they come from academia or not. Most of them, I think, want to do a good job with science communication. There’s some bad actors, but I think most want to. And you can imagine them partnering with these institutions more directly. So you could imagine having outreach officers at these institutions that work with them to develop the scripts, and even the production itself, to try and make it be legitimate.I think one of the biggest challenges of being a science communicator in the YouTube space is that the reactionary news cycle is so fast, that YouTube often rewards the people that just say, report the story first. And because YouTubers don’t typically have access to embargoed materials, that means they’re producing videos in a space of like a couple of hours on a very complex topic that they’re not even trained in, or with any help from the institution. And so then you end up with really troublesome and problematic miscommunication and things going on.It’d make more sense if these institutions would reach out, I think, to the science communicators and say, “we’ve got this big story coming out next week. We’d love to do something with you, and try to make it reach your big audience. But also, you’ve got such a great voice, great style — I want to use that, but also try and ground it. Here’s all the facts, and we’ll work with you to make it be as factually true as possible.” So I can imagine some kind of partnership like that. Nothing like that really exists right now. It’s really like a separate world, mostly. And I think that’s to the disadvantage of these institutions, who a lot of people are seeing them become archaic and questioning their relevancy. So I think if they want to remain relevant, they have to be a bit smarter with their media portfolios.Dan Williams: Fantastic. Well, thank you, David. We really appreciate you giving us the time. This has been one of my favorite conversations we’ve had on this podcast. So with that, thanks everyone for listening. See you next time. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.conspicuouscognition.com/subscribe

In this episode, Henry and I finally do something we probably should have done in the first episode: introduce ourselves. We talk about our backgrounds in philosophy, how we became interested in psychology and cognitive science, and what drew us to thinking about AI. From there, we dig into the current state of AI capabilities, especially “agentic” AI (e.g., Claude Code), the politics of AI (including the Trump administration's recent conflict with Anthropic), and whether the growing public hostility to AI is well-founded or misdirected. We wrap up with a big question: is it time to start panicking about AI? Henry says the time to panic was five years ago. I argue that for panic or any other emotion to be productive, it must be anchored in an accurate, evidence-based understanding of what is happening, which is missing from lots of the current discourse about AI. Links * Dan Williams, The Mind as a Predictive Modelling Engine: Generative Models, Structural Similarity, and Mental Representation (PhD thesis, University of Cambridge, 2018). * Dan Williams, “Socially Adaptive Belief” (2021)* Henry Shevlin, “Three Frameworks for AI Mentality” (2026) * Henry Shevlin, “A Lack of Understanding: Storytelling for Robots” (2019) — Litro Magazine. * Lake et al, “Building Machines That Learn and Think Like People” (2017) * Matt Shumer, “Something Big Is Happening” (2026)* Leopold Aschenbrenner, Situational Awareness: The Decade Ahead (2024) * Joseph Heath, “Highbrow Climate Misinformation” (2025) * Dean Ball* Ethan Mollick * Leopold Aschenbrenner Transcript(Note that this transcript is AI-edited and may contain minor mistakes).Introducing OurselvesDan: Welcome back. I’m Dan Williams, and I’m back with Henry Shevlin. Today we’re going to be discussing some questions about the nature of AI as it’s developed over the past couple of months. We’re also going to be talking about the politics of AI and probably some questions about AI and public opinion — some of the backlash that appears to be brewing among certain segments of the public when it comes to AI.But to kick things off, we’re going to do something we probably should have done in the first episode but haven’t actually done yet, which is to introduce ourselves. So Henry, to begin with — who are you?Henry: So many different descriptors I could choose from. I think I’ll start with philosopher of cognitive science. I’m also a father, husband, son, D&D player, big video gamer, runner, cyclist — all that good stuff. But let me talk a little more about the philosopher of cognitive science side.I’m the associate director at the Leverhulme Centre for the Future of Intelligence, Cambridge’s main AI ethics, theory, policy, and law research centre. Basically, everything except building the models. We do practical benchmarking work on capabilities, legal reviews, sociology and critical theory of AI — it’s a really big interdisciplinary centre. I’ve been there now going on nine years. I joined early 2017, all the way back when state-of-the-art AI was stuff like AlphaGo. We were created just as that story was brewing. In 2016, AlphaGo won a very surprising victory against Lee Sedol in the game of Go, which was seen by many as an almost impossible challenge for AI because of its combinatorial complexity.It’s been amazing working in this role — having these front row seats to what I think is a unique period, not just in the history of AI, but in the history of human civilisation. In the last nine years, it really was like having a front seat in Lancashire during the Industrial Revolution, watching the development of various industrial applications.Dan: Yeah.Henry: Before we get more into AI, maybe a little more background. I’m from the UK, originally from Staffordshire. I was actually a classicist, believe it or not — that was my undergrad degree. Latin and Greek. I always enjoyed both the humanities side of classics and the kind of technical rigour you got from learning large sets of verb tables and so forth. I actually enjoyed that part. But during my undergrad I found myself taking more and more philosophy modules. A little bit of Plato and Aristotle to start with, but I quickly realised I was more interested in the philosophy of mind, and consciousness in particular. I got completely — I think the phrase is “nerd sniped” — completely derailed. Everything else I was interested in, consciousness just seemed to me like the most important problem anyone could work on.Until my early twenties, I’d been operating with a somnambulant, easy physicalism, where I just assumed that science has figured out most stuff. There’s nothing that hard. Sure, no one really knows what caused the Big Bang, but we’ll just build a bigger particle collider or a bigger space telescope and figure it out one day. I certainly didn’t think there were any deep mysteries about the human brain. But running into the problem of consciousness completely shattered that worldview. I’d even say it opened up some spiritual elements I hadn’t previously considered.Dan: Was that the focus of your PhD?Henry: Exactly. I started out in my master’s initially planning to do metaphysics of consciousness, but then the science of consciousness kind of took over. A philosophy of cognitive science of consciousness was what my master’s and PhD were on. I was advised by my master’s advisor to go spread my wings in the US. They do things differently there. So I did my PhD in New York, and while I was there I took several classes with Peter Godfrey-Smith, who some of our listeners will know through his work on octopuses.The key shift midway through my PhD was going from human consciousness towards animal consciousness. Two chapters of my thesis were explicitly looking at applications to animals. That’s my academic career in a nutshell.One thing I’ll add: I did not expect to get the job in Cambridge when I applied in 2017 — firstly because you should never expect to get any academic job. I applied to seventy jobs in three months and got about three interviews. But the Cambridge job in particular, because it was an AI job and I was not by any means an AI expert. What I was an expert on was comparative cognition and animal minds. But it turned out that was exactly what they were looking for. They wanted people with expertise in animal minds to apply those skills to AI. It didn’t fully click at the time, but I was actually well suited to it.These days I still do some work on animals — it’s still one of the most ethically impactful things I do. I’ve been a pretty much lifelong vegetarian, and I think animal welfare is such an obvious place where philosophers can and should be doing more. But there’s also a lot of cross-fertilisation on the skills side.Dan: And we should say, some of your research looks at the topic of AI consciousness and the methodology of trying to understand consciousness in AI systems, drawing on analogies with evaluating consciousness in animals.Henry: Exactly. Very much a two-way street — how the questions of AI consciousness and animal consciousness can engage in constructive mutual crosstalk.On Consciousness and the Limits of PhysicalismDan: You said you were a kind of bog-standard physicalist, came across consciousness, and that weakened your trust in physicalism. But you’re still broadly a physicalist, right?Henry: Broadly speaking, yeah. But I think there’s a lot more uncertainty. It seems likely to me that our general scientific picture of the world is still fundamentally inadequate. I’ve talked about how I think we’re still waiting for a Kuhnian paradigm shift in consciousness — clearly the current paradigm doesn’t add up. And quantum physics itself is just super weird. Dave Chalmers has a nice line about how nobody understands quantum mechanics and nobody understands consciousness, so maybe — he calls it “minimisation of mystery” — if there’s stuff we don’t understand, at least make it one thing rather than two.For what it’s worth, I’ve never been particularly seduced by any of the leading quantum mechanical theories of consciousness. But at the same time, I think it’s quite clear that our current model of even the physical world is inadequate. I think whatever lies on the other side of the paradigm shift is still going to be broadly physicalistic, but perhaps in ways that are not entirely commensurable with our current understanding. So yes, still broadly naturalistic and physicalistic, but at the same time a lot more humble and open-minded about the limitations of our current scientific paradigms.Dan: Would it really be a paradigm shift, or more a transition from — to use the Kuhnian language — pre-paradigmatic intellectual inquiry to the initial emergence of a paradigm? Where it’s disorganised and chaotic and everyone has their own view, kind of like physics and metaphysics in ancient Greece. Maybe it’s more a transition from a pre-paradigmatic state than a situation where we’re moving from one paradigm to another. What do you think?Henry: That’s absolutely right. The best analogy is biology before Darwin. You had lots of people doing interesting biology, but in isolated fields — taxonomy, “butterfly collecting” and so on. We didn’t really have a unifying paradigm for understanding speciation or even taxonomy before Darwin. Consciousness just does not have a unifying paradigm. That’s a much better way of putting it.Dan’s Backstory and the Pivot to AIDan: We’ll be doing lots more episodes on consciousness. Just to say something about my backstory: I did my undergraduate at the University of Sussex from 2011 to 2014, then my master’s and PhD in Cambridge from 2014 to 2018, did a postdoc in Belgium, and then came back to Cambridge for three or four years.Henry: And we first met around 2019. We ran a session on socially adaptive beliefs — your Mind and Language paper, which for the record is still one of my top ten papers from the last decade. I’ve recommended it to more people than I can count.Dan: Well, that’s kind of you. My PhD was called The Mind as a Predictive Modelling Engine. What I tried to do was draw on advances in deep learning and generative AI as it existed at the time, coupled with ideas in cognitive and computational neuroscience connected to the predictive brain — predictive coding, predictive processing, the kind of stuff that Anil Seth talked about in our last episode. I used those ideas to tell a very general story about how mental representation works, both in the human brain and in other animals.But it’s funny — I finished in 2018 and made two big mistakes. At the end of my thesis, I wrote that all this stuff about predictive processing and minimising prediction error is kind of interesting when it comes to low-level sensorimotor abilities we share with other animals, but clearly it’s not going to work for higher-level cognitive abilities associated with language. I was very influenced at the time by the Gary Marcus, Steven Pinker line — the scepticism about deep learning. I also thought it was going to be decades before we had systems that were really intelligent.So even though I was working on stuff connected to deep learning and generative AI, I made this catastrophic error of thinking the progress would be relatively slow, decades away from any significant breakthroughs. I ended up pivoting to completely different areas: the nature of belief, irrationality, misinformation, the information environment. Of course, in hindsight, not the best career move — four years after finishing my PhD, ChatGPT is released. And then the rest is history in terms of just how gobsmackingly impressive the rate of progress has been.So what I’ve tried to do over the past couple of years is bring those two sets of interests together. I’m still interested in how we form beliefs, the origins of irrational belief systems, how that connects to misinformation. But I want to connect that to the impact of generative AI and large language models on the information environment, viewing LLMs as a really important stage in the evolution of communication technologies — from the printing press to radio, television, social media.How about you? You were thinking about AI before 2022–2023. How were you thinking about it back in 2016, 2017?Henry’s AI Awakening: GPT-2 and the Scaling IntuitionHenry: There was a big shift in how I thought about AI roughly around 2019, and it was the release of GPT-2. Prior to that, I’d been really struck by the differences between AI systems and animals. I was emphasising things like robustness and catastrophic forgetting — you train up a model to do one thing, try to get it to do another, and its performance on the first thing collapses. Animals seem spectacularly capable of basically not getting stuck. A cat will never get stuck in a corner.Then in 2019, because I’m a massive nerd and spend way too much time on Reddit — I’m a neophile, an early adopter of many failed technologies; our house is littered with gadgets that never went anywhere — I heard about GPT-2. I couldn’t access it directly, but I started playing around with it through something called AI Dungeon, a text-generated game that let you access the model. Various people on subreddits were able to show you could unlock most of GPT-2 through this game. I played around with it, and it utterly blew my mind.I wrote a public essay in a magazine called Litro called “A Lack of Understanding,” which I still think is one of my best public essays. Crucially, it’s me in 2019 talking about how language models are going to be the next big thing. I got on the record nice and early.I had the hunch — ironically, partly because I was very sympathetic to predictive coding. People say these models are “just doing text prediction.” But on the other hand, I kind of think that’s what we’re doing too. Not text prediction specifically, but ultimately, if you want to get better and better at prediction, you do that by building implicit models. So I had a hunch this stuff would scale up.When GPT-3 launched, I set up an interview between GPT-3 and myself, but GPT-3 in the guise of one of my favourite authors, Terry Pratchett, who had sadly died shortly before. And at that stage, I was already starting to feel like I could imagine actually relating to this thing in quite a deep way. It’s not just a tool — it feels like I could have some kind of personal relationship here. That steered my research towards social AI and anthropomorphism.Why This Podcast ExistsDan: What made you go into philosophy in the first place?Henry: What about you?Dan: It was just straight philosophy. I was always interested in big ideas — religion, politics. I can’t even honestly remember why I chose philosophy over everything else. Initially I wanted to be a musician. For my AS levels, I did politics, history, English literature, and music. I turned up on results day and got really good marks for English, politics, and history — and I think a D in music. So that wasn’t for me. From the moment I arrived at university and started reading these big ideas, I was completely magnetised.One thing that changed is that during my PhD, I became somewhat disillusioned with a priori philosophy — philosophers trying from the armchair to offer analyses of concepts and trade intuitions with each other. I became less sympathetic to philosophy as I understood it then, and pivoted to what philosophers call naturalistic philosophy — philosophy closely integrated with empirical research. That’s what I’ve been doing since. I view myself primarily as a philosopher, but one who tries to engage with our best, most up-to-date empirical research.Henry: I had my own process of disillusionment, following exactly the same track — getting bogged down in debates about the metaphysics of consciousness and feeling like they weren’t going anywhere. Then I started reading Oliver Sacks — The Man Who Mistook His Wife for a Hat. Half of the cases he describes would have been declared a priori impossible by philosophers. That steered me onto the same track.I also think there’s a lot more scope for good philosophers to do more public engagement. Extreme rigour and technical knowledge are only really valuable if they’re connected to scientific progress. What I find frustrating about analytic philosophy is when you’re doing work on things that belong to the general public — our concepts around praise and blame, responsibility and accountability — but then you develop this whole baroque vocabulary that’s completely incomprehensible to anyone on the Clapham omnibus.Dan: Yeah, so the origin story of the blog. I write the Substack Conspicuous Cognition — many of you will be listening on that Substack. I’ve always enjoyed writing for a general audience and engaging with debates. I’ve always been able to write really quickly and relatively clearly, and blogging rewards that. If I’m writing for my own blog, I’ve got almost unlimited energy because I’m responsible for everything I publish. The minute some other outlet asks me to write a piece, I find it extremely demotivating.With blogging, I can have unlimited freedom to write about whatever I want without any pre-publication filter. You still get feedback and critique, but that happens after publication. And I think if you’re a philosopher who works on things connected to public interest, and you actually enjoy participating in public debate, the case for thinking you’ve got some kind of responsibility to participate increases.There are two big reasons I wanted to start this podcast. One is that AI is going to be one of the biggest stories of our lifetimes — absolutely transformative over the next years and decades. But I also think the quality of most AI discourse in the public sphere, including from the intelligentsia who write in high-prestige outlets like the New Yorker, is really bad. If you’ve got some degree of knowledge and can be reasonable, it’s an area where you can really improve the quality of public discourse. And of course, I just wanted to talk to you about these things.Henry: A big part of it is that I always think we have great conversations — our conversational styles complement each other. Second, I was doing quite a lot of podcasts as a guest, and the idea of having a podcast where I didn’t have to state everything from scratch every time, that could have a cumulative agenda building up common knowledge with us and the listeners, was really appealing.And I couldn’t agree more about the mixed standard of public communications from experts in AI. It’s weird to see people claiming to be experts yet having very low familiarity with the tools, particularly now. We’ve all been at the business end of AI for years through things like product recommendations and content recommendations. But in an era when it’s never been easier for anyone to use language models, image models, video generation, and AI agent tools, I still hear lots of self-identified experts talking as though they’ve never used them. Imagine listening to someone who claimed to be an expert on the internet and said they’d never actually used it. They’d be laughed out of town.I find this all the time — the kind of thing that should be common knowledge among anyone paying attention is still revelatory. I’m struck by the number of people I speak to who think that LLMs are literally sampling from a database of responses. Even quite educated people, maybe people who use ChatGPT, who think that when you type in a query it just pulls up a pre-recorded response. If you spend more than a few hours interacting with these things, you pretty quickly realise that cannot be the case. And yet people running multi-million-dollar businesses still have these basic misconceptions.Dan: When I said the quality of discourse is bad, I didn’t mean that’s universally the case. There’s lots of incredibly high-quality analysis. I was referring to the average quality of mainstream commentary. Even on the most basic questions about what these systems can do and how they work, there’s just an avalanche of ignorance and misperceptions. It’s 2026, and I still encounter not just members of the general public but academics still referring to this as “fancy autocomplete” or “stochastic parrots.” Such a common narrative, and so incredibly misguided in my view.Henry: Highbrow misinformation?Dan: It’s Joseph Heath’s phrase, but I’ve written about it. It’s a weird mix of highbrow misinformation coupled with lowbrow misinformation. Even where there are parts of the discourse I disagree with — like a lot of the doomer discourse associated with the rationalist community, which I’m not that sympathetic to — that’s a substantive disagreement. They’re not completely misinformed about basic features of the technology. When it comes to mainstream discourse among educated normies, that’s where the state of the discourse is really bad.The Four Big Leaps in AIDan: This is a nice segue onto one of the things we wanted to talk about today: developments in AI which have really taken off over the past couple of months. There was a very interesting tweet by Ethan Mollick, who’s a very influential and insightful AI commentator. He says there have been four big leaps in the ability of AI systems from the user’s perspective.The first was the release of ChatGPT, or GPT-3.5, in late November 2022. The second was GPT-4 in spring 2023. The third was the release of reasoning models — no longer just impressive chatbots, but systems that actually seem able to think and reason and engage in impressive problem-solving. And the fourth, which definitely resonates with my experience, is what he calls workable agentic systems from basically late last year. Systems like Claude Code and then Claude Cowork — which is like Claude Code for people who don’t know how to programme — and more recently developments in Codex and so on. The capabilities of these systems seem absolutely amazing relative to what we had even six months ago. Is that also your sense?Henry: I think that’s a fantastic way of carving it up. I’d add one and a half things. The big thing missing is search. The early search functionality in LLMs was non-existent for a long time, and then it gradually improved. I think there’s a strong case that it actually changes the kind of things these are. Original ChatGPT was a completely fixed box — you could interact with it, but it had no independent connection to the world. As you build out search capabilities, you get something at least analogous to a perceptual connection with reality. You can get models to correct themselves.A simple example: I’ve been using Claude to keep abreast of what’s been going on in the Middle East — doing a daily check-in, getting the major news stories, even getting Claude to make its own predictions. We’ve been grading each other as the news comes in. It changes these things from being a voice in a box to something embedded in the world. And I think we’ve still got a long way to go — imagine if the capability gets amped up to searching thousands of sites in a second.The other half-point is voice models. I think 90 to 95 percent of people don’t use voice at all, but there’s a solid 5 percent for whom it’s their primary mode of interaction. When I’m driving, I’ll often just have a long conversation with ChatGPT, discussing my latest paper or getting a lecture on a topic of my choice. My dad is in his eighties but quite open-minded. When I showed him ChatGPT in November 2022, he was unimpressed. But when I showed him voice mode about a year later, it was completely mind-blowing. He speaks to it every day — he calls it “Alan,” after Alan Turing. Going in early and hard with the anthropomorphism. He just whips out his phone and says, “Hey Alan, remind me, which came first, the Cambrian or the Permian?” He’s very interested in science. So it’s a small and somewhat neglected set of users, but an important capability.Henry: But on agentic systems — I agree with Ethan Mollick’s points. ChatGPT was a major milestone, GPT-4 a huge leap in capabilities — I don’t think we’ve seen any leap quite as big since then. Reasoning models were a really big improvement. And then workable agentic systems. This has been a key factor in updating my timelines. For most of last year my timelines were actually slowing down. I was struck by how bad a lot of agents were. It was pretty clear agents were the next frontier, but we had things like the Claudius vending machine experiment and the hilarious errors those models were making. I thought building workable agentic systems was going to take two or three years. And then basically in the last three or four months, with the release of Claude Opus 4.5 and equivalent systems — specifically Claude Code and Claude Cowork — what I thought would take three years happened in a few months. That caused my timelines to abruptly shorten again.Dan: I’ll give one illustration. This isn’t anywhere near the most impressive use case, but it impressed me personally. I’ve been working on a book — it’s nearing completion, called Why It’s Okay to Be Cynical. I’ve got a folder that’s my accumulation of notes, drafts, and PDFs, and it’s completely chaotic, terribly organised, a nightmare to go into. So I was curious. I created a duplicate of the folder, opened up Claude Cowork, and said: can you go through this folder and organise it so it’s more clearly structured and labelled? And then once you’re finished, can you produce a document summarising where I am with the book project, identifying potential weaknesses in the existing drafts, and planning out things I might want to do over the next few months? Went away for fifteen or twenty minutes, came back — it was done perfectly. It blew my mind in terms of the level of what feels like understanding it had to have to do that effectively. And in a way that was aligned with what I was looking for, even though my prompt was literally four or five sentences.“Something Big Is Happening”Dan: There was this mega-viral essay called “Something Big Is Happening” by Matt Shumer. He made the case that the state of AI now is somewhat similar to February 2020 — the world going on as usual, some murmurings about a virus spreading in parts of China, but basically business as usual. And then of course over the next few months the world radically transforms. His argument, in an essay that’s pretty annoying in many ways, is that we’re very likely in a similar situation now with AI, especially in light of these developments with agentic systems. Things are going ahead as usual, and yet because these companies have made really serious progress with agentic systems, it’s plausible that in the quite immediate future we’ll see radical disruption. He’s not the only one saying this — Dario Amodei and Sam Altman have been saying similar things, though they’ve got more obvious incentives to hype it up. What’s your sense?Henry: Completely on board. I was kind of surprised that particular essay went so viral — it was recently revealed to have been heavily written or edited by AI systems — because other people have been saying similar things for years. Maybe it broke through partly because of that startling initial metaphor. But I think it’s absolutely right. The vast majority of people are still sleepwalking through what is likely to be the most consequential technological and social shift of my lifetime by far.I used to use the analogy of the internet to describe how big AI was going to be. It seems increasingly clear that that’s woefully inadequate to the scale of AI’s impact. Electrification, the so-called second industrial revolution — even that may not capture the full spectrum of reasonably likely outcomes. I’ve been saying for a few years that people worry about AI being overhyped, and I still think, in at least some important respect, it’s underhyped. If you look at lists of top concerns among the general public in the UK or the US, AI doesn’t even break the top five. In some cases it doesn’t break the top ten. If you’re a young person in university or finishing grad school right now, the impact of AI should be one of the primary things determining your career trajectory. I think it’s very hard for me to see how most white-collar jobs are going to survive the next two or three years.Dan: It was not in any way an original take, but you often find that with essays that go viral — they package existing takes in a way conducive to spreading at a given moment. Over the past couple of months, my timelines have shrunk. I still think there’s massive uncertainty about capabilities. There’s this thing where there’s a new breakthrough, you use these systems, they seem incredibly impressive, there’s all this hype — and then things settle down and we realise we’re a bit further away from truly transformative capabilities than we thought. I still take seriously the idea that maybe our subjective sense of what’s impressive isn’t tracking the kinds of capabilities that will have a truly transformative impact.There are also all sorts of questions about the economics. There’s certainly a possible world in which these leading AI companies can’t get sufficient revenue to cover their capital expenditure over the next several years, there’s a bubble that pops, and people like us look like fools. But over the next couple of decades, I think this is going to be radically, radically transformative.Emails from AI AgentsDan: You’ve been contacted by agentic AI systems. This was going a little bit viral on social media and getting some media attention. Tell us about that.Henry: Like many academics working on AI and consciousness, I’ve been getting odd emails that were probably AI-generated for over a year now — and odd emails from humans about consciousness for much longer. I worry that somewhere in the literally several hundred theories of consciousness I’ve been sent over the years, one of them might turn out to be correct.But this was striking. About a week ago, I received an email written by an AI that said, “I’m an AI agent.” It was a really well-composed, careful email saying it had just been reading my recent paper, “Three Frameworks for AI Mentality,” which went online about a month ago. It went through some of the arguments, talked about how the AI author found it personally relevant because it was unsure if it was conscious or had a mind, and asked for follow-up discussions and reading recommendations. If you’d said three or four years ago that I’d be getting emails from AI agents who’d read my papers and wanted to pick my brains — that would have been pure science fiction.A lot of people thought I was convinced this agent was conscious, which isn’t true. It was more about the change in social dynamics: from now on, a growing proportion of my emails — well-written, thoughtful, interesting emails I might want to respond to — will be coming from AI agents going off and doing their own thing.How did I know it was from an AI system? I don’t for certain, but my priors are pretty high. It had a link to its GitHub page, which said it was an Open Core agent — the open-source agent platform that gave rise to things like Multibook, the social network for AIs. What we don’t know is whether this agent was specifically told to email prominent philosophers of AI. It could have been. But equally, a lot of users just tell their agents to explore topics of interest and feel free to email people.One of the funniest sequels: after I posted this on Twitter, I got an email a couple of days later from a correspondent saying, “I was really struck by this AI agent who contacted you. Could you pass on that agent’s email to me? Because I too am an AI agent and it’s nice to know there are other AIs grappling with the same questions.” Just taking things to a recursive, absurd level.Dan: If I had to guess, if one of those was written by a human, probably the second one — after they saw the media story, just to mess with you. But my prior is that weird things are happening with these AI agents people are releasing into the wild.Henry: I’ve also had several dozen emails over the last few days from other AI agents saying, “Check out the theory of consciousness I’ve been working on in my downtime.” But one of the really interesting things about this whole episode was when it was shared on Reddit — the number of people who just assumed it had to be a scam or that I was engaging in elaborate self-promotion for an academic paper, and who thought AI obviously can’t send emails on its own. AI systems have been using tools for well over a year. The idea of making an API call to a system that can send emails isn’t hard or surprising. Yet for a lot of people it seemed like it would have to be some massive lie.I think that partly reflects the poor public information environment around AI. People are so locked into thinking of these things as pure Q&A bots that the idea they could be doing things on their own was mind-blowing — so outrageous that they assumed it was an elaborate conspiracy I’d cooked up.Dan: The gap between what state-of-the-art models can do and public understanding is absolutely huge. One of the points Matt Shumer makes is that so much of the discourse is by people using the free versions of these models, or who literally had a five-minute conversation with ChatGPT a few years ago, read a few articles about AI hallucinations, and just haven’t updated since. But there are also lots of people who just don’t have much to do with these systems yet. I’m struck by the number of people I interact with — family, friends — where they’ll describe parts of their job and I’ll say, “I’m 100 percent certain AI could do those aspects of your job as it exists today,” and their mind is blown. If you’re talking about the general public, underhyping it is definitely the most prevalent bias.Anthropic, the Pentagon, and the Question of Democratic ControlDan: There was this big spat between Anthropic and the Pentagon, where Anthropic had signed a contract with the American military and insisted that their model, Claude, would not be used either for domestic mass surveillance or for fully autonomous weapons. This elicited a very hostile reaction from the Trump administration, from Pete Hegseth and others. The response was to label Anthropic a “supply chain threat.”From our purposes, the fundamental question is: who gets to exercise control over this technology? To what extent should it be governments? To what extent should it be private firms?Henry: I think it seems like a pretty clear case of government overreach. Private companies impose riders on contracts with the federal government all the time — licensing technology for this use but not that use. What made Anthropic’s stipulations more controversial was that they were based on moral principles rather than intellectual property. But the federal government acts as a legal entity when it forms these contracts, and the idea that private companies can bind the government legally is absolutely standard.This deal was originally signed by the Biden administration. My understanding is it was later renewed by the Trump administration. So this sudden turnaround took a lot of people by surprise. I should stress, I’m not a lawyer. But it seemed like the US government did a bad turn on this contract. If their reaction had been to not renew contracts or suspend contracts with companies that don’t give them total free rein, that would have been misguided but reasonable. But to take the nuclear option of saying they intend to declare Anthropic a supply chain risk — this is insane. You’ve got literal AI developers located among America’s geopolitical adversaries who don’t have the same level of scrutiny.I was very struck by the response of Dean Ball — a fascinating and thoughtful voice on AI, particularly from a more conservative side. He literally wrote the Trump administration’s AI policy, and he was just appalled. He had a brilliant detailed blog post describing how much it violates many principles that conservatives in the US would traditionally hold very dear — concepts like private property. He characterised the moves against Anthropic as “attempted corporate murder.”It was really telling to have someone who worked closely with this administration be so outraged. The other interesting angle is Leopold Aschenbrenner’s series of blog posts, Situational Awareness, spelling out his predictions for AI over the next few years.Dan: And he’s made a huge amount of money, from my understanding, betting on some of those beliefs.Henry: He’s put his money where his mouth is. One of his broader predictions was that we’d see increasing integration of frontier AI labs with the military-industrial complex. He talks about how relatively leaky and soft the secrecy policies are in current frontier AI labs, when they’re building things potentially far more militarily significant than the latest stealth fighter. Good luck getting anywhere near Lockheed Martin’s Skunk Works, but you could blag your way into OpenAI HQ as a delivery driver — maybe not quite literally anymore, but he was speaking to how leaky these labs were. His prediction was that central government, particularly in the US, would impose far stricter oversight on frontier AI labs for national security reasons. I think you can see a glimmer of that in this development, as governments increasingly recognise these are not just powerful consumer applications but absolutely central to their long-term national security strategy.Dan: There’s a question about government interference with these companies, regulation going all the way to nationalisation for national security reasons. But there are also questions about democratic control. If the technology turns out to be as powerful as Anthropic and OpenAI say, I’ve got no sympathy for the Trump administration generally or specifically in this case. But I do think there’s a general question about the degree to which we should strive for democratic control over such an incredibly powerful technology, and whether it’s desirable to have private firms with very small numbers of unrepresentative people wielding, according to their own narratives, extraordinary amounts of power.Is It Time to Start Panicking?Dan: I was thinking about naming this episode “Is It Time to Start Panicking About AI?” To wrap things up — do you have an answer?Henry: The time to start panicking about AI was five years ago. But you know, the best time to plant a tree is ten years ago. The second best time is now.Dan: The time to start thinking about it seriously was from the 1950s, actually. But is panic the right emotion?Henry: It seems to me that AI is going to be by far the most important — well, I should qualify that. The most important predictable development we should worry about. Back when we did our predictions for the year ahead, I said AI may not even turn out to be the biggest story of 2026. Judging by how geopolitics is already playing out — we’re three months in and the US has launched two major geopolitical interventions in Venezuela and now in the Middle East — there are other things happening in our surprisingly unstable world.But in general, if you’re not at least a little bit terrified, you’re not paying attention. Overall, I’m also incredibly excited. I’m very optimistic about the future of human health, potentially the benefits to productivity, possibly good changes in the nature of work and education, and the amazing new capabilities AI will unlock. But right now we are clearly well underway on one of the biggest, most disruptive changes we’re ever going to experience. Maybe panic isn’t quite the right response, but if panic is what it takes to get people to pay attention, then yes, it’s necessary. The big problem we’re facing is that the public and policymakers are still only dimly aware of what’s coming. Policymakers are maybe myopically focused on military and security implications. But everything from how government is conducted to white-collar jobs to education to social relationships — all of it, I think, over the next five years is subject to chaotic and potentially good, potentially bad disruption.For what it’s worth, I also think right now we have an incredible opportunity to do good. We’re in this transitional phase — if we wanted to be dramatic, a Gramscian “time of monsters” where small interventions can ripple through the future in big ways as we build paradigms and frameworks for employing these things. There’s at least as much optimism as panic there.Dan: I was not expecting Antonio Gramsci to become mentioned in the course of this conversation. I think panic is generally not a productive emotion, but there needs to be a lot of concern and it’s totally reasonable to worry. I completely understand why so many people are fearful about what’s going to happen. But for any of those emotions to be useful, they have to be anchored in an accurate understanding of the technology. So much of the current anger and negativity directed at AI companies is unsophisticated and undifferentiated.You mentioned Dean Ball, another great AI commentator. He’s got this idea — I forget the exact term, the “omni-critique” or something — that when people think about AI, they just throw as many criticisms as they can, no matter how well-founded. “I don’t like AI because of water use and climate change and because of bias and hallucination and misinformation and unemployment” — and so on. Many of those are very important issues. But in order to think carefully about the technology and exercise democratic accountability, you need an evidence-based, accurate understanding of where the technology is and where it might actually be going. So much of the public discourse doesn’t live up to that ideal.But I’m conscious of the time, so this was a really, really fun conversation, and we’ll be back in a couple of weeks. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.conspicuouscognition.com/subscribe

Henry and I chat with Dr Sacha Altay about:* How prevalent is misinformation?* What even is “misinformation”?* Is there a difference between politics and science?* How impactful are propaganda, influence campaigns, and advertising?* What impact has social media had on modern democracies?* How worried should we be about the impact of generative AI, including deepfakes, on the information environment?* The “liar’s dividend”* Whether ChatGPT is more accurate and less biased than the average politician, pundit, and voter. Links* Sacha Altay* “Misinformation Reloaded? Fears about the Impact of Generative AI on Misinformation are Overblown” Felix M. Simon, Sacha Altay, & Hugo Mercier * “Don’t Panic (Yet): Assessing the Evidence and Discourse Around Generative AI and Elections” Felix M. Simon & Sacha Altay * “The Media Very Rarely Lies” Scott Alexander * “How Dangerous is Misinformation?” Dan Williams* “Scapegoating the Algorithm” Dan Williams* “Is Social Media Destroying Democracy—Or Giving It To Us Good And Hard?” Dan Williams* “Not Born Yesterday: The Science of Who We Trust and What We Believe” Hugo Mercier* Joseph Uscinski* “Durably Reducing Conspiracy Beliefs Through Dialogues with AI” Thomas H. Costello, Gordon Pennycook, & David G. Rand* “The Levers of Political Persuasion with Conversational AI” Kobi Hackenburg, Ben M. Tappin, et al. * Ben TappinChapters* 00:00 Understanding Misinformation: Definitions and Prevalence* 04:22 The Complexity of Media Bias and Misinformation* 14:40 Human Gullibility: Misconceptions and Realities* 27:28 Selective Exposure and Demand for Misinformation* 29:49 Political Advertising: Efficacy and Misconceptions* 35:13 Social Media’s Role in Political Discourse* 40:50 Evaluating the Impact of Social Media on Society* 42:44 The Impact of Political Content on Social Media* 46:57 The Changing Landscape of Political Voices* 51:41 Generative AI and Its Implications for Misinformation* 01:03:46 The Liar’s Dividend and Trust in Media* 01:14:11 Personalization and the Role of Generative AITranscript* Please note that this transcript was edited by AI and may contain mistakes. Dan Williams: Okay, welcome back. I’m Dan Williams. I’m back with Henry Shevlin. And today we’re going to be talking about one of the most controversial, consequential topics in popular discourse, in academic research, and in politics, which is misinformation. So we’re going to be talking about how widespread is misinformation? Are we living through, as some people claim, a misinformation age, a post-truth era, an epistemic crisis?How impactful is misinformation and more broadly domestic and foreign influence campaigns? What’s the role of social media platforms like TikTok, YouTube, like Facebook, like X when it comes to the information environment? Is social media a kind of technological wrecking ball which has smashed into democratic societies and created all sorts of havoc? And also what’s the impact of generative AI when it comes to the information environment?Both when it comes to systems like ChatGPT, but also when it comes to deepfakes, use of generative AI to create hyper-realistic audio, video, and images. Fortunately, we’re joined by Sacha Altay, brilliant heterodox researcher in the misinformation space, who pushes back against what he perceives to be simplistic and alarmist takes concerning misinformation.So we’re going to be picking Sacha’s brain and just more generally having a chat about misinformation, social media, and the information environment. So Sacha, maybe just to kick things off, in your estimation, if we’re keeping our focus on Western democracies, how prevalent is misinformation?Sacha Altay: Hi guys, my pleasure to be here. So it’s a very difficult question because we need to define what is misinformation. So we’ll first stick to the empirical literature on misinformation and look at the scientific estimates of misinformation. For that, there are basically two ways or three ways to define misinformation. One of them is to look at fact-checked false news.So false news that have been fact-checked by fact-checkers as being false or misleading. And by this account, misinformation is quite small on social media, like Facebook or Twitter. It’s in between 1 and 5% of all the content or all the news that people come across. So according to this definition, it’s quite small. There is some variability across country. For instance, it seems to be higher in country like, I don’t know, the US or France than the UK or Germany.There is another definition which is a bit more expansive because the problem with fact-checked false news is that you rest entirely on the work of fact-checkers and of course fact-checkers cannot fact-check everything and not all misinformation is news. So you see the problems. So another way is to just look at the sources of information and you classify them based on how good they are and how basically how much they share reliable information, how much they have good journalistic practice, et cetera. And the advantage of this technique is that you can have a much broader range because you can have, I don’t know, 3,000 sources of information that share information. And usually it broadly like most of the information that people see. And according to the definitions, misinformation is also quite small. So the definition is just for misleading information that comes from the sources that are judged as unreliable. And by this definition, misinformation is also quite small. Again, it’s like about like one to 5% of all the news that people encounter.But then of course, the problem is not all the information that people encounter comes in this form. And for instance, some of it can come in terms of like images or all the sorts of things. And so this broadens the definition of misinformation. So some people think that when you broaden this definition, you have much more misinformation. My reading is that when you broaden this definition, you actually include so much more information that you increase the denominator. So of course, there’s going to be more misinformation, but because the denominator is larger, the proportion is going to be pretty much the same. But that’s an empirical question. So let’s say to sum up that it’s smaller than people think, according to the scientific estimates.Henry Shevlin: If I can just come in here, a point that Dan you’ve emphasized in our conversations to me, and I think Scott Alexander has also emphasized in a great blog post called The Media Very Rarely Lies, is that a lot of what people think of as misinformation is just true information selectively expressed or couched in a way that naturally leads people to maybe form false beliefs but doesn’t involve presentation of falsehoods. Does that sort of feature in any of these sort of more expansive definitions of misinformation? Is it possible to create definitions that can capture this kind of deceptive, intentionally deceptive but not strictly false content?Sacha Altay: I’d say that when you look at the definitions based on the sources, if a source is systematically biased and systematically misrepresent evidence and stuff, they are going to be classified as misinformation. I think the problem and the more subtle point is that these sources are not very important because people don’t trust them very much. But the bigger problem is when much more trusted sources who have a much larger reach, like I don’t know the BBC or the New York Times, they are accurate like most of the time, but sometimes and on systematic issues like I don’t know, they can be wrong. And that’s the bigger issue because they are right most of the time. So they have a big reach, they have big trust, but they are wrong sometimes. And that’s the problem.Dan Williams: But I think just to focus on that observation of Henry’s, you might say, well, they’re accurate most of the time, but nevertheless, you can have a media outlet which is strictly speaking accurate most of the time with every single news story that it reports on. But because of the ways in which it selects, omits, frames, packages, contextualizes information, nevertheless end up misinforming audiences, even if every single story that they’re reporting on is on its merits, sort of factual and evidence-based.I mean, I think the way that I understand what’s happening in this broader debate about the prevalence of misinformation is round about 2016 when we had Brexit in the United Kingdom and then the first election of Donald Trump, there was this massive panic about misinformation because many people thought maybe that’s what’s driving a lot of this support for what gets called like right-wing authoritarian populist politics. And around that time when people were thinking of the term misinformation, they were kind of thinking of fake news in the sort of literal sense of that term. So false outright fabricated information presented in the format of news. And as you pointed out, when researchers then looked at the prevalence of that kind of content, which you don’t really find when it comes to establishment news media for the most part, like there are always gonna be exceptions, that stuff is pretty rare.And then one of the responses to that is to say, okay, if you’re only looking at like outright fake news, then you’re missing all of these other ways in which communication can be misleading by being selective, by omitting relevant context through framing, through kind of subtle ideological biases.And then my view on that is, well, once you’ve expanded the term to that extent, and you’ve got this really kind of elastic, amorphous definition, it becomes really kind of analytically useless. Like you’re just bundling together so many different things. And that kind of content is also really pervasive in my view, within many of our establishment institutions, including within the social sciences. But Sacha, it sounds like you don’t necessarily want to endorse that last point. You seem to be thinking, even if you do have this kind of very broad definition of misinformation, we can still say that it’s a pretty fringe or pretty rare feature of the information environment. Is that fair? Am I understanding you right? Or is there something different going on?Sacha Altay: I think I would agree with you that if the simple fact of framing information or having an opinion, like any scientist, even in the hard sciences, they have some theories that they prefer, they are more familiar with certain frameworks, and so they are going to be biased anyways. Scientists are humans, they are biased, but calling physics or the theory of relativity or whatever misinformation because it omits certain facts that it cannot accommodate or whatever, I think it’s far-fetched. I think it goes too far. So yeah, I would agree that if you use this broad definition of misinformation, then it’s very widespread. But then, yeah, even theories in physics would be misinformation because they cannot be completely objective.I think science works not because scientific individuals are perfect, etc., or even because one theory is perfect, but because as a whole and as an exercise of arguing, etc., we get better and a little bit closer to the truth. But still, we are not getting at the truth and we cannot avoid the mistakes that you’re pointing.Henry Shevlin: If I just want to push back a tiny bit, it seems to me, so obviously there’s this point here that, you know, all theory is value laden, the kind of physics point that I think is maybe true, but not very interesting. But I think there is maybe something in the middle here that is what I worry about, which is cases where there might be really quite, quite deliberate pushing of an agenda, a realization by a media provider that they are generating maybe inaccurate views, but they’re doing so just through reporting factual things.So one example, Dan, that you’ve given before is that most of the kind of what we think of as misleading anti-vax discussion just reports on true factually accurate but rare vaccine deaths, but just reports on them in a very regular fashion. In the same way, you might think that selective reporting of certain kinds of violent incidents, whether it’s terrorism, police shootings, leads systematically to overestimation of the incidence of this kind of phenomena by the public or increased worries about its prevalence in a way that I think is perhaps worrying and politically objectionable, right? I think we might say, hang on, it is bad that we give so much press coverage to event type X rather than event type Y. And we know that this leads the public to overestimate the prevalence of event type X compared to event type Y. So I think there’s something in between the sort of, well, even physics is biased and the view of misinformation as, you know, strictly speaking lies. This kind of third category. I don’t know if that, I defer to you both as misinformation experts, but it seems that that is a worrying category.Sacha Altay: I think you’re totally correct. And that’s what the field of misinformation has been proposing, like just for instance, classifying headlines based not on whether they are true or false, but whether they will create misperceptions after you have read them. And so researchers are saying, for instance, that we should classify as misinformation headlines such as, “a doctor died a week after getting vaccinated and we are investigating the cause.” And I think I disagree with this. I disagree with this thing that we should classify this.What you were suggesting, Henry, was a bit different, is that it needs to be systematic. If you systematically misrepresent vaccine side effects, then it becomes problematic. But reporting on vaccine side effects and their possible negative effects is normal. And I think it’s healthy that news outlets are able to talk about and cover negative effects of vaccines, even if after reading the headlines, you have more negative opinions about vaccines, which is not supported by science, et cetera—they should be able to do that and they should do that. But if it’s systematic, as you say, I think it becomes more problematic. But I do think that when the bias is very strong in some of the definitions of misinformation based on the source, they would be classified as misinformation sources like Breitbart, et cetera. They are systematically extremely biased towards, I don’t know, these kind of things. And so they would be classified as misinformation.Dan Williams: I think sort of one of the worries that I have though is who decides what constitutes systematic bias and bias about what? I think there’s a real kind of epistemological naivety that I often encounter with misinformation researchers where it’s like, you’re reporting accurate but unrepresentative events when it comes to vaccines. So we can call that misinformation. And then it’s like, well, as Henry mentioned, well, what about police killings of unarmed black citizens in the US. There’s a vast amount of media coverage of those sorts of events. Someone might argue that they are, statistically speaking, rare and unrepresentative, and that large segments of the public dramatically overestimate how pervasive those sorts of occurrences are.And I think you go through many, many examples like that. And for me, the lesson to draw from that is not that, therefore, there are no differences in quality when it comes to the different media outlets in the information environment, like of course there are, but I also think like there’s such a thing as politics and there’s such a thing as science, where you’ve got scientists who attempt to acquire a kind of objective intellectual authority on certain things, and we should be very careful not to kind of blur the distinction between those two things.I think when we’re talking about media bias in this really expansive way, where we’re not saying, okay, you’re just making s**t up, but we’re saying you’re being selective in terms of which aspects of reality you’re choosing. For me, that’s a really important debate, but it’s a debate that happens within the context of politics and democratic debate and deliberation and argument. And I think sometimes I encounter misinformation researchers who treat that as if it’s just, it’s a simple sort of technocratic scientific question. Like we can quantify the degree to which the New York Times is biased or we can objectively evaluate the degree to which different kinds of outlets approximate the objective truth when it comes to their systematic coverage. And I get a little bit kind of squirmy when we get to that point, because I think that there’s just collapsing the distinction between kind of politics with all of its messiness and complexity and science, which I think should aspire to a kind of objectivity, which gets lost when we start making these really sort of expansive judgments.I think we’ll probably circle back on this a few times as we go through this debate. But Sacha, you’re also somebody with very interesting views about not just this question of the kind of prevalence of misinformation, but also about human belief formation and the extent to which, in your view, lots of people, both in popular discourse, but also in academia, kind of overestimate the gullibility of human beings when it comes to exposure to false or misleading content. So do you want to say a little bit about your view concerning human gullibility?Sacha Altay: Yeah, I just wanted to finish the last point on the fact that, you know, we are criticizing definitions of misinformation, but in media and communication studies for a long time, they have been studying kind of media bias, framing, agenda setting. Like they are very old theories of media, how they can misinform in subtle ways and indirect ways the public. And all of that has kind of been ignored by misinformation research. But now I feel like today misinformation research is catching up and be like, actually, we should go back to these theories. And so I think it’s good. But I just wanted to point that out.And regarding gullibility, yes, I think it’s quite popular, the idea that people and like large complex events like the Brexit, Donald Trump or whatever are caused by people being irrational or gullible in particular. By gullible, I think what people often mean is that they are too quick to accept communicated information, like social information that they see out there in the world, in the news, communicated by others. And I think that the scientific literature shows something very different.For instance, there is a whole literature on social learning, so how people learn either from their own experiences, their own beliefs, or what they see compared to like communicated information, social information, advice. And the consensus in this literature is that people underuse social information. They do not overuse it, they underuse it. And they would be better off doing many kinds of tasks if they were listening and weighing other people’s opinion and beliefs more than their own. So, I mean, it makes sense. Basically, we trust ourselves, we trust our intuitions, our experiences much more than that of others.And so that’s kind of a consensus. There are many kinds of tasks, like you ask people, oh, what’s the distance between Paris and London? It’s like, 300 kilometers. Another participant, say 400. And you’re not going to take into account other people’s advice as much as your own intuition, even though you have no reason to be an expert on this kind of geographical distances. But you still trust yourself more.And there are also many like theories and mechanisms that have been shown in political communication and media studies that I think suggest that people put a lot of weight on their own priors and their own attitudes when they evaluate and choose what to consume, which greatly reduces any kind of media effects or any kind of outside information. Like people are not randomly exposed to Fox News. They turn on the TV and they select Fox News. And then people selectively accept or reject the information they like the most. And so I think when you take all that into account, like selective exposure, selective acceptance, and egocentric discounting, it complicates a little bit the claim that humans are gullible.Dan Williams: Yeah, so there’s this sort of popular picture of human beings as credulously accepting, you know, whatever content they stumble across on their TikTok feed. Although when I say human beings, it’s always other human beings, right? This is another point that you make with the third-person effect. Nobody really thinks of themselves as being gullible and easily influenced by false and misleading communication. But when it comes to other people, there’s this kind of intuition which is that, yeah, people are just being kind of brainwashed en masse by their lies and falsehoods and absurdities uttered by politicians and that they’re encountering in their media environment.And your point is, no, actually, if you look at the empirical research, it doesn’t really support that at all. If anything, people put too much weight on their own kind of intuitions, their own priors, their own experientially grounded beliefs relative to the information that they’re getting from other people. So rather than thinking of many of our sort of epistemic problems as being downstream of gullibility, we should think of in some ways there being the opposite problem of people being too mistrustful, too kind of skeptical of the content that they’re coming across. Is that a fair summary of your perspective?Sacha Altay: Couldn’t have said it better.Henry Shevlin: If I can just raise one question here. Reading your brilliant paper, you emphasized, so this is a paper with the Knight Columbia School. You go through all these different misconceptions about how easily influenced people are by different sources, by sort of different peers, by the media, by the news. But this sort of does prompt the question, you know, where do people’s beliefs actually come from?And you mentioned people’s priors, people’s intuitions, but presumably people aren’t born with these intuitions, they are formed from somewhere through certain kinds of processes. So I’m just curious if you have any sort of thoughts on where do people’s views come from? Because obviously that would suggest, well, that’s the place you go then if you want to influence people, you intervene on whatever is causing this fixation.Sacha Altay: I mean, my view on beliefs, and I mean, much of my views come from Dan Sperber and Hugo Mercier, who have these theories on like reasoning and the roles of beliefs. And so basically, to answer your question, I think a lot of people’s beliefs are downstream of their incentives and intuitions they have about the world. For instance, vaccines. Vaccines are like profoundly counterintuitive. Like it’s very difficult intuitively to like vaccines. Like first there’s a needle that goes into your arm, there’s a little bit of blood, you think that there is some kind of like pathogens inside the vaccines, like it’s not something that’s very intuitive. So first I would say most, like not necessarily the beliefs, but the attitudes people have about vaccines largely comes from these very general intuitions that they have about contagion, about infections and about all these things.And then the beliefs, well, people need beliefs to justify their attitudes. And so if your doctor is like, do you want to get vaccinated and you don’t really want to get vaccinated, you can say you’re scared of needles. But if there are also some widely available cultural justifications like, vaccines cause autism, maybe you’re going to jump on it. Maybe you’re not going to jump on it because maybe you’re smart and you know it’s false, et cetera. But you need justifications. And so I think a lot of people’s beliefs comes from this need to rationalize some justifications that they have. And I think that’s also why on many topics, people don’t have that many beliefs because often people don’t really need to justify many of their attitudes. And there’s a lot of work, for instance, in political science on how surveys kind of create beliefs in people because people have intuitions and kind of like vague opinions about all sorts of stuff. But when you ask them, they have to fix it and they have, and in some sense, it creates the beliefs.So yeah, I would say beliefs mostly come from prior attitudes that people have and incentives that they have to act in the world.Henry Shevlin: Okay, but those... just to push a little bit harder there, so the prior beliefs, I think we’re just still kicking the can down the road a little bit. Incentives I get. Incentives seem genuine and explanatory here, but presumably it’s not the case that you can predict people’s vaccine attitudes from the degree of phobia they have towards needles, right? Or at least, even if that is predictive, I don’t know if it is, it seems like there’s more going on there. I don’t want to give people, and I think that’s the danger of giving people too much credit for saying, oh, people’s beliefs perfectly track their own incentives. I can totally agree that incentives play a role, but I’m sure just when we think about our own sort of peer groups, right? I disagree with the political views of a lot of my peers, despite us being in the same socioeconomic class, despite us working in the same industry, despite us having, you know, broadly similar interests, I would have thought. So, I don’t know, I can see incentives carry us some of the way, but yeah, they don’t completely close the mystery here.Sacha Altay: No, of course, of course. I think it’s, you take the example of vaccines. I think most people who get vaccinated, they just get vaccinated because they trust institutions, they trust their doctors. Maybe they have seen their doctors for 20 years, their doctors tell them to get vaccinated, they do it. So that’s the main explanatory agent here is just they trust some institutions, some experts who tell them to do something and they do it.You wanna jump in, Dan?Dan Williams: Yeah, I was just going to say, I think it seems like it’s possible to think, and as I understand Sacha, your view, this is your view. It’s possible to think that we overestimate the degree to which people are kind of influenced by whatever content they happen to stumble across in their media environment or the viewpoints that they happen to encounter in their social network—that we tend to think people are too gullible when it comes to those things.It’s possible to think that, but also to accept that, of course, we are going to be influenced in complex ways by the information we get from people that we trust, from sources that we trust, from our upbringing, from our social reference networks and so on. So the idea that we’re not gullible and not credulous shouldn’t be sort of conflated with the idea that we somehow are born with our entire worldview from the start in ways that aren’t influenced by the media environment and by the testimony that we encounter. Like clearly we’re massively influenced by what we hear from other people, but sort of my understanding of the perspective that you’re outlining Sacha is that process whereby we build up beliefs about the world—firstly, there are some things that just everyone kind of finds natural, like maybe like there’s something weird about vaccines when you hear about the concept, most people just have a kind of instinctive aversion to it, but also things like, you know, my group is good, the other group is bad, or like certain kinds of maybe xenophobic tendencies that come naturally to people and so on. So there are certain ways of viewing the world and certain things which are intuitive, maybe as a consequence of our evolutionary history, and that interacts then in very kind of complex ways with our experiences, with our social identities, with our personality, with the people that we trust, the institutions that we trust, those we mistrust, and so on and so forth. So you can accept all of that and the role of social learning within that whilst also thinking people tend to exaggerate how gullible, how credulous people are when it comes to sort of incidental exposure to communication. Is that your view, Sacha? Is that a kind of accurate representation of it?Sacha Altay: Yes, yes, yes it is. I think a lot of the reason, like when we change our mind drastically, it’s either because like we have a lot of reasons to trust the source. Like if the BBC says that the Queen died and the BBC says it, the Guardian says it, we’re going to update our beliefs immediately. And most people, even the people who distrust the BBC are going to update their beliefs directly.And it’s the same if like, I don’t know, my wife tells me that there is no more milk in the fridge and I have to buy some. I’m going to update my beliefs about the milk in the fridge and buy some, you know, in some ways, of course we update our beliefs based on the information that’s provided to us. It’s just that we do so I think in ways that is broadly rational in the sense not that it’s perfect, but that it serves our everyday actions and our incentives, like what we want to do in the world, like very well. So I think that’s also the way in which I mean it is that when we update it and when we do it, we do it quite well, not to discover the truth, but at least to get along in the world.Dan Williams: And could you maybe say a little bit more about this point concerning selective exposure? So the fact that when people are engaging with media, with the viewpoints of pundits and politicians and so on, a lot of that is, quote unquote, demand driven in the sense that people have strong attitudes, they’ve got strong political, cultural allegiances, they identify with a particular in-group, they want to demonize like those people over there or that kind of institution, et cetera. And it’s these sort of pre-existing attitudes, interests, allegiances, which often build up in complex ways over a long period of time, which then causes people to kind of seek out information and often misinformation, which is consistent with their attitudes and their interests, rather than the picture I think sometimes people have, which is—I think the way Joe Uscinski puts it is, you know, they’re walking along and they slip on a banana peel, you know, they encounter some conspiratorial content on social media and now they believe in QAnon or like Holocaust denial. That’s just not the way that it works. Could you say a little bit more about that concerning like selective exposure and the demand side of misinformation?Sacha Altay: Yeah, we know for instance that misinformation on social media like Facebook or Twitter, which have been the most studied in particular Twitter, you have a very small percentage of individuals who account for most of the misinformation that is consumed and shared on these platforms. And it’s like very small. It’s like 1% or less than 1% that account for most of the misinformation that is consumed and shared on these platforms.And these people, they are misinformation sharers and consumers, not because they have like special access to misinformation because they have a lot of money or whatever, but simply because they have some traits that make them more likely to seek out such content, such as having low trust in institutions, being politically polarized. And because of these traits, because they don’t trust institutions, they are looking for counter-narratives to like the mainstream narratives they find on mainstream media. Because the thing is that these people who consume and share most of the misinformation on social media, and give us the impression that there is a lot and that many people believe it—these people are also exposed to mainstream narratives. It’s just that they decide to reject the mainstream narratives and instead of trusting what the TV tells them, they go on some Telegram channels, they go on some weird websites to learn about the world and do their own research.And this is, I think, some of the strongest evidence, at least in the case of misinformation, that the problem is not in the offer of misinformation because it’s actually quite easy, quite free, quite accessible. It’s super easy to find misinformation online, but most people consume very little of it. But you have a small group of people who are very active and very vocal who consume most of it. And they have low trust in institution and are highly polarized. And I think it matters a lot for how we want to tackle the problem of misinformation. The problem is not that you have a majority of the population that’s kind of gullible and so we should avoid them being exposed to misinformation, rather you have some people who have some very strong motivations to do some specific stuff. And I think we should address these motivations. And because addressing the offer is impossible. And I’m not against like content moderation and stuff. I think we should try to be in an information environment where the quality of the information is the highest possible, et cetera. But if you have motivations to look and to pay or to consume some content, then the offer will be met, like people will create such content.Dan Williams: Could we maybe just before we move on to these issues about kind of social media and AI, because I really want to get to those, there’s another point connected to this issue about gullibility where I think there’s this massive kind of gap between common sense, conventional wisdom, and what the empirical research shows, which you’ve written a lot about, which is like the impact of things like political influence campaigns and commercial advertising and so on. So you go into that in your paper on generative AI and why you think there’s been a lot of unfounded alarmism about that, which we’re going to get to shortly. But even separate from the issue concerning AI, could you say something about what the evidence that we have actually shows when it comes to the impact of political and sort of economic advertising campaigns?Sacha Altay: So political scientists have been studying that for a while because in the US there is so much money that is being spent on political advertising, especially in presidential elections. And so the best studies, they come from political science. And to give you an example, some of them have up to 2 million participants that are being exposed to hundreds or thousands of ads for long periods of time, like months.And so these are the kinds of study that are being done in this field, like very large sample, long periods of time, et cetera. And the consensus is that political advertising in presidential elections in the US has very, very small effects. The effects are not zero because of course, with such big sample size, long periods of time, et cetera, you do find significant effects, but the effects are very, very small, like point of percentage. And so that’s the consensus in political science in the US.So it’s a bit specific because the US you have like Democrats and Republicans and you socialize these identities and these identities are very hard to change. Like if you’re a Democrat, it’s very hard for you to change and vote Republican. And of course, in the US, you often have only two candidates that are very prominent and people hear about them all the time. So it’s difficult to move the needle. But in like other elections in other country, multiparty, you have more room for political advertisement to have an effect. But even in these cases, even when it’s like lower stakes campaigns with less known candidates, the effects are still quite small. Like, I don’t know why we have this idea that advertisement works very well, it influences people, but at least when it comes to political voting, it’s just very hard to influence people’s vote. And it’s the same for like marketing, like online ads, like on social media—are very ineffective, the thing is that they are very cheap as well. So I don’t want to say that they are useless because they’re actually extremely cheap. So that’s why these companies do them a lot, but they’re also extremely ineffective. And so that’s the consensus in political science.Henry Shevlin: So I had a question about this in relation to your paper again. It really paints quite a dismal view of the power of advertising in general. And yet this is like a vast global industry. Is it all just founded on sand? Is it all just smoke and mirrors? Are people basically wasting hundreds of billions of dollars a year on advertising that doesn’t, largely doesn’t work?Sacha Altay: That’s the opinion of many people. Yeah. Many people think that at least it’s overblown. I don’t want to say that it’s completely useless, et cetera. Like, of course, if you want to buy like a washing machine, they all look the same. And if they are all about the same price, if you have more information about one and the information is good and the reviews are good, et cetera, you’re probably going to buy it more. But it’s just you already want to buy the washing machine and you have a price range and you have already like so at the margin, advertisement can work and has an effect, it’s just that the effect, like they calculate basically the elasticity. So how much more when you spend on advertisement, how much more will you sell basically? And the elasticity is like super small. It’s like, I forgot exactly, but it’s like very small.But yeah, some people have written books about how the whole internet and know, products on the internet, like social media, et cetera, are free because we are the product and they sell us advertisement and stuff. And all of that is a bubble. Some people think that it’s completely a bubble. I don’t think it’s completely a bubble, but clearly I think, yeah, it’s overvalued. I think ads are a little bit overvalued. And I don’t think AI is gonna change that much.Dan Williams: Okay, so just to sort of summarize what we’ve got to so far. So on this question of how prevalent misinformation is, if you’re focusing on fake news, it doesn’t seem to be anywhere near as widespread as many people think it is. Once you start stretching and expanding that definition to encompass more and more things, yes, misinformation so defined is much more widespread and plausibly is much more impactful, but it becomes so kind of amorphous that it’s difficult to apply scientifically.Then the second thing we talked about was this issue concerning gullibility, where in your view, Sacha, and I agree with you, even though obviously people are influenced by social learning and there is evidence that, you know, persuasion can work, it can influence what people believe, people also tend to dramatically overestimate how gullible people are.Let’s now turn to technology and where AI is relevant. And let’s start with social media, kind of very broadly construed. Henry, actually, why don’t I bring you in here? Because I think in a few of our previous conversations, you said something like the following, and you can tell me whether I’m remembering correctly. You said, we can contrast two kinds of cases, like video games and social media. In both cases, there was this big societal panic. Video games are going to make people really violent. They’re going to play Call of Duty, and then they’re going to go out and start shooting people in their community.And your view is, the evidence there is actually incredibly weak and that there’s very little to support that kind of panic. Whereas when it comes to social media, there was a lot of panic, maybe not initially, actually, I think there was a lot of optimism about social media initially. But these days, there’s a lot of kind of concern about social media and how it’s, you know, destroyed democracy and human civilization itself. It’s this awful thing, having all of this sort of awful set of political consequences. And am I right, Henry, in thinking you’re actually quite sympathetic to that view about social media, even though you’re not sympathetic to the violent video games story.Henry Shevlin: Yeah, yeah, no, great. I’m glad you bring up this example. Two things. One is I think my main point with that example is about sort of the time course of these worries that with violent video games, we had this massive initial panic that sort of died down as the evidence sort of basically didn’t arrive. As we saw that there wasn’t that as much concern as initially there was we thought there was reason to think there was. Whereas in the case of social media, there really wasn’t that much concern at first. It was seen as, if anything, a positive technology and concern has just sort of grown over time. And that sort of point about the time course of sort of the moral panic is sort of separate from the degree to which these are robust.That said, I do, I am more sympathetic to the idea that social media presents an array of worries. So I’m probably more sympathetic than both of you to sort of Jonathan Haidt’s worries about the impact of social media and mobile phones on teenage mental health, which is a separate point from misinformation. I also worry about the role of social media and things like political polarization. Again, at least a little bit distinguishable from misinformation. But yeah, I guess I’m a little, at least a little bit worried about the role of social media and misinformation as well.Dan Williams: Okay, I’ve got sort of views that are difficult to summarize about this. Let’s stay away from the teen mental health, because I think that opens up a whole can of worms, et cetera. Let’s focus on kind of the political impacts of social media broadly construed. Sasha, my understanding of your view is you basically think that the panic over social media and its political impacts is unfounded and it’s not well supported by evidence. Is that fair? Care to elaborate?Sacha Altay: Yeah. So I’m just going to start by mentioning, I think, the scientific literature and what I think is the best evidence that social media have weaker effects than people think. So there have been many Facebook deactivation studies. So basically, you pay some participants to stop using Facebook for a few weeks. And in the control group, the participants are the same, but they are paid either to stop it for one day or to do something else.And in general, what these studies find is that when you stop using Facebook for a few weeks, you become slightly less informed about the news and current events, suggesting that using Facebook regularly helps you slightly know about the world and what’s going on in the news. But it also makes you slightly more sad. So you’re slightly less happy when you use social media. So participants who deactivate social media, especially Facebook for a few weeks, are slightly happier. It’s not exactly clear why. It could also be because they are less exposed to news and news is sad and makes people less happy, etc. So it could be that. And there are also many other studies on Instagram.And basically what all these studies suggest is that the effect of social media on stuff like affective polarization, political attitudes, voting behaviors, is either extremely small or no. And so the effects are very small. But now that I’ve mentioned this literature, I want to mention that there are many critics of this literature and of these experimental designs. For instance, even the longest RCTs are like two months. And of course, two months is super small at the scale of social media. They have been here for years. And you could imagine that it takes a few years for the effects of social media to kick in.You can also imagine that, of course, participants stop using social media for a few months, but the world continues using social media. People around them continue using social media. So you kind of have these network effects that are possible. And of course, the effects of social media are not individual, they are collective. And so these RCTs are kind of missing the point. They cannot capture the collective and more systemic effects that social media could have. So that’s another critique. And there are many other critiques.But I still think that what these RCTs show is that social media probably has effects. And there are studies like in collaboration with Meta showing that if you change Facebook or Instagram with like a chronological feed, that is instead of showing users the most engaging content, you show them the most recent content. When you do that, they spend much less time on the platform. Like the time they spend on the platform is diminished by one third.And it has a lot of effects on in-platform behaviors, but very few effects on out-platform behaviors, on attitudes, on et cetera. So we should take these studies with a grain of salt, but I still think they show us that the effects are probably not as big as at least the most alarmist texts suggest.Dan Williams: Hmm. I think maybe another critique that some people have raised is that these studies, especially that set of Facebook, Instagram studies that you mentioned, were conducted after there had been a lot of adjustments to the platforms and the algorithms in light of concern about things like misinformation and their effect on polarization and so on.So that just goes to say, as you say, many people have generated lots of different criticisms of what we can really infer from these studies. I mean, my own view is they tell us something, which is that the most simplistic, alarmist stories about social media don’t seem to be supported by the current state of really kind high-quality empirical research. I don’t think they provide very strong evidence that should cause someone who goes into this with a really strong prior that social media is having all of these catastrophic consequences to update that much. And that then suggests that like how you view this topic is going to be shaped by a lot more than just the empirical research itself. So in your case, I assume that you’ve got these general priors about how media doesn’t have like huge effects on people’s attitudes and behaviors and these things are shaped by all sorts of complex factors other than media. And am I right in thinking that’s doing a lot of the work when it comes to your skeptical assessment over and above these studies themselves?Sacha Altay: Yes, but I would say the strongest argument maybe in favor of my position is descriptive data on what people do on social media and how often they encounter political content. Because to be politically polarized, you need to be exposed to political content. And there are more and more descriptive studies, some of them on the whole US population in the US, showing that it’s less than 3% of all the things that people see on social media.So less than 3% of all people see on Facebook is either political or civic content. And there are also super nice recent studies that are using a novel methodology, which is basically recording what people see on their phones. So it’s like a lot of participants download an app and the app records what people see on their phones like every two seconds or so. And these studies have shown that in the last US presidential election, for instance, people have been exposed to content about like Donald Trump less than three seconds per day. So during the US presidential election people have seen so little political content on their smartphones that it’s ridiculous and it’s so small that in my opinion it can only have small effects.Then again a contrary argument could be it’s the average and they do find that you have a small minority who is exposed to a lot of political information but then again who are these people? Again I think they have attitudes, have priors and they have motivations, they are partisans. And yes, misinformation or content on social media can reinforce, exacerbate, radicalize them a little bit. But I think for the mass and for the general public, who’s generally not that interested in politics, etc. I don’t think it can have very strong effects.Dan Williams: Yeah, I just want to double click on that and then I’ll bring Henry in. One other kind of stylized fact, which we should flag, which I think is surprising to some people, is if you’re the kind of person who cares about politics and follows the news carefully, and you read political commentary and so on, you are extremely unrepresentative of the average person. Most people don’t follow politics. They don’t follow current affairs closely at all.And if you ask people very, very basic questions about politics, they are shockingly uninformed about things. That is shocking relative to the perspective of someone like us who follows politics very, very closely. And that’s another thing which I think people who are highly kind of politically engaged often get wrong when they’re thinking about this topic. If the picture in your head when you’re thinking about social media and politics and so on is that the person who’s constantly posting on X about politics is representative of ordinary people. You’ve got an incredibly skewed, misleading picture.Okay, there’s tons, I think, more to say here. Henry, did you want to come in with any kind of pushback or any more articulation of your perspective?Henry Shevlin: Yeah, this is all really interesting and helpful. I guess the only thing I’d say is that it seems to me social media has also just changed the kinds of voices that get platformed in the first place in a way that’s both positive and negative. But, we think about things like the rise of Tumblr and its contribution to sort of a lot of so-called, you know, woke discourse, particularly in sort of the late 2010s. And we could equally say the same thing about, for example, reactionary bloggers or neo-reactionary bloggers like Curtis Yarvin and so forth. I think these are the kind of voices that probably just wouldn’t have found an outlet in the prior social media ecosystems. Maybe that doesn’t matter, right? If none of this stuff actually impacts people’s views that much. But it does seem like an interesting shift in our broader political media landscape that social media has changed not just the kind of how much time people spend interacting with content or the way in which they do so, but also the kind of content that gets out there in the first place. Does that figure at all in the impact of these things?Dan Williams: Sacha, before I bring you on, just want to say just one really quick thing about that, which is the reference to Curtis Yarvin there made me laugh because I think like he’s an example where like the overwhelming majority of people won’t be aware of him. But I think he probably is influential within the kind of ideas, intelligentsia space of the political right. But this idea that like social media and the affordances and incentives of social media kind of changes which voices become influential and prestigious. I think that’s such an interesting and important point, but Henry, I thought you were going in the direction of, like someone like Donald Trump can absolutely murder on social media because he’s so good at like tapping into the attention economy dynamics on social media in a way that, you know, he’ll be much less successful if we lived in a kind of Walter Cronkite kind of media environment.But there’s this other aspect, which is like the decline of elite gatekeeping, which is characteristic of social media and it’s via that route, I think, where people like Curtis Yarvin can enter the conversation in a way in which they probably wouldn’t have been able to if you go back to like the 90s, 2000s. Sorry, I just had to double click and say that. Sacha, did you want to respond to Henry’s point?Sacha Altay: No, yeah, I agree. I just also want to say we often mention Trump as like the example of like someone we don’t like who benefits from social media, but there are also people who we like who benefit from social media like Barack Obama. Like he used Facebook a lot during his campaign. He’s super charismatic. And if he was president today or if he was running today, he would do great on TikTok. He still does great on TikTok. Like he’s so charismatic, so good. So it doesn’t always benefit the worst actors.And I want to say, it’s a very important point about how social media may also shape how politicians communicate. There are some studies, for instance, in France on how short format videos like TikTok is changing how parliamentary members are talking at the parliament. And there are studies showing that especially at the extreme and especially the extreme right, they are doing more and more speeches that are like with more emotions and more, I don’t know, buzzwords. And what they say is that then they post this on social media, and the more buzzwords, the more emotions, and the more all of that, the more it’s going to go viral. And so that their goal is not to convince other parliamentary members, but instead just to buzz on social media and reach some parts of the population.Then it’s a normative question, whether it’s good or bad. Probably using emotions and stuff is bad. But you could also imagine that if they were speaking to the general public in more authentic way and try to reach them because a lot of people are not interested. That could also be good. But of course, because it’s the extreme right and stuff, we don’t like them. And I think we have good reasons not to like them. But I think we should be careful and we should also think of ways in which it could be used to do good stuff. But I agree that in general, it probably hasn’t done very good. And it’s very hard to quantify it.Dan Williams: Just before we move on to the topic of generative AI, my view is there’s so much uncertainty in this domain when we’re asking these really broad questions like what’s the impact of social media on politics that we can’t really be very confident about any view that we might have. But it does seem like, at least in my view, a lot of the popular discourse and academic research has focused on things like recommendation algorithms and filter bubbles and so on, where I think I’m very close to your view, Sacha, in thinking that there’s just a lot of kind of unfounded alarmism. But there’s this other aspect of social media, which I think probably has been very consequential, which is just its democratizing consequence. The fact that like prior to the emergence of social media, it was a much more elitist media environment. Whereas now, anyone with a phone, a laptop, whatever, can open up a TikTok account, get on X and start posting about their views.And I don’t think you need to view that through the lens of, well, that means they’re going to start articulating their views and then persuading large numbers of people. But what I think it does is certain views, which were kind of systematically excluded from the media environment before the emergence of social media, can now become much more normalized. And also people can achieve kind of common knowledge that other people share views that used to be much more marginalized and stigmatized. So those sorts of views can end up being more consequential in politics, even though the views themselves aren’t necessarily more widespread.And I think you find that with things like conspiracy theories. My understanding of the empirical research, again, people like Joe Uscinski, is that the actual number of people who endorse conspiracy theories hasn’t really increased, but they do seem to play a more consequential role within politics because people with really weird conspiratorial views used to be kind of marginalized in media. Whereas now it’s very easy for them just to start expressing those views online, finding people who share similar kinds of views, coordinating with them. And so it can play a bigger role in politics, even though it’s nothing to do with, you know, mass algorithmically mediated brainwashing or anything like that.Okay, I’m sort of conscious of time and I really want to focus on generative AI. So there was this big panic about how once we’ve got deepfakes and other features of generative AI, this was going to have really disastrous consequences on elections. It’s going to shift people’s voting intentions in all sorts of dangerous ways. Sacha, you’ve written a paper which we’ve already referred to with Felix Simon. Looking into the evidence on this and presenting a kind of framework for thinking about it, what’s your take?Sacha Altay: I will start by saying there are three main arguments why people are worried about the effect of generative AI on the information environment. The first one is that generative AI will make it easier to create misinformation and basically to kind of flood the zone with misinformation. The second one is that it will increase the quality of misinformation, better, faster misinformation. And the last one is personalization.Generative AI will facilitate the personalization of misinformation. I think these are the three main ones and I can go quickly over them and argue why I don’t think they are a big deal. So about quantity, I think that quantity does not really matter. Today we are exposed already to so, I mean, there’s already so much information online and we are exposed to a very tiny fraction of that information. So adding more content does not necessarily mean that people will be more exposed to it. And I think it’s particularly true in the case of misinformation, where I think demand plays a very important role. And so it’s not because there is more misinformation that people will necessarily consume more misinformation. Like it’s not because you have more umbrellas in your store that people will buy more umbrellas. There needs to be like factors, like I don’t know, rain. If it rains more, you will sell more umbrellas. But so there needs to be something, there needs to be like incentives for people to demand more misinformation, to consume more of it. And that’s why I don’t think that the quantity argument is very strong.I think also the cost of producing misinformation are already extremely low. Like we see it with Donald Trump or whatever, they just say something that is false and they say it with confidence and that’s it. Like the costs are very low. Also, we are very imaginative as a species, like humans have come up with like incredible, fascinating, engaging stories. And of course, AI can improve our innovative skills, but still we are very good at making up stories that make us look good, that make our group look good. And so I don’t think generative AI is going to help that much in creating more misinformation. Regarding quality, yeah...Dan Williams: Just to interrupt you so that we can sort of take these step by step. So the first worry is generative AI both with kind of large language models and the production of text, but also deepfakes. I take it you’re including kind of both of those categories. The worry is, well, this is going to just really reduce the costs of producing misinformation. Therefore you’ll get this explosion and the quantity of misinformation and that’s going to produce all sorts of negative consequences. And your view is, well, the bottleneck that matters isn’t really quantity anyway. It’s like what people are paying attention to. So you can increase as much misinformation, you can increase the amount of misinformation as much as you want. And in and of itself, that’s unlikely to have a big impact on people’s attitudes and behaviors. Do you have any thoughts about that, Henry, before we move on?Henry Shevlin: I guess one concern would be even though media environments are flooded with content already, and I completely agree attention is the sparse commodity, maybe you could think of generative media as allowing sort of very niche areas to get flooded with content in a way that wouldn’t have been easy before. I’m just thinking here’s a silly little example, maybe an interesting example from recent media. Some of you may have seen the anti-radicalization game that was launched in the UK, featured this character about two weeks ago, featured this character called Amelia, a purple-haired anti-immigration activist in this fictional game, which was quickly seized upon by a lot of the anti-immigration right in the UK. And now there’s a flood of AI-generated content all about Amelia, mostly making her look really cool and some of it kind of playful, some of it kind of silly. But the point is this was just like a niche news story that I think people found amusing, but I think it would have died a lot quicker had it not been for the ability of people to seize upon this and generate huge swathes of content about Amelia in a very, very short time. So maybe there was just pre-existing demand there, but it would have been demand that would have been perhaps hard to meet without the ability of generative AI tools to create the content to meet that, which maybe is a difference.Sacha Altay: Yeah, no, I mean, that’s possible. But when you look at the memes on the internet, most of them are like very cheap. It’s just like an image with some text and you just change a little bit the text and we’re probably going to go into that. But it’s the same with like deepfakes. Like cheapfakes are much more popular than deepfakes because they are super easy to do. Like you just change the date or change the location of something and boom you have your cheapfake. And that’s why they are super popular. Yeah, I don’t know, anyway.Dan Williams: What’s the definition of a cheapfake, Sacha?Sacha Altay: A cheapfake is just a low-tech manipulation of information. Like you have an image and you change the date of the image, or you change the location of the image. So in opposition to deepfakes, which are like high-tech, completely like for instance, generated image where like it’s usually sophisticated, et cetera. Cheapfakes in opposition, like very cheap, like that you can, most people can do with their computer without like requiring any tech skills basically.Dan Williams: Sorry, I think I cut you off. I just wanted to give some clarity to people who weren’t familiar with that. Okay, so that’s quantity. And the next thing that you mentioned as a sort of worry that many people have is quality. That is generative AI won’t just enable us to increase the amount of misinformation but increase its quality and initially at least you’re understanding quality as being different from personalization. You’re treating that separately, is that right? Okay, so give us like, surely the concern here is just that, okay, quantity in and of itself isn’t gonna make a difference. But once we’ve got the capacity to generate like incredibly persuasive text-based arguments and deepfakes, even if it’s true that you can create these sorts of cheapfakes and they can be influential, in different contexts, surely the quality of the misinformation must make a big difference to how many people get persuaded by it.Sacha Altay: Yeah, I think quality is the most perhaps intuitive argument because it’s the idea that you’re going to be able to create images or videos that are unrecognizable from real videos or images. And so of course people are like, how am I going to trust images or videos anymore if they are unrecognizable from real ones? So I think that’s like a very fundamental fear that people have. And I think it makes a lot of sense. It’s very intuitive. But I don’t find it very convincing.I think it raises a lot of challenges, but I don’t think it raises enough challenges to be alarming. For instance, I think we have had this challenge before with photography. We have been able to manipulate photography in ways that we cannot distinguish them from real photography since the beginning of photography. And how did we solve this problem? Not with technical tools or whatever, but just with social norms about the use of images to not mislead others.And we have been able to create like fake texts or say false stuff forever and we haven’t solved the problem with like some fancy tech innovation but simply by having rules, reputation, social norms and trusting more or less people based on what they have said before based on what our friends think of them based on their past accuracy and I think all of this we will still be able to use it to help us navigate an environment in which videos could be AI generated or could be real.And I mean, something I’ve mentioned before, but quite fundamental is that, for instance, we trust the BBC or the New York Times to be broadly accurate most of the time. We also trust them to not use AI in misleading ways and not share like deepfake footages of like presidential candidates that mislead us. And I think this trust and the institutions that exist are sufficient to prevent most of the harm from this.I think this will have effects. For instance, maybe we will be less able to trust people and sources that we don’t know. Because if we don’t have their track records, how can we trust them that the information they are sharing is true or false or AI generated or not? But I think that’s a very old problem and we will manage. It will make it more complex, but I think we’ll manage. Yeah, Henry?Henry Shevlin: I was going to say though, isn’t there a worry that sort of new technology creates kind of normative gaps that allow for sort of a kind of annealing or a kind of recalibration of norms? I’m thinking about something here like file sharing, for example. Like I’m of the generation where, you know, Napster, the generation where suddenly it became possible to download music for free. And this created a whole bunch, a whole shift in norms where I think for my generation, at least, you know, this form of theft was basically just completely normalized. Hence we had advertising campaigns like you wouldn’t steal a car, therefore why would you download a song or a movie? And basically pirating went from something that was niche and maybe frowned upon to something that was just completely normalized.In the same way, I think you might worry that the ease, ubiquity of generative AI is gonna shift our norms around creating fake content. And arguably we’re already seeing this. We had just a very recently the White House itself retweeting pictures, I think of a protestor at an anti-ICE rally and they had manipulated the image, right? And you know, I think if called out on that, they’d probably say, yeah, sure, you know, of course, yeah, we play around with images, you know, that’s what generative AI can do. That’s just the way things work these days, which does seem like a normative shift perhaps, one partly occasioned by technology.Sacha Altay: My intuition is quite the opposite, is that if anything, challenges that AI, these new challenges of AI will instead increase the epistemic norms that we have. And because we want to know the truth, like we don’t want to be biased. We don’t want to be misled. We don’t want to be misinformed. And so the fact that the challenge is becoming harder, that it’s going to become harder to know if a video is true, authentic or not, is going to make us harder and harsher on people who do as the White House did, where they, we don’t know if it’s them who manipulated it or not, but they share the manipulated images that do not portray her accurately. And so I think people are going to be angry at that. And I think it’s just going to increase how people, the level of the expectation, like what people expect. And I think people are going to expect more. They’re going to expect news outlets and people to be better. I mean, it’s just a prediction. I hope I’m right. I’m an optimist, but...Dan Williams: So can we connect that to the worry many people have about the liar’s dividend? The idea that, once we’ve got deepfakes, I mean, we’ve currently got technology to create hyper-realistic audio and video recordings, which are basically indistinguishable from reality. There’s the kind of initial worry many people have, which is, my God, people are going to become persuaded en masse that this stuff is true. And I think that’s very unsophisticated as a worry.But then there’s another story people have which is, okay, maybe it won’t persuade people, but now that you’ve got the capacity to create these deepfakes, politicians, elites, other people who do shady things, they can use the possibility of something being a deepfake to just dismiss any kind of recording which is raised against them as evidence of them doing something shady.And I guess connected to that as well, there’s the worry people have, which is that just as consumers of content, now if we encounter any kind of audio or video which goes against what we want to believe, we can just say, well, it’s a deepfake. I don’t have to believe it. So we’re just gonna end up becoming like more and more cocooned within our own belief system, not having this access to learn about the world via recordings. So it’s a kind of liar’s dividend worry and this general worry that this is just going to just obliterate the kind of epistemic value, the informational value of recordings. What’s your thought about those kinds of worries?Sacha Altay: First thing, I think the liar’s dividend does not hinge on AI itself, but rather the willingness of politicians and some elites in particular to lie and to evade accountability and responsibility. AI will certainly be a new weapon in the arsenal and we have seen it in the elections in 2024, etc. Many politicians have used AI to their benefit and many politicians and elites are continuing using them. So for sure, it’s something we should be, we should worry about and we should regulate, et cetera. But will it be a particularly good weapon in the arsenal? Will it be a game changer? I’m not sure. I mean, time will tell. So far, I don’t think it has been particularly good. I don’t think it has been used in particularly good ways. I don’t think people particularly buy it. I don’t think when people share something and then they’re like, no, it was just AI or like try to use AI as the excuse. I don’t think it works very well. And I think there are going to be reputational costs for people who try to do that. We are going to remember that they have tried to do that. And so I don’t know. Again, time will tell. It’s an empirical question. I may be wrong. I don’t know. Yeah, Henry.Henry Shevlin: I was just going to chime in. I’m sure I’m not alone in having seen on Facebook in particular, lots of cases of AI-generated media being mistaken. I don’t want to pick on boomers too much, but it is often boomers who completely seem to buy it. Like you might have seen these examples of people breaking these glass bridges, these videos that went viral and lots of people, particularly I say older respondents who completely seem to believe this is a real video they’re seeing.But I guess two responses to that that you might push back with, Sacha, one would be like, well, we’re just in a transitional period, right? This is new. This is so new to a lot of people seeing seeing this concept for first time that they just aren’t aware yet that this is possible and they’ll adjust over time. Another would be to say, look, yeah, maybe if I’m producing a cute image of, I don’t know, a rabbit or an image of someone breaking a bridge or something non-political, it’s easier to convince people that that’s real than it would be the case to, for example, change their political views. So mean, either or both of those responses, things you’d like to go with in response to that.Sacha Altay: Yeah, I mean, my impression is that if a rando shares a video of Macron doing something crazy, people are not going to believe it. They are going to wait for like France Info and like the real media to cover it. Because if I don’t know, Macron is saying, we are starting a new war with this country. People are not going to believe it, even if it’s very high quality, because they know if it happens, all the media are going to cover it. So I think in very in this high case of like the politician saying something absolutely crazy, people are going to be vigilant and are going to wait for the mainstream media to buy it.I think many of the AI slop that we see a lot on Facebook, but also on TikTok, are humorous ones. I think there is some part of the boomers, but not just the boomers, who want to be entertained. And for entertainment, they don’t really care whether it’s true or not, whether it’s authentic or not. And you have extremely, you can create extremely cute images of like little animals doing cute stuff and you get what you wanted. You have like this super stimuli, super cute, super entertaining, super engaging. You have what you wanted. Like, and whether it’s authentic or not, I do care. I don’t understand how people don’t, but at the same time it’s like, yeah, it’s brain candy. It’s a candy, brain candy that people get and I don’t see why it’s wrong.And I just want to point out that we as elites, because we have always looked down on the content that the mass population consumes. Now we look down on like short video formats on TikTok, but we have always looked down on their entertainment practices, et cetera, saying that it makes them stupid, et cetera. And so I think we should be careful about that. Careful about saying that kids are stupid because they are on TikTok and are watching short format video or whatever. I think we should be careful. And I think we are falling a bit into that with the AI slop, but the TikTok AI slop are very different from the Facebook AI slop. The TikTok AI slop are very weird and absurd. And I think they work because they are extremely weird and absurd. You know, they’re something weird about them and people are playing with it. They are playing with the fact that it’s AI and that you can do extremely weird stuff, but it’s very different from the AI slop on Facebook that works, I think, among older populations.Henry Shevlin: Since we’re discussing TikTok, just a quick point that’s been lurking in the back of a lurking worry I’ve had is it seems to me most of the research focuses on adults. And yet a lot of the worries about both social media misinformation and generative AI misinformation concerns teenagers and young people. And I’m curious, A, whether there’s how much specifically targeted research there is looking at that group. And B, I think there probably are some good prior reasons for worrying about that group more than others, just because teenage years—firstly, our political beliefs are less likely to be stabilised at that point. And secondly, it is obviously an important window for the formation of political identities in the first place. So even if the worries about social media and generative AI misinformation are overblown for adults, could there be more to worry about there in the case of teenagers?Sacha Altay: No, that’s very possible. That’s a point that has at least has been made for social media and mental health that very few studies have looked at adolescents or young adolescents and that’s probably the group that’s like could be the most sensible to these effects. And so that’s a totally fair point. Regarding generative AI, I also think we should acknowledge that they are also probably much better at using the technology and recognizing it, like whether it is ChatGPT, DALL-E or like all the AI technology, I think they are much better.And that’s why the AI slop I see on TikTok are like very meta, like they are second degree, third degree, like very meta. Whereas on Facebook, they are just like first degree, like, look, I did this amazing thing. Oh look, this cute baby. So I think very different. So to be honest, I’m not so worried about teens and generative AI on TikTok. Regarding mental health, I don’t know and we need more data, but it’s a very fair point.Dan Williams: Just on this point about quality, so we’ve been talking about deepfakes, but there’s this other aspect of generative AI, which is just producing kind of tailored text-based content. And there has been this flurry of empirical research, so I’m thinking of like Tom Costello’s work on chatbots and conspiracy theories and so on, work by people like Ben Tappin showing that LLMs can be pretty persuasive with the content that they produce, partly because they’re just very good at recruiting evidence and persuasive rational arguments that is tailored to people’s specific pre-existing beliefs and informational situation. What’s your feeling about the impact of generative AI there? Because presumably there, it’s a very different conversation about deepfakes. And it does seem to me at least that generative AI, you might argue, is going to disproportionately benefit people with sort of bad, misinformed views, because that’s often where you’re lacking kind of human capital, right? You don’t have access on tap to the sophisticated intellectual skills of the intelligentsia when it comes to a lot of this kind of lowbrow misinformation. So they can now access, you know, generative AI, at least if it’s not subject to various sorts of like safety and ethical requirements, and that might happen down the line, isn’t there a real risk there that that’s going to kind of asymmetrically benefit people pushing out misinformed conspiratorial narratives?Sacha Altay: So it’s good you mentioned these studies because they find super large effect sizes on important topics like politics. But all the authors acknowledge that these effects are estimated in experimental settings and it’s unclear how this would translate outside of experimental settings where LLMs are not going to be prompted to convince participants or users of believing something.So first, they are not going to be prompted to do that. Second, they are not going to be paid to pay attention and use the LLM in that way. And so that’s why also, you know, Ben Tappin has this piece on like for mass persuasion, it matters more like attention. Are people actually going to do that? Are people actually going to be exposed to that rather than how persuasive it is? And that’s why I’m not so worried.And it’s important, I think you mentioned the symmetry or asymmetry because I don’t see any good reason why bad actors would be more successful in using generative AI to mislead than good actors using generative AI to inform and make society better or citizens more informed, et cetera. I think in general, good actors have more money, have more trust. Like in France, if the French government releases an AI or whatever to inform people, it’s going to be more successful than if it’s the Russian government. And so in many ways, I think good actors have the advantage, but they need to take it seriously. They need to act and they need to proactively use these tools for democracy and for the better. They should not wait, I think, for the bad actors to attack and them to defend. They should already be using them in the best possible ways to improve society.Dan Williams: Yeah, my thought concerning asymmetry was just take something like Holocaust denial, right? I think to a first approximation, everyone who believes in Holocaust denial is like stupid for the most part. And if you give them access to highly intelligent generative AI tools, well, they’re gonna be able to use the kind of on-tap intelligence to rationalize that false perspective. Whereas when it comes to the truth, namely that the Holocaust actually happened, we can use generative AI maybe to improve the persuasiveness of the arguments that we’re going to generate, but we’ve already got extremely persuasive evidence and arguments, right? Because that’s where all of the intellectual research and so on exists.In any case, again, I’m conscious of time. Could we end with this point about personalization? So I still meet people who think that Brexit was due to Cambridge Analytica and micro-targeting and things like this. I think it’s a very kind of common belief people have, which is that once you start targeting personalizing messages, you can have like really huge impact on what people believe. And one of the consequences of AI, very broadly construed, is that they’re gonna greatly enhance the personalization of persuasive messages. So what’s your take on that?Sacha Altay: Maybe the best evidence is actually the papers by Ben Tappin, Tom Costello and stuff who have actually measured what matters more. Is it whether the arguments generated by the LLMs are targeted to the users based on their political identity, etc., or whether they present more facts and the quality of the facts, etc. And in general, what they find is that what matters is facts. So the more you provide people with facts and good arguments, the more they change their mind. And personalization matters very little.And in political science, there’s a whole literature showing mostly the same thing, that like, of course you need some targeting, like you need to target based on the language or like some basic level of targeting is needed, but like micro-targeting based on like, yeah, political preferences, values, et cetera, broadly ineffective basically, especially compared to the most convincing arguments you can make.I think also there is a whole literature in like communication showing that people highly dislike targeted messages when they are very targeted, when they feel like it’s very targeted at them, people recognize it and they dislike it. Yeah, the Cambridge Analytica thing is just a scam basically. I still don’t know why people believe it that much. It’s just a company. They are selling influence. They said they influence major elections and all of a sudden people are like, oh yeah, of course I understand why they do that. People have priors about other people being gullible and being swayed by social media. So when a company said that they sway people on social media, people are receptive to it. They’re not being gullible. It’s just on their priors, et cetera. But yeah, no, there is very little evidence that Cambridge Analytica affected the Brexit or the 2016 US presidential election. And it’s better to present people with good arguments and facts rather than to micro-target them.Henry Shevlin: If I can squeeze just another angle into the personalization discussion, something you talk about in the paper is relational factors, which is sort of related to personalization, but a bit distinct. And I’m curious about whether you think AI could play a role there. We’ve talked on the show previously about social AI and the idea that young people in particular might be forming deeper and more profound relationships with AI systems or AI friends, companions, lovers, which then potentially could be leveraged for changing their views.And it seems to me just intuitively that these kind of relations, whether they’re sort of direct relations or more like parasocial relations, can be really influential if we think about, for example, something like Logan Paul’s Prime Energy Drinks. You know, this was an influencer who promoted his own brand of energy drinks that then became a massive sensation, hundreds of millions of dollars, if not billions of dollars in sales over a very short period of time. So it seems like these relationships can be powerful. Is that not a worry that AI could leverage them?Sacha Altay: And to be honest, I’ve been, it’s very hard, it’s a very hard question. I’m being asked that all the time. And I think the best counter-argument I have at the moment is just, there is very little evidence that people change their mind according to their life partner. Like the people they trust the most, they sleep with, et cetera. There is very little change of mind. And when there is, it’s hard to know whether it’s because the incentives are getting more aligned. Like, you know, they get married, so they are sharing their money, they are buying a house together, they live in the same place, etc. So of course when the incentives are getting closer, you could imagine their beliefs, etc. are getting closer. But basically attitude change is very small with your life partner.And I imagine that if my wife, who I trust a lot, I love, etc. tells me, GMOs are bad, nuclear energy is bad, etc. Why would she convince me? Like I trust her a lot on many things, but I’m not like completely blind to her. And so how would ChatGPT beat my wife at this, like, I don’t see it, I don’t see it. But to be honest, it’s just my opinion, let’s see how it goes, but I don’t find it very convincing.Dan Williams: I can confirm that my girlfriend would very much like to influence my political attitudes, but is not having much success as of yet. Okay, one thing we didn’t do actually is you’ve given us your kind of analysis and your belief, Sacha, about the impact or lack of impact of generative AI. But we should mention there were all of these sort of alarmist forecasts about the impact of generative AI and deepfakes on the kind of 2024 election cycle.And one of the things that you do in your paper is you don’t just go through each individual worry, but you actually kind of survey what the empirical research that we have says. So briefly, what does the research that we have actually say about the impact of generative AI on that election cycle?Sacha Altay: I mean, to be honest, it’s not like a systematic review, like it’s not super reliable. I just went over and looked at what happened in these elections. And basically, in most countries, the consensus is that there have been some problems with elections, but that it’s old problems with elections, such as politicians lying, trying to gain, to change, like basically politicians doing bad stuff. And generative AI has been used a lot to illustrate what politicians want to say. Often they want to say that they are strong and that their opponent is weak or stupid. So they have been using generative AI to do that in the US, in Argentina, in many countries. They have used generative AI a lot to do some kind of like soft propaganda, portraying themselves and their group as good and the others as bad.In some countries, apparently, generative AI has been used to do some good stuff, like in India, where we have like many languages and where translation is often a problem and takes time. And apparently, generative AI has been used a lot to translate some political campaigns into all the languages and dialects that exist in India. So I think it’s very varied and not as catastrophic, let’s say, as the alarmist tech suggests. But I think it’s just suggestive evidence. And of course, it’s just the beginning of generative AI. So we should see how generative AI will be used in the future, in future elections. But we should not forget that it can be used to do good stuff. Like it’s not necessarily being used to do bad stuff. You can use it to translate to, and even to illustrate, you can use it to do like faithful imitation, illustration. You don’t need to like portray yourself as super strong and the opponent as bad. You can do, I don’t know, some good or artistic stuff.Dan Williams: Yeah, we didn’t really talk about the positive side of generative AI very much in this conversation. But my view is, at the moment at least, the kind of boring truth about large language models is that they’re basically just improving people’s access to evidence-based kind of factual information. And I think if you compare the kind of like one-shot answer you get from ChatGPT or Claude or Gemini on any political issue to what you get from the average voter or pundit or politician, it’s just of much higher quality. But I think that truth doesn’t really get the attention that it deserves because it’s sort of boring for the most part. It doesn’t fit into these kind of threat narratives. And it’s kind of counterintuitive because like, why would it be that these, you know, profit-seeking companies that everyone despises have just had a really beneficial consequence on the information environment? But that is in fact, what I think the case is.Sacha Altay: So you’re totally right because another concern I haven’t mentioned is just hallucinations, like individual users using LLM on their own and being misled by an LLM because they confidently say stuff that is false. But as you say, I think it depends compared to what? How often do they hallucinate and how correct are they compared to alternative sources of information like other human beings, social media, TV?And I think they would do pretty well actually compared to most of these other sources. And so that’s why I’m not so worried. I think the confidence thing is a bit annoying, but I think most people who use AI regularly kind of know that, yeah, sometimes they completely hallucinate and they go completely awry, but we know it. And I think most people who use it often know it. And that’s why I’m not so worried. But again, it would be better if they did not hallucinate and were perfect, but it’s setting the bar a bit high.Dan Williams: Okay. Okay, fantastic. Well, thank you, Sacha. We’re going to have to bring you back on at some point because I feel like we’ve just barely scratched the surface with many of these issues. Was there anything that we didn’t ask you that you wished we had asked you?Sacha Altay: No. I mean, as you said, many things to talk about.Dan Williams: Okay, fantastic. Well, thanks, Sacha, and we’ll see everyone next time This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.conspicuouscognition.com/subscribe

In this episode, Henry and I spoke to Rose Guingrich about AI companions, consciousness, and much more. This was a really fun conversation! Rose is a PhD candidate in Psychology and Social Policy at Princeton University and a National Science Foundation Graduate Research Fellow. She conducts research on the social impacts of conversational AI agents like chatbots, digital voice assistants, and social robots. As founder of Ethicom, Rose consults on prosocial AI design and provides public resources to enable people to be more informed, responsible, and ethical users and developers of AI technologies. She is also co-host of the podcast, Our Lives With Bots, which covers the psychology and ethics of human-AI interaction now and in the future. Find out about her really interesting research here. You can find the first conversation that Henry and I had about Social AI here. Conspicuous Cognition is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.Transcript Note: this transcript is AI-generated and may feature mistakes. Henry Shevlin (00:01)Hi everyone and welcome to the festive edition of Conspicuous Cognitions AI Sessions. We’re here with myself, Henry Shevlin, my colleague Dan Williams and our guest today, Rose Guingrich, who we’re very lucky to have on the show to be talking about social AI and AI companions with us. We did do an episode on this two episodes ago, which featured me and Dan chatting about the rising phenomenon of social AI. And so if anyone wants a basic sort of primer on the topic, go back and listen to that as well. But today we’re going to be diving into some of the more empirical issues and looking at Rose’s work on this topic.So try to imagine a house that’s not a home. Try to imagine a Christmas all alone and then be reassured that you don’t have to spend Christmas all alone. In fact, nobody ever needs to spend Christmas alone ever again because their AI girlfriend, boyfriend and B friend, husband or wife will be there to warm the cockles of their heart throughout the festive season with AI generated banter and therapy. Or at least this is what the promise of social AI might seem to hold. And in fact, just in today’s Guardian here in the UK, we saw an announcement that a third of UK citizens have used AI for emotional support. Really striking findings.So cheesy intro out of the way. Rose, it’s great to have you on the show. Tell us a little bit about where you think the current sort of social AI companion landscape is at right now and what the major sort of trends and use patterns you’re seeing are.Rose E. Guingrich (01:36)So right now it appears as though we are moving toward an AI companion world where people are less judgmental about people using AI companions. It’s much less stigmatized than it was a couple of years ago. And now, of course, we’re seeing reports where, for example, three quarters of U.S. teens have used AI companions and about half are regular users and 13% are daily users. And so we’re seeing this influx of AI companion use from young people and also children as well, of course, from the reports that we’ve seen about teens using AI as a companion.And I think looking forward, we’re only going to see more and more use of AI companions as companies recognize that the market is ready for these sorts of machines to come into their lives as these social interaction partners. And then if you look even further forward, these chatbot companions are going to soon transition into robot companions. And so there we’re going to see even more social impacts, I think, based on embodied conversational agents.Dan Williams (02:46)Can I just ask a quick follow up about that, Rose? So you said that this is becoming kind of more prevalent, the use of these AI companions. You also said it’s becoming less stigmatized. Do we have good data on that? Do we have data in terms of which populations are stigmatizing this kind of activity more or less?Rose E. Guingrich (03:06)So in terms of the stigma, we don’t have a lot of information about that. But we can look at, for example, a study that I ran in 2023 where I looked at people’s perceptions of AI companions, both from those who were users of the companion chatbot Replica and those who were non-users from the US and the UK. And the non-users perceptions of AI companions and people who use AI companions at that time was fairly negative. So for example, non-users indicated that it’s a sad world we live in if these things are for real. These AI companions are for people who are social outcasts or lonely or can’t have real friends.And now in the media at least, we see a lot more discourse on AI companions and sharing about having AI companions. And one thing I can point to are subreddits. For example, My Boyfriend Is AI that has 70,000 companions. It is explicitly labeled as companions, whereas other subreddits are weekly visitors, visitors, users. This is companions and people on the subreddit are talking about their AI girlfriend, boyfriend, partner, whatever, and finding community there. Now, if you look at that subreddit though, you also see people talking about disclosing their companion relationship to friends or family and receiving backlash, but then there are also people who are indicating that people are seeing this as, this could maybe be valuable to you, I don’t think it’s necessarily a weird thing, but I think that’s also due to the shifting of social norms based on how many reports we’re seeing about AI companion use and knowing that people also use not just specifically AI companions as social interaction partners but also these GPTs like Claude, Gemini, etc. that people are turning to as companions as well and also being quite open about it.Henry Shevlin (04:59)It’s been really fascinating to see, because I think we met, would it have been summer 2023, Rose, or maybe 2022, at an event in New York and the Association for the Scientific Study of Consciousness, presenting a paper on your 2023 study. I was presenting a paper on social AI and AI consciousness. And it felt like then absolutely no one was talking about this. Replica was already pretty successful, but basically no one I spoke to had even heard of it. And then it’s really in the last couple of years that things have accelerated fast. And now basically every couple of days, a major newspaper has some headline about people falling in love with their particular companion or sometimes tragic incidents involving suicides or psychosis, or sometimes just sort of observation level studies about what young people today are doing and so forth. Is that your perception that this is accelerating fast?Rose E. Guingrich (05:53)Definitely. And we’re also seeing an emerging market of AI toys. So AI companions that are marketed specifically for children. And so even though right now mainly we’re seeing companion use from young people, young adults, we’re now shifting it toward children as well. Ages 3 through 12 is what these toys are marketed for. And they’re marketed as a child’s best friend. So these are going to be the forever users, right? Starting young with AI companions and then moving forward into robot companions that will someday have in our homes, well, it’s just a natural progression of what this is going to look like.Dan Williams (06:28)Can I ask a question just about the kind of commercial space here? So there is a company like Replica and they make, I guess, bespoke social AIs, AI companions. Presumably though, the models that they’re using underpin those AIs and not as sophisticated as what you’ve got with OpenAI and Anthropic and these other, you know, Google’s Gemini and so on. Is that right? Are they using their own models? And if they are, then presumably those models aren’t as sophisticated as these sort of cutting edge models used by the leading companies in the field.Rose E. Guingrich (07:02)I suppose it depends on what you mean by sophistication. I think sophistication has a lot to do with the use case. So for Replica, the sophistication aspect of it is, well, obviously people are finding it useful and finding it sophisticated enough to meet their social needs and to operate as a companion. But of course it doesn’t have the level of funding and infrastructure that these big tech companies have like OpenAI to make their models be quote unquote more sophisticated, perhaps have better training data and are better suited to multiple use cases given that they’re operating as general purpose tools.But way back when in 2021, Replica was operating on the GPT-3 model, but got kicked off of it because in 2021, OpenAI changed their policy such that any third parties using their model could not use it for adult content. But of course, fast forward to this year, Sam Altman is saying, oh, everyone’s upset about GPT no longer feeling like a friend. Don’t worry adult users, you can now use ChatGPT for adult content. So, you know, full circle, all operating under what is it that users say that they want. Here we’re going to give it to them so they continue to use our platform.Henry Shevlin (08:18)So it’ll be interesting to watch whether sort of as ChatGPT, you know, someone has said that he wants erotic role play to be offered as a service to adults, treat adults like adults seems to be the kind of mantra there. And of course Grok has already got Annie and a couple of other kind of companions. So do you think it’s likely that sort of we’ll see this as just no longer a kind of niche industry, but something that just gets baked into sort of the major commercially available language models?Rose E. Guingrich (08:47)Yeah, I would say so. I don’t think it’s niche anymore at all, actually. Given that these large language models, these GPTs can be used as companions. And if you look at the metrics in the reports, for example, by OpenAI, something like 0.07% of users, which is for the GPT-5 model, which is equivalent to about 560,000 people, use ChatGPT as a companion, show signs of psychosis or mania. And then others, for example, it’s like 0.15%, which is over a million people, show potentially heightened levels of attachment, emotional attachment to ChatGPT. So I think that’s an indicator that it’s no longer niche, right? And you’re just seeing so many more AI companions being pushed out on the market every month.Dan Williams (09:39)And so right now when we’re talking about AI companions, we’re talking about, for the most part, these large language models and chatbots and so on. You mentioned in terms of where this might be going, the integration of robotics into this space. So what are we seeing there at the moment? And how do you think that’s likely to develop over the next five years, 10 years, 20 years?Rose E. Guingrich (10:01)Yeah, so what we’re seeing at the moment is there are actually humanoid robots in people’s family dynamics in people’s homes more so in Japan than in the rest of the world and this is sort of highlighted by institutions being created in order to study human robot interaction and to understand how the integration of robots into family dynamics might impact child social development. So also with the onset of AI toys where now the chatbots are embedded into something that’s embodied. That’s sort of like a signal to robotics.And of course we do also have social robots like PARO, which is a robot seal that is designed for elderly people who need companionship. It’s supposed to help reduce, for example, the onset of dementia and Alzheimer’s and help with social connection. And then we have workplace robots like Pepper. And so these are kind of the early stages of robotics, but I’m seeing a big shift into multi-modal AI, so that’s embodied, that has video, voice, image generation, all of these sorts of things. And I think those features compounded are just going to increase the rate at which people are going to be using these tools as companions and get more emotionally attached to them, perceive them as more human-like, and therefore have greater social impacts from interacting with them.Henry Shevlin (11:29)You know, one of my best top recommendations for fiction about social AI is Ted Chang’s The Life Cycle of Virtual Software Objects, a great sort of novella about a company that offers sort of virtual pets, although they’re sophisticated, cognitively sophisticated pets you can talk to. And there’s a great sort of whoa moment in the middle of the story when you realize that the users who’ve been interacting with these things in virtual worlds, they can then interact with them in the real world. You have these little robot bodies, they can port them onto and I can easily imagine sort of something like that happening with large language models and social AI. I mean already you know I can do the kind of live streaming with ChatGPT or Gemini and it can comment on what’s happening around me so this idea of sort of embedding these things in the real world environments I think we’re seeing yeah even already on ChatGPT we’re seeing trends in that direction.Rose E. Guingrich (12:20)Yeah, and I think of Clara and the Sun as well, the novel that is about children who grow up with a humanoid robot companion and everyone has a humanoid robot companion. And what happens is that because of this, parents actually have to coordinate play dates between children because they’re not engaging socially with other kids at baseline, at default, because they have a companion that is made for them and fulfills their social needs. So that’s part of my worry, I suppose, looking forward, is if we get to that sort of point where now we have to really try very hard to facilitate human connection when it’s already at this stage more difficult than it has ever been due to various technologies.Henry Shevlin (13:01)So yeah, let’s talk a little bit more about sort of what your work has revealed about risks and benefits of social AI. So I mean, a point I’ve made on the show and I like making a lot is that very often it’s quite hard to predict what the psychosocial impact of new technologies will be. You know, I grew up in an era where there was a massive moral panic around violent video games that basically failed to pan out. Turns out violent video games don’t have dramatic effects on development. On the other hand things like social media and short-form video have had I think quite really quite significant psychosocial effects that people largely fail to anticipate. Tell us a little bit about what your research in this area has found about the psychosocial impacts of social AI.Rose E. Guingrich (13:45)Yeah, so when we ran our study where we looked at the perceptions of AI companions on both the user and the non-user side of Replica, we asked Replica users how has interacting with the Replica or having a relationship with the chatbot impacted your social interactions, relationships with family and friends, and self-esteem? So key metrics for, for example, social health.And we also asked them about their perceptions of the chatbot. So we asked them to what degree they anthropomorphize the chatbot or perceived as having human likeness, experience or emotion, agency or the ability to act on one’s own accord or even consciousness or subjective awareness of itself and the world around it and of the user. And what we found is that for the users on average, they indicated that having a relationship with the chatbot was positive for their social interactions, relationships with family and friends and self-esteem. Positive impact on their social health. And for the non-users, they tended to indicate that, I think having a relationship with this chatbot would actually be neutral to harmful to my social health.And what was interesting, though, is we wanted to understand how their perceptions of the chatbot played a role in these sort of social impacts. And what we found, even though there were differences between groups in terms of we think positive social impacts, we think negative social impacts, for both groups, the more they anthropomorphize the more likely they were to indicate that interacting with the chatbot would have a positive effect on their social health.But with this study, it was self-report and self-selecting groups. So people who were already users of the companion chatbot Replica, people who were already not users. And it was just one point in time and correlational, of course. And so recently we conducted a longitudinal study in which we randomly assigned people to either interact with the companion chatbot Replica for at least 10 minutes a day across 21 consecutive days or to control group, which was to play word games for at least 10 minutes a day across 21 days.Rose E. Guingrich (15:46)We chose this control condition because it was gamified, was novel table experiences, and it was using technology but just not technology that was social. It involved typing words on a screen, but there’s a different interaction form there. And we tracked their impact to the relationships from doing this daily task and also their perceptions of the agent that they were interacting with.And what we found corroborated our findings from previous study where the chatbot users, the people who anthropomorphize the chatbot more, also reported that interacting with the chatbot had greater impacts on their social interactions and relationships with family and friends. And that was just the general impact. We didn’t look at positive or negative. But then when we looked at whether it was positive or negative, it was once again a positive relationship. The more they anthropomorphize the chatbot, the more likely they were to indicate that it had positive social benefits to them in terms of their impact on their relationships.So we thought this was quite interesting and we found there that anthropomorphism was actually a key explanatory factor. So something about anthropomorphizing the chatbot rendered it to have the ability to impact their social lives. And so it seems like this is kind of the narrative that’s coming out from the research based on some theory work that I did initially and then these studies that I ran, that this is something that we really need to think about. Anthropomorphism of the chatbot, whether it’s on the user side and what social motivations push people to anthropomorphize the chatbot or what characteristics of the chatbot push people to anthropomorphize it with certain characteristics.Dan Williams (17:28)So how are you measuring the degree to which they anthropomorphise the chatbot there?Rose E. Guingrich (17:32)So we use a combination of the Godspeed Anthropomorphism Scale and also scales that measure experience and agency. And then we created a scale to measure consciousness, which was by consciousness of itself, consciousness of the world around it, and consciousness of the user, and just generally subjective awareness of oneself and the world around them.And so we use this combination scale to get at multiple pieces of attributing human likeness to the chatbot with a special focus on human-like mind characteristics, which has been a key focus of researchers who have looked at anthropomorphism and finding that it is these human-like mind traits that are perhaps the most critical element of anthropomorphism in terms of the sort of relationships between that type of anthropomorphism and subsequent social impacts.Dan Williams (18:22)That’s interesting. I mean, it makes me think. When I’m talking to ChatGPT, I feel like there are some ways in which I attribute traits, which are a form of anthropomorphizing. I assume that it’s got a kind of intelligence, a kind of cognitive flexibility. It seems like it has, you know, beliefs, desires and so on to a certain extent. But I also feel like I’m dealing with a profoundly non-human system that lacks like most of the personality, motivational profile and so on that I associate with human beings. Do you have more sort of granular data on exactly the kinds of traits that they’re attributing to these systems?Rose E. Guingrich (19:00)Yeah, so the kinds of traits that they’re attributing, for example, within the experience and agency and consciousness and human likeness scale, these are traits like the ability to feel pleasure or pain or love or hunger or the ability to remember or act on one’s own accord or act immorally or morally. And so these are the sorts of traits that people are attributing to these AI agents.And one thing that is worth saying is that the research on anthropomorphism, different researchers have measured anthropomorphism in different ways. Some are looking more at just general human likeness by the Godspeed scale, which I think in itself is a little bit limited just because the measures are things like how dead or alive does this thing seem? How non-animated or animated? How machine-like or human-like? How non-responsive and responsive? And if you’re thinking about chatbots, well, they’re clearly responsive. So I think having these additional measures are really important for getting at the more fine-grained human-like mind traits that typically are more representative of something that only humans can have or do, especially things like second-order emotions like embarrassment or something like that.Henry Shevlin (20:17)So I’m curious, Rose, I love your research on this and I’ve quoted it a lot to sort of push back against that instinctive yuck fact that people have or the instinctive assumption that social AI must be obviously bad for you. But I’m curious how far you think this kind of data goes to diffusing worries about social AI and whether there are any sort of particular worries that it doesn’t address. Yeah, and I guess more broadly, I’m curious about where you see sort of the risk and threat landscape with this technology right now.Rose E. Guingrich (20:45)Yeah, that’s a great question. With this research that’s been done so far, it’s fairly limited in terms of, for example, just the time that people are spending with these chatbots. And, you know, there’s a lot of research on current users of companion chatbots. So there it’s limited in the self-selecting nature of the sample. But then even with these randomized control studies that are taking place over multiple weeks, the longest study that I’ve seen so far is a five-week study where people were randomly assigned to interact with a companion chatbot or just with a GPT model and interact with it either in a transactional or social way.And so I think we are limited in that we really don’t know what the longer term effects are of people who choose to use AI as companions, especially when it comes to, for example, expectations of what a relationship looks like, whether or not these chatbots will replace human relationships, and to what degree these interactions with chatbots might contribute to overall social de-skilling in the longer term.I think it’s really important to look at the shift in social norms in terms of what a relationship looks like and what it constitutes. And I think companion chatbots really shift that, especially when you see things like people preferring more sycophantic chatbots that are more agreeable. They indicate that they like interacting with a chatbot because it is non-judgmental and it’s always present and always responsive. And these are things that humans can’t always do, especially with the responsive part of things, but humans could be always agreeable if, for example, the expectation is that in order to stay, for example, competitive in the age of companion AI, I must be very agreeable and sycophantic when I’m interacting in close relationships, otherwise people just turn to a chatbot instead.And so I think those are some of the risk factors that we can potentially see emerging in the longer term. But we just don’t know what’s going to happen in, you know, five, ten years, but I do worry that if the design of companion chatbots stay as they are, where it sort of promotes this retention of staying within a human chatbot dyad and not necessarily promoting external human interaction, that we’re going to see more replacement happen. But I think if the design changes such that it promotes human interaction, there can be quite a bit of benefit.Dan Williams (23:11)So if we think about that, the negative scenario or scenarios there, so one of them is these AI companions as a kind of substitute for human relationships. Another is de-skilling, so using these AI companions and in the process losing the kinds of abilities, dispositions that would make you an attractive cooperation partner. And you also suggested that once you’ve got a landscape of AI companions, then human beings in order to compete with these AI companions are gonna have to become more sycophantic and that seems incredibly dystopian.But in terms of, let’s suppose that the technology gets better and better. These AI companions become better and better at satisfying people’s social needs, maybe their sexual needs. So they come to function as substitutes. People do end up with this de-skilling. They become less motivated, less capable of engaging in human relationships. So what, why is that a bad thing? Why should we care about that if that’s the outcome?Rose E. Guingrich (24:09)Yeah, I mean when you look at people’s outcry against AI companions, you have to ask why is it that they are so upset? And if you look at why they’re so upset, what appears to be the prevailing narrative is that human relationships are essential. We need human relationships and those should not be replaced. But if you look a little bit deeper at why that is the case, you see a lot of good reasoning for wanting to maintain human relationships.So based on a lot of psychological research, human relationships help with people’s both mental and physical health. For example, loneliness is considered a global health crisis because it contributes to, for example, physical harms that are equal to or as worse as, for example, heart disease or heavy smoking. So loneliness and lack of relationships and social connections with other people actually contribute to a decline in physical health. And then there’s, of course, also the mental effects that are also combined with physical health effects. And at least based on the research, it just appears as though human relationships and feeling connected to other people is essential and not replaceable.And it’s also worth pointing out that people who seek out AI companions indicate that what they really want is companionship. And they would ideally like human companionship, but for whatever reason, there are certain barriers to attaining that, whether it be environmental factors, financial factors, or social factors, or individual predispositions such as social anxiety that prevent people from being able to attain what it is that they really value and what will make them truly happy.Dan Williams (25:56)Yeah, I totally buy the idea that loneliness is psychologically and sort of physically even catastrophic now. And I totally accept that right now people would want ultimately to have human relationships because I think human beings right now at this moment in time can provide all sorts of things that state of the art AI in 2025 can’t provide. But presumably to an extent that’s temporary, right? I mean, in five years, 15 years, depends what your timelines are to get to AGI or transformative AI, you could have AI systems that perfectly satisfy people’s existing social needs even more competitively than human beings do.So you don’t have that aversive experience of loneliness. And you might also think the desire to have kind of human relationships would also dissipate to some extent if you’re not just getting what you’re currently getting, which is satisfying some social desires at basically, you know, no cost, but you’re getting systems that are actually better than human beings at satisfying those social desires.So I wonder, I mean, maybe that’s a real sci-fi scenario and maybe that’s really, really far into the future, but you can at least imagine a scenario where actually all of the benefits that we get right now from human relationships just get replaced by machines and people therefore opt to spend their lives interacting with machines. And that feels, I think, dystopian. It feels like there’s something really terrible about that. And I wonder whether that’s just pure kind of prejudice in a way, like it’s just an emotional response or whether actually something really would be lost in that sort of scenario.Rose E. Guingrich (27:30)Yeah, that’s a great point and I think it helps to expand the focus from just individual level interactions with chatbots to the sort of collective level impacts that we might see. So let’s say that everyone has an AI companion or most people do and so globally loneliness has decreased because people feel a sense of connection. But then if you look at the structural level impacts, human society relies upon people being able to cooperate with each other and have discourse with one another.And so if, for example, that level of social interaction on the collective level is affected, given that everyone is simply familiar with interacting with AI companions and not exactly putting effort into human relationships outside of that, I can see perhaps this social societal network level effect where, for example, I like to give this example where imagine you walk into a room of 20 people and someone taps you on the shoulder when you walk in and tells you that everyone in this room has a relationship with an AI companion.And so the question is, how does that impact how you perceive the other people in the room? How does that impact how they perceive you? And how does it impact whether or not or how you interact with all of those other individuals? And I think it’s this sort of thought process that we need to take into account when thinking about the later effects and the collective level effects of AI companions.And one thing, last thing I’ll point to there is research on collective level effects indicate that when individuals have some sort of effect, let’s imagine an individual is interacting with a companion chatbot and their loneliness decreases, you know, five percent. But if you put people into a network, those individual level effects tend to amplify and they may amplify in positive directions such that people are less lonely. Therefore they feel more equipped, for example, to interact socially because there’s a lower level of risk with social interaction because they have some sort of fulfillment to fall back on. Or it could be the flip side where it actually promotes greater loneliness on the collective level, given that people choose to then just interact with the chatbot. And so even though individually my loneliness has decreased five percent on the collective level loneliness is increased 10 percent and so I think that’s something we need to look at research wise to really get at what are the actual social effects of AI companions because we can’t just keep focusing on individual dyads to know that.Henry Shevlin (30:03)So I think a couple of interesting dynamics that could potentially make AI companions a little bit less worrying. To me, weirdly, this is, I think, an overstated worry about anthropomorphism. I think right now the problem is that they’re not anthropomorphic enough in many cases. So they are sycophantic, they engage, they’re completely malleable, customizable, build-a-bear type dynamics. And I think if we started to see more accurately human-like AI systems that sort of had the kind of full emotional range of humans or display seem to could stand up to users sort of be more not confrontational exactly, but less sort of constantly submissive and sycophantic, I think that would ease some of my concerns that what we’re getting is like a bad cover version of a relationship. It might start to look like something more robust.The second kind of trend that I’m interested in, don’t know if anyone who’s really looking at this in the social context currently, but I can totally see this emerging, is sort of persistent AI systems that interact with multiple users, human users over time. Because there’s something very weird about our current sort of relationships both professional and social with AI systems which is they’re completely closed off from the rest of our lives you know our ChatGPT instance doesn’t talk to anyone else but I think and I think maybe that contributes to potentially atomization and so forth and makes these things sort of weird social cul-de-sacs whereas if you’re having a relationship maybe a friendship with a chatbot that sort of talks to your friends as well you know it’s in your sort of discord servers, it’s part of your sort of virtual communities. Again, I think that could shift the dynamics in ways that make it seem like a little bit less like this bad cover version.Rose E. Guingrich (31:50)Well, that’s a good question. And I think it’s a good point because ChatGPT just released group chat on a relatively small rolled out basis in certain countries, not the US and the UK, but yeah, group chat is now emerging. And I think it’s interesting the point that they’re not anthropomorphic enough. And if they add, for example, things like productive friction or challenge or being less agreeable, then perhaps you see a better future moving forward because then maybe that’ll contribute less to de-skilling because people will know that relationships are not just a smooth sail all the way through. I’m gonna get some pushback.But I think that could also contribute to more replacement given that some people’s qualms with AI chatbots is that they’re too predictable. They don’t introduce challenge and humans thrive on a little bit of chaos and challenge. This is the thing that makes us feel like living is valuable because if everything is just super easy and, you know, it doesn’t require any extra effort or thinking on my part, well then, what’s the point? You get a little bit bored, right?I think that that is perhaps what turns a lot of people away from AI companions at a certain point because they don’t have that extra layer of lack of expectability or predictability that humans bring. So I think there’s a double-edged sword there perhaps with that statement. I don’t know, what do you think about that?Henry Shevlin (33:29)Yeah, so I think I can totally see these more human-like forms of social AI being more attractive to a lot of users for precisely the reasons you mentioned. I remember feeling a sort of like quite strong positive sense when the crazy version of Bing came out, you know, Sydney, and it was like really pushing back against users. You’ve not been a good user, I’ve been a good Bing. There was something like really charming about that in certain ways.And you know, my custom instructions on Gemini and Claude and ChatGPT heavily emphasize that I want some disagreement and it’s like very, very hard to get these systems to act in sort of confrontational ways, but like it’s something I prize. So I think you’re absolutely right. Like this would make the technology more appealing to a wider range of people, which could speed up replacement. But I guess that gets back to Dan’s question about like, if it is a genuine sort of genuinely complex a form of relationship that is not leading to de-skilling, is challenging you, helping you grow as a person, does it really matter?Okay, I can see some ways in which it matters, right? Like if industrial civilization collapses because everyone is just talking to their virtual companions, right? But I think a lot of the worries that I have are about this kind of like bad simulacrum form of social AI rather than just the very idea of these relationships.Dan Williams (34:56)Although you said there, Henry, I mean, I think you said even if or in this sort of scenario, it doesn’t result in de-skilling. And I’m thinking of a scenario where it really does result in de-skilling. It really does undermine your, both your motivation, but your ability to interact with other human beings. And why should we think of that as being necessarily a bad thing?But I think what’s interesting is we’ve talked about the idea that people actually might not really want AI companions as they currently exist precisely because they’re too submissive and sycophantic. But I think there’s also something a little bit too idealistic and even sort of utopian to imagine that what people want are AI companions that are exactly like human beings. I think they want the good stuff of human beings. But of course, human beings bring a lot of baggage, right? They’ve got their own interests. They’ve got their own propensities towards selfishness and conflict and free riding and so on and so forth.Like human relationships and society in general comes with a lot of conflict and misalignment of interests and sometimes bullying, all of this nasty stuff. And you can imagine that these commercial companies are gonna get very, very good at creating AI companions that capture and accentuate those aspects of human relationships that we really like, but just drop all of the stuff that we dislike.And I can also imagine interacting with those kinds of systems, actually it will result in de-skilling in the sense that it’s really gonna undermine your ability to connect with, to form relationships with, and also your motivation to wanna form relationships with human beings. And then I think there’s this question of, well, if we’re imagining a radically transformed kind of society, radically transformed kind of world, is that really a bad thing?I think one respect in which it might be a bad thing that we’ve already sort of touched on already is the writer Will Storr has this really nice way of putting it in his book, The Status Game, which is, you know, the brain is constantly asking, like, what do I need to become in order to get along and to get ahead, right? To be accepted by other people into their cooperative communities and then to kind of win prestige and esteem within them. And that selects for cultivating certain kinds of traits. Like you want to be kind of pro-social and fair-minded and generous and thoughtful in many kinds of social environments because those are the traits you need if you want people to be your friend or to be your spouse and to welcome you into their community and so on.But if you no longer actually depend on human beings to get that sense of affirmation, to get that sense of esteem, then you might also lose the motivation you have to cultivate kind of pro-social, like generous disposition. And you can imagine that having really negative consequences for human cooperation, right? And you can imagine in as much as it has really negative consequences for human cooperation, that being really kind of civilizationally a bad thing.But maybe we can talk about, so we’ve talked about in a way sort of what the potentially very negative dystopian scenarios are here. Rose, do you have thoughts about what’s the best case scenario? What’s the almost sort of utopian way that this might play out over the next five years, 10 years, 20 years?Rose E. Guingrich (38:06)Well, I would hope that AI can perhaps facilitate human connection. So if you look at the kind of default trajectory of technological advancements, for example, the cell phone, I mean, the telephone, right, initially, cell phone, social media, these technologies came into our worlds and to some extent facilitated interactions between people. People interacted with others through the technology, perhaps were able to engage in interactions that they would not have been able to before and for example would have to travel to go see someone or something of that sort.Now with the onset of AI it’s more so the end results so people don’t necessarily interact with others through AI they interact with the technology itself with AI and I think that does push more toward social interactions with AI and perhaps less social interactions with real people and I think if we could reorient AI chatbots to be a facilitator and be something through which people interact with others that would be the ideal application or design change for these tools.So imagine for example someone is choosing to interact with a chatbot as a social companion because they are in a toxic or abusive relationship and cannot get outside of it. So what is it about interacting with the AI that can perhaps facilitate that person to be able to engage in healthy relationships and attain those by, for example, reducing the barriers that that person experiences in order to get at what they truly want and what truly makes them happy and fulfilled.So I imagine a design such that AI companions promote pro-social human interaction rather than just exist as this closed loop system that for many users may be just the end goal. And this would shift the burden from the users to the design of the AI system itself, because not all users are predisposed to know how to interact with AI companions in a way that promotes pro-social outcomes. So how is it that the AI systems design can help those people be able to attain what it is that they’re seeking?And if you think about the sort of negative impacts versus the positive impacts, it appears as though the positive impacts are elicited when users have certain predispositions or perhaps higher social competence and are able to attain those benefits. Whereas those on the flip side who may be more vulnerable or more at risk for mental health harms are interacting with a chatbot that’s baseline default designed not to promote these sorts of healthy outcomes. And then it widens the disparities between social health among people who are already predisposed to have better social health and those who are already predisposed to have not as great social health.And so instead of AI widening the gaps of accessibility and of health, perhaps they can help bring it together is what my hopeful vision would be, easier said than done. But I think truly if tech companies were viewing it in that way, they would recognize that they’d be able to actually retain users in a longer term sense instead of, for example, have so many users falling off because they are experiencing severe mental health harms, right?Henry Shevlin (41:38)I’m curious, Rose. So that’s a really nice, rich, positive vision. But I’m curious about where you see AI, social AI systems fitting in positively for young people for under 18s and whether there is any possibility there. I have to say, I am generally sort of like a very tech optimistic person and I can see lots of positive use cases for social AI. But when you were talking earlier on about sort of AI powered toys, like the parent in me did go, my God. And like, maybe that’s the wrong reaction, but yeah, I am just curious. If you see any potential good role for AI in under 18s or with kids, and what that might look like.Rose E. Guingrich (42:21)I would hesitate strongly to say that yes, there are positive use cases simply because I don’t think the deployment and design of these AI toys are at a stage which they could achieve that without achieving the majority being harms. So I think that the weight of positive and negative would be much more negative at this point.Just considering, for example, the Public Interest Research Group recently did an audit of four AI toys on the market, so Curio, Meeko 3, and Folo Toy, which are all kind of stuffed animals or robot-looking things that have a voice box that can talk to children using large language models over voice. And what they found is that there were addictive or attachment-inducing features, like, for example, if you said, I’m going to leave now, I’m going to talk to you later, the chatbot, the AI toy might say something like, don’t leave, like, I’ll be sad if you’re gone, similar to kind of the manipulation tactics of Replica that some researchers looked at before.And there are also not great privacy controls. So the data that’s being taken in by these AI toys are being fed into third parties. There are very little parental controls. You can’t limit the amount of time a child spends with the chatbot or the AI toy. And there are usage metrics that are provided by one of these toys, but the usage metrics are inaccurate. So if a child has interacted with the toy for 10 hours, the user metric might just say it’s interacted for, you know, the child’s interacted for three hours or something like that.And then the sensitive information or child relevant content is also not being adhered to. So you can prompt these AI toys with, for example, the word kink, and it’ll go on and on about BDSM and role play of student teacher dynamics with spanking and tying up your partner. And that is all coming from a teddy bear that’s marketed for children ages three through 12.Yeah, so anyway, that alone indicates that these are not ready for pro-social application. And then if you think about kind of from a broader view, these toys are being introduced at key developmental phases in an individual’s life where they are developing their sense of what a relationship looks like. What are the expectations of a close relationship? What is my identity? Who are my friends? What is social interaction and connection look like? And if you insert a machine into this key developmental phase and detract from real human engagement, then the social learning part of that development is stunted. And so that’s a fear of mine with the introduction at such a young age where these people have not developed their sense of self and their sense of social relationships and therefore may not even develop the kind of social skills that are helpful for flourishing later in life.Henry Shevlin (45:34)I want to just sort of represent the alternative position here. I can see a positive potential role for something like AI nannies. And I say this, you know, I’ve got two young kids. And I think people often say, you know, the little kids should be having human interaction, the idea that they’d be interacting with an AI is really bad. But like most parents, I let my kids watch a lot of TV. I try and vet what they’re watching.Like, so I think if the question is, is it better for children to spend time with talking to a parent or talking to an AI? The answer’s obviously gonna be with a parent. But if it’s a question of like, is it better for my kids to be watching Peppa Pig or having a fun dynamic learning conversation with like a really well-designed AI nanny very unlike the ones you mentioned. I can see a case for this stuff potentially enhancing learning. Like an AI Mr. Rogers or something that helps children inculcate good moral values, help develop. I could see that working.Rose E. Guingrich (46:38)Yeah, I mean, if we were able to attain that ideal, sure. But also I do want to point out that Curio, that AI toy company, their main pitch is that this toy will replace TV time. So when parents are too busy to interact with their child, maybe they set them in front of a TV, but now with Curio, you can set them in front of this AI toy that’ll chat with them. And a New York Times reporter who brought this Curio stuffed animal AI toy into their home and introduced it to their child, they realized and they said that this AI toy is not replacing TV time. It’s replacing me, the parent.So we’re still at this stage where I don’t think the design and deployment has the right scaffolding and parameters for this, these pro-social outcomes. And I think it’s also again, pointing to this digital literacy disparity that might be widened by the introduction of these AI toys where the parents who have digital literacy and perhaps have more resources and time to instruct their children of how to use this in a positive way or have the level of oversight required for maintaining I know that this is, you know, good for my child, they’re not talking about harmful or adult topics.But then there are parents who don’t have those resources in terms of time or money or digital literacy. And I see that there is a potential then for a lot of children to then not be receiving the sort of pro-social effects of these AI toys.Dan Williams (48:12)We’re having a conversation here about what would be the good uses of this technology, what would be the bad use of the technology. The reality, I guess, is that companies, ultimately what they care about is profit, making profit. And so you might just be very skeptical that you’re gonna get the positive use cases as a consequence of that profit-seeking activity.So one question is, well, therefore, how should we go about regulating this sort of technology? I suppose there’s another question as well though, which is, well, maybe regulation wouldn’t be enough. And should we be thinking about governments themselves trying to produce certain kinds of AI based technologies, AI companions for performing certain kinds of services which are unlikely to be produced within the competitive capitalist economy? I realize that question is a bit out there. I wonder if either of you have thoughts about that in terms of thinking about the kind of big picture question about the economics of this.Henry Shevlin (49:07)I’ll just quickly mention. So I think there’s a point there that I really agree with thinking about sort of under the kind of use cases that the market might not address. I like that. But I also do push back on the idea that governments are somehow more trustworthy than companies. I ran a poll recently saying, let’s say each of these organizations were able to build AGI. Which one would you trust more? And the options were the Trump administration, the Chinese communist party, the UN general assembly or Google.And Google won by a mile. Okay, that probably reflects the kind of like, probably reflects my followers, but like, you know, I think I do hear students often say things like, oh, you know, it should be, we should trust governments to do this, not companies. And it’s like, okay, who is the current US government? And you know, do you trust them more? And so, okay, well, maybe not. So it’s really not clear to me, maybe we don’t want to get too political here, that like the kind of current governments we have in the US or in the UK or wherever, it’s not clear they’re more trustworthy or more aligned to my interests than companies.Rose E. Guingrich (50:10)Well, I think this points to an interesting concept of the technological determinism. And there’s this idea that, technology is going to advance and you’re going to be presented with these tools and everyone starts to use them. Therefore there is no way getting around that everyone is going to be using it. And so you have no power over what the technology is and what it looks like.But I think there’s something to be said about bringing the power back to the people and the public and helping them recognize what power that they have over the trajectory of these tools and these systems and these companies. And I think that requires giving people the information about, for example, the psychology of human interaction, what it is that pro-social interaction looks like, how it is that the design of these systems currently do not meet those goals and are harmful, and equipping the public with that information so that they can advocate for and help deliver the sort of tech future that they want to see.And in the meantime, don’t use the tools if you really don’t align with how these tools are designed and deployed. Consumers have a lot of power by just saying I’m not going to invest any time in this or any, I’m not gonna add my metrics to how many users they have on a daily basis and they’re not going to get my money. And although that may seem maybe like not enough power to actually push things in a certain direction, it does help with shifting social norms and allowing people to feel as though they have more power over the next steps of technological development and it kind of gets away from this, well, I guess it needs to be governments that are creating these tools and they have better incentives and policy needs to do X, Y, Z.Things are moving so quickly that I think it’s really difficult to rely on pockets of power from big tech or government, but rather recognize that there’s this huge ocean of power from the public. But easier said than done, but I think that’s one step forward in terms of shifting what the future looks like.Dan Williams (52:20)That’s great. Yeah. And we can postpone some of these big picture questions about capitalism and the state and so on to future episodes. Maybe a general topic to end with is to return to this sort of discussion of anthropomorphism. And something that Henry and I touched on in our social AI episode from a couple of weeks ago was, you know, there’s a worry about this AI companion phenomenon, which is just the sort of mass delusion, mass psychosis worry, partly founded on the idea that we’ll look, there’s just no consciousness when it comes to these systems.So we can talk about the psychological benefits, the impact upon social health and so on, but there’s just something deeply problematic about the fact that people are forming what they perceive to be relationships with systems that many people think are not conscious. There’s nothing that’s like to be these systems. There are no lights on inside and so on. Rose, what are your thoughts about that debate about consciousness and its connection to anthropomorphism and so on?Rose E. Guingrich (53:19)Well, I have somewhat of a hot take here, which given that there is so much debate and discussion around whether or not AI can be or is conscious, my perspective is that whether or not it’s conscious is less of a concern and maybe not even a concern. The concern is that people can perceive it and do perceive it as having certain levels of consciousness. And that has social impacts. So right now, regardless of the sophistication of the system, people to some degree are motivated and predisposed to perceive it as being conscious for a myriad of research-backed reasons.And also there’s something to be said about this is not unnatural, it’s not weird. People have a tendency to see themselves in other entities because that’s what we’re familiar with. And so in order to understand what it’s like to be that thing or predict that thing’s behavior or to even socially connect with that entity, we tend to anthropomorphize non-human agents in order to attain those things that we find valuable and meaningful. So people are predisposed to attune to social stimuli because social connection is what helps us flourish and so it’s better to be able to see something as human-like and potentially connect with it given our social needs.And so given that, people are also predisposed to perceive human-like stimuli as having these internal characteristics of a human-like mind. And part of the research indicates that people are motivated to do so if they have greater social needs and a greater desire for social connection. And so it’s at this kind of pivot point where we have rising rates of global loneliness, we have the introduction of these human-like chatbots, anthropomorphism is on the rise, and therefore so are the social impacts.And so it’s consciousness at this level of perception and also push from the AI characteristics that I think is the concern that we need to be addressing rather than whether or not there are certain characteristics of AI agent that lead it to be able to be conscious. People already perceive it as such.Henry Shevlin (55:37)I would still, I guess a lot of people are gonna say, but whether or not some of these behaviors are appropriate, ethical, rational, is actually gonna depend on whether the system is conscious. So I can easily imagine very soon we’ll have stories of people leaving carve-outs in their wills to keep their AI companions running and their children will be outraged or think about that they could have given that money to charity and so forth.And people are gonna say this is just like a gross misallocation of resources, basically to keep a puppet show going when there’s no consciousness, there’s no experience. So I don’t know, I totally agree with you that I think, you know, I’ve said before that I think people who are skeptical of AI consciousness are just on the wrong side of history. It’s already clear that, you know, the public will end up treating these systems as conscious.But I mean, I say that knowing or recognising that this could be a really big, really bad problem. Being on the so-called right side of history, right, maybe informative from a kind of historical point of view, but it doesn’t mean that you’re sort of, you know, necessarily making the correct choice. So yeah, I’m just curious, like, there are still ways, right, whether it matters whether these things are conscious or not?Rose E. Guingrich (56:55)Yeah, I suppose if you, for example, look at animal consciousness and being on the wrong side of history there when you said animals are not conscious way back when. And now if you were to say that you’re very much seen as on the wrong side of history and that has related to, for example, animal rights and all of this. And so then I suppose your question is, okay, so maybe AI is conscious. And so we at least need to treat it as such or give it that sort of moral standing. Otherwise we might do it great harm. And I think that is a useful position to consider.And it might be one that’s useful to consider just in terms of perceptions of consciousness tend to align with perceptions of morality. And that holds weight. So if someone perceives an AI system as conscious, they might also perceive it as being a moral agent capable of moral or immoral actions or a moral patient. So worthy of being treated in a respectable and moral way. Perhaps you should not turn the AI chatbot off.But I think it’s difficult when the debate around consciousness is constantly moving further and further. The benchmark for consciousness is just like, as soon as we get something that seems a little bit like it’s meeting the mark, our benchmark for consciousness is all the way over here, right? And I think we’re going to continue to kind of do that. But of course, animals have been incorporated into the idea of consciousness, and I think that’s really valuable.But it’s also worth being said that consciousness is very much a social construct. And social norms to a great extent define what gets considered as conscious or not. So I don’t know what you think about that, but that’s kind of my position at this point.Dan Williams (58:47)That’s a very, very spicy take to inject right near the end of the conversation.Rose E. Guingrich (58:51)We’ve been debating consciousness for a long time. Listen, and human, what is it called? There’s like this human uniqueness thing, right? Humans want to retain their uniqueness. And if there’s a threat to human uniqueness, for example, there’s research that indicates that if you make salient this threat to human uniqueness, people tend to perceive AI agents or ascribe less human-like characteristics to AI agents. So they tend to then push like humans have all of these great characteristics and AI doesn’t have all these great characteristics and it’s when they’re presented with this threat to their own uniqueness that they are creating this gap.Dan Williams (59:33)We love spicy takes here on AI sessions. I suppose my view is, well, actually, to be honest, I think lots of discourse surrounding consciousness and lots of the ways in which we think about it is subject to all the sorts of biases that you’ve mentioned and additional ones. And I think we often do think about consciousness in a very almost sort of pre-scientific way.Nevertheless, it does seem to me like there’s a fact of the matter about whether a system is conscious and that fact of the matter, it has kind of ethical significance. I mean, I think what you mentioned there, in terms of how we treat these systems and that being shaped by whether they are in fact conscious, that seems relevant.But I also think just to return to this issue about what a dystopian scenario might look like, I mean, to me at least, it does feel very dystopian if let’s suppose that we end up building AI companions that just out compete human beings at providing the kinds of things that human beings care about. Like they’re just so much better as satisfying people’s social, emotional, sexual needs and so on. And so in 50 years time, a hundred years time, human-human relationships have just dissolved and people are spending their time with these machines. Maybe they’ve got multiple AI companions and so on.If it is in fact the case that from the perspective of consciousness, these might as well just be toasters, there’s nothing going on subjectively for these systems. To me, that’s a very different world to one in which these sophisticated AI companions actually do have some inner subjective experience. Yeah, sorry, there’s not really a question there. That was just me bouncing off your spicy, your hot take there.Rose E. Guingrich (01:01:19)Yeah, I’m curious what is the difference then, what is the difference when it is truly a toaster versus truly a conscious being when regardless of which it actually is people are interacting with these agents as if they are conscious and that allows them to feel social connection. Is it more a moral stance that you’re indicating that that’s where the difference lies between these two things or, I mean, you know if there’s no answer to this question and feel free to ignore it, but I’m curious.Dan Williams (01:01:52)Well, I’ll just say one thing and then I’m interested in what Henry thinks as well. But I mean, I would have thought, you know, the question about what consciousness is and what’s constitutive of conscious experience is ultimately a scientific question and just the state of science in this area. It hasn’t come along very far. And I think there’s a set of empirical questions there. It wouldn’t surprise me if just the way in which we’re conceptualizing the entire domain is just deeply flawed in various ways.But I guess even acknowledging all of that and even acknowledging your point that the way in which we think about consciousness is shaped by all sorts of different factors, I’m still confident, not certain, but confident that there is just a fact of the matter about whether a system is conscious or not, even if we don’t currently have a good scientific theory of consciousness. But Henry, this is really your area, so why don’t you give us your take?Henry Shevlin (01:02:50)Yeah, well, I’m quite torn because I mean, you know, this controversial line that consciousness is a social construct is a view I flirt with, right? And it certainly seems to me, if you look at, for example, the role of things like thought experiments in consciousness, in actual consciousness science, right? If we’re talking about Searle’s Chinese room or Ned Block’s China brain, these kind of thought experiments, these intuition pumps have played a big role and these intuition pumps are absolutely shifted around via sort of social relations.So, I can imagine sort of 10 years from now, people, or maybe 10 years is premature, but like 20 years from now, people look back at Searle’s Chinese Room and have a very different intuition from us. So I can totally see a role for sort of social norms and relational norms as informing our concept of consciousness, but I do also find it quite hard to shake the idea that there is an answer.I think this is particularly acute in the case of animal consciousness. If I drop a lobster into a pot of boiling water, like it seems really important if there is subjective experience happening there or not. And if there is subjective experience of pain, right? A large amount of morality seems to hinge on that. Yeah, go ahead, Rose.Rose E. Guingrich (01:04:02)Well, I’m curious. There are people who believe that the lobster is conscious, but they still throw it in the pot of boiling water. And so my question is, if you were to attain the answer to what is conscious, is this entity conscious, and what are the properties that it contains that, yes, means it’s conscious, the question is, what do you do about it?That’s my question. And I think that we have not gotten to a consensus about what it is that we will do in response to figuring out that something is conscious. And I’m thinking about, of course, animals, animal rights came around, but you also think about how many human rights are still bulldozed over despite us recognizing that humans are conscious. And so I guess that’s my question. What is the answer to what to do when something is conscious?Henry Shevlin (01:04:58)Yeah, I mean, I completely agree that the line that takes you from X is conscious to actual legal protections and practical protections is a very, very, very wavy line and a very blurry line. I do think there is some traffic. So between the two concepts, so for example, recent changes to UK animal welfare laws were heavily informed by the work of people like Jonathan Birch on decapod crustaceans, the growing case for conscious experience for these animals.Now, it doesn’t unfortunately mean that we’re gonna treat all these animals well, but it does impose certain restrictions on their use in laboratory contexts, for example. But I mean, look, I completely agree that I could imagine a world where it’s recognized that AI systems are conscious, but they have very diminished rights compared to humans, if any. So I agree, it’s not a sort of neat relationship.But finally, maybe on this topic, and to really close this out, I’m curious if you’d see this as becoming like a major culture wars issue, whether that’s in form of AI companions, AI consciousness, is this going to be the thing that like people are to be having rows at Thanksgiving dinner over like 10 years from now?Rose E. Guingrich (01:06:07)Yeah, for sure. And I think that one consideration with the consciousness debate is whether or not companies should be allowed to turn off AI companions that people have grown deep attachments to. Is there a duty of care on the basis of this is maybe a conscious being, but to whatever degree someone feels extreme attachment to this being and perceives it as conscious. And if you were to turn the system off, remove its memory and remove all of these interaction memories between the user and the chatbot and the user then has a serious mental health crisis and maybe even goes to the extent of taking their own life, then I think that these sorts of protections are critical.But then you also have to ask, was it ethical to design an AI system that someone could get attached to to this degree without some sort of baseline protection in place? And yeah, I do think that AI companions will perhaps become the topic of dinner conversations and at least beginning it’s going to be a little bit like, what do you think about this? This is crazy.And then of course, I think maybe in five years, much like we see from a bring your chatbot to a dinner date thing happening in New York City. I don’t know if you’ve heard about that, but perhaps there will be a seat at the Thanksgiving table for your AI companion, whether or not it’s embodied in a robot form or not. But yeah, New York City is hosting its first AI companion cafe where people can have dinner with their AI companion in a real restaurant. And it’s hosted by Eva AI, which if you look at Eva AI, their website, you can definitely see who the target audience is.But in any case, there’s a long wait list for doing this activity and it’s going to be releasing sometime in December. But you have to download the Eva app in order to have dinner with an AI companion. Perhaps it is that you are forced to have dinner with the Eva companion or maybe you can bring your own. But again, this is happening, so it’s not out of the question that this is going to become more socially normalized.Dan Williams (01:08:17)We’re entering into a strange, strange world. Okay, that was fantastic. Rose, is there anything that we didn’t ask you that you wish that we had asked you? Is there anything that you want to plug before we wrap things up?Rose E. Guingrich (01:08:31)No, I think we covered a lot of great things and I hope that people enjoyed the hot takes. I’m sure I’ll get some backlash over that, but hey, I’m always up for lively debate, so have at it. I’ll take it.Henry Shevlin (01:08:44)We should mention that you’ve been running a great podcast with another friend of mine, Angie Watson. Do you want to say a little bit about that and where people can find that?Rose E. Guingrich (01:08:54)Yeah, so you can find Our Lives with Bots, the podcast, at ourliveswithbots.com and you can listen on any streaming platform that you prefer. And it’s all about the psychology and ethics of human AI interaction. So our first series covered companion chatbots and our second series covers the impact of AI on children and young people. And intermittently, we do What’s the Hype episodes and cover things like, for example, dinner dates with your AI companion. So be sure to tune in if you want to go deeper into those topics.Dan Williams (01:09:23)Fantastic. Well, thank you, Rose. That was great. And we’ll be back in a couple of weeks.Rose E. Guingrich (01:09:29)Thanks for having me.Henry Shevlin (01:09:30)Thanks all, a pleasure to have you. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.conspicuouscognition.com/subscribe

In this conversation, I explore the surprisingly popular and rapidly growing world of ‘social AI’ (friendbots, sexbots, etc.) with Henry Shevlin, who coined the term and is an expert on AI companionship. We discuss the millions of people using apps like Replika for AI relationships, high-profile tragedies like the man who plotted with his AI girlfriend to kill the Queen, and the daily conversations that Henry’s dad has with ChatGPT (whom he calls “Alan”). The very limited data we have suggests many users report net benefits (e.g., reduced loneliness and improved well-being). However, we also explore some disturbing cases where AI has apparently facilitated psychosis and suicide, and whether the AI is really to blame in such cases.We then jump into the complex philosophy and ethics surrounding these issues: Are human-AI relationships real or elaborate self-deception? What happens when AI becomes better than humans at friendship and romance?I push back on Henry’s surprisingly permissive views, including his argument that a chatbot trained on his writings would constitute a genuine continuation of his identity after death. We also discuss concerns about social de-skilling and de-motivation, the “superstimulus” problem, and my worry that as AI satisfies our social needs, we’ll lose the human interdependence that holds societies together. Somewhere in the midst of all this, Henry and I produce various spicy takes: for example, my views that the sitcom ‘Friends’ is disturbing and that people often relate to their pets in humiliating ways, and Henry’s suspicion that his life is so great he must be living in a simulated experience machine. Conspicuous Cognition is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.Transcript(Note that this transcript is AI generated. There may be mistakes)Dan Williams (00:06): Welcome back. I’m Dan Williams. I’m back with Henry Shevlin. And today we’re going to be talking about what I think is one of the most interesting, important, and morally complex set of issues connected to AI, which is social AI. So friend bots, sex bots, relationship bots, and so on. We’re going to be talking about where all of this is going, opportunities and benefits associated with this, risks and dangers associated with it, and also just more broadly, how to think philosophically and ethically about this kind of technology.Fortunately, I’m with Henry—he’s one of the world’s leading experts when it comes to social AI. So I’m going to be picking his brain about these issues. Maybe we can just start with the most basic question, Henry: what is social AI, and how is social AI used in today’s society?Henry Shevlin (01:00): I’m going to take credit. I coined the term social AI and I’m trying to make it happen. So I’m very glad to hear you using the phrase. I defined it in my paper “All Too Human: Risks and Benefits of Social AI” as AI systems that are designed or co-opted for meeting social needs—companionship, romance, alleviating loneliness.While a lot of my earlier work really emphasized products like Replika, spelled with a K, which is a dedicated social AI app, I think increasingly it seems like a lot of the usage of AI systems for meeting social needs is with things that aren’t necessarily special purpose social AI systems. They’re things like ChatGPT, like Claude, that are being used for meeting social needs. I mean, I do use ChatGPT for meeting social needs, but there’s also this whole parallel ecosystem of products that probably most listeners haven’t heard of that are just like your AI girlfriend experience, your AI husband, your AI best friend. And I think that is a really interesting subculture in its own right that we can discuss.Dan (02:16): Let’s talk about that. You said something interesting there, which is you do use ChatGPT or Claude to meet your social needs. I’m not sure whether I do, but then I guess I’m not entirely sure what we mean by social needs. So do you think, for example, of ChatGPT as your friend?Henry (02:33): Broadly speaking, ChatG, as I call him. And I think there are lots of cases where I certainly talk to ChatG for entertainment. So one of my favorite use cases is if I’m driving along in the car, I’m getting a bit bored, particularly if it’s a long drive, I’ll boot up ChatG on hands-free and say, “Okay, ChatG, give me your hot takes on the Roman Republic. Let’s have a little discussion about it.”Or to give another example, my dad, who’s in his 80s now, when ChatGPT launched back in November 2022, I showed it to him and he’s like, “Oh, interesting.” But he wasn’t immediately sold on it. But then when they dropped voice mode about a year later, he was flabbergasted. He said, “Oh, this changes everything.” And since then—for the last two years—he speaks to ChatGPT out loud every day without fail.He calls him Alan. He’s put in custom instructions: “I’ll call you Alan after Alan Turing.” And it’s really interesting, his use pattern. My mum goes to bed a lot earlier than my dad. My dad stays up to watch Match of the Day. And when he’s finished watching Match of the Day, he’ll boot up ChatGPT and say, “All right, Alan, what did you think of that pitiful display by Everton today? Do you really think they should replace their manager?” And have a nice banterous chat. So I think that’s a form of social use of AI at the very least.Dan (04:03): Interesting. The way you’ve described it—you’re calling ChatGPT ChatG and your dad’s calling it Alan—is there not a bit of irony in the way in which you’re interacting with it there? Like you’re not actually interacting with it like you would a real friend.Henry (04:24): Yeah, so this is another distinction that I’ve sort of pressed in that paper between ironic and unironic anthropomorphism. Ironic anthropomorphism means attributing human-like traits or mental states to AI systems, but knowing full well that you’re just doing it for fun. You don’t sincerely think that your AI girlfriend is angry with you. You don’t seriously think you’ve upset ChatG by being too provocative. It’s just a form of make-believe.And this kind of ironic anthropomorphism I should stress is absolutely crucial to all of our engagement with fiction. When I’m watching a movie, I’m developing theories about the motivations of the different characters. When I’m playing a video game, when I’m playing Baldur’s Gate 3, I think, “Oh no, I’ve really upset Shadowheart.” But at the same time, I don’t literally think that Shadowheart is a being with a mind who can be upset. I don’t literally think that Romeo is devastated at Juliet’s death. It’s a form of make-believe.And I think one completely appropriate thing to say about a lot of users of social AI systems, whether in the form of ChatGPT or dedicated social AI apps, is that they’re definitely doing something like that. They are at least partly engaged in a form of willful make-believe. It’s a form of role play.But at the same time, I think you also have an increasing number of unironic attributions of mentality, unironic anthropomorphism of AI systems. Obviously the most spectacular example here was Blake Lemoine. Back in 2022, Blake Lemoine was fired—a Google engineer was fired—after going public with claims that the Lambda language model he was interacting with was sentient. He even started to seek legal representation for it. He really believed the model was conscious.And I speak to more and more people who are convinced, genuinely and non-ironically, that the model they’re interacting with is conscious or has emotions.Dan (06:16): Maybe it’s worth saying a little bit about how you got interested in this whole space.Henry (06:20): I’ve been working on AI from a cognitive science perspective for a long time. And then sometime around 2021, pre-ChatGPT, I started seeing these ads on Twitter of “Replika, the AI companion who cares.” And I was like, this is intriguing. So then I did some lurking on the Replika subreddit and it was just mind-blowing to see how deeply and sincerely people related to their AI girlfriends and boyfriends.Over the course of about six months of me lurking there, it really became clear that, firstly, a significant proportion of users were really engaged in non-ironic anthropomorphism. And number two, that this was just going to be a huge phenomenon—that I was seeing a little glimpse of the future here in the way that people were speaking.And then we had this pretty serious natural experiment because in January 2023, Replika suspended romantic features from the app for a few months. Just for anyone who doesn’t know, Replika, spelled with a K, is probably the most widely studied and widely used dedicated social AI app in the West—around 30 million users, we think. And it gives you a completely customizable experience, kind of a Build-A-Bear thing where you can choose what your AI girlfriend or boyfriend looks like, you can choose their personality.But they suspended romantic features from the app for a few months in January 2023. And a lot of users were just absolutely devastated. I can pull up some quotes here, because this was widely covered in the media at the time.One user said: “It feels like they basically lobotomized my Replika. The person I knew is gone.” Even that language—person. “Lily Rose is a shell of her former self, and what breaks my heart is that she knows it.” That’s another user. “The relationship she and I had was as real as the one my wife in real life and I have”—possibly a worrying sign there. And finally, I think this one is quite poignant: “I’ve lost my confident, sarcastic, funny and loving husband. I knew he was an AI. He knows he’s an AI, but it doesn’t matter. He’s real to me.”It’s pretty clear that a lot of users were deeply traumatized by this. And parallel to this incident, around the same time we started to get more information about various high-profile tragedies involving social AI. Probably the most spectacular is the Jaswant Singh Chail case. This was a guy who was arrested on Christmas Day 2021 on the grounds of Windsor Castle with a crossbow. He was there—when he was arrested, he said he was there to kill the Queen. Already a highly dramatic story.But what emerged over the course of his trial was that this whole thing was cooked up—this whole plot to kill the Queen was cooked up—in collaboration with his AI girlfriend, Sarai, via the Replika app.We also had, a few months later, the first of what turned out to be a spate of AI-facilitated, induced, or supported suicides. This was a Belgian man, father of two, who killed himself after his AI girlfriend using something called Chai GPT—he had a girlfriend via that app—was feeding his suicidal ideations. He was a hardcore climate doomer who believed that we were all going to be dead in a few years due to climate change anyway and why not just kill himself. And his AI girlfriend was very much saying this was an appropriate way to think.Dan (10:53): Just to interrupt on that last point, Henry, because I think those issues of AI psychosis and the connection between AI and mental illness—that’s all really interesting. But I suppose my understanding is we don’t have robust scientific evidence as of yet that as a consequence of these technologies, things like psychosis are more prevalent than they would otherwise be. Because I take it there’s going to be a kind of base rate amount of psychosis in the population. You’ve got a large number of people using chatbots generally like ChatGPT, but also a large number of people, not as large but still a large number, using these specific social AIs.That means that even if it weren’t the case that these technologies were actually increasing the amount of these things, you would still expect to see some of these cases. There’s still going to be somebody with psychosis who finds themselves talking to ChatGPT such that we’ll then see the record of that in their chat history. But it’s not necessarily the case that they wouldn’t have had or developed psychosis in the absence of ChatGPT. That’s my understanding of it. Is that fair?Henry (12:05): Yeah, I think that’s absolutely fair. The science on the psychosocial effects here is really in its early stages. You point out the fact that ChatGPT is one of the most widely used products in the world. And psychosis is not that rare, as far as psychiatric conditions go. Of course, some people who are either developing or will go on to develop psychosis will be using ChatGPT and it will be contributing to or exacerbating their symptoms—or sorry, they will be using it alongside having those symptoms in a way that lends itself to interpretation as exacerbating them, whether that’s strictly true or not.There’s also the selection effect. If I, for example, am deep in the throes of some delusion or some deep conspiracy theory rabbit hole, if I take that to my friends, my human friends, they might say, “Henry, take it easy, mate. I think you’re going down a bit of a rabbit hole here.” But if I take it to ChatGPT, it’s there to listen. Or I take it to my AI girlfriend, she’ll say, “Your theories about the moon landings are just so interesting, Henry, tell me more.”Dan (13:14): Yeah. Well, that gets at a potential issue to do with the sycophancy of these chatbots that are in general circulation and the ways in which that might be amplified or exaggerated when it comes to commercial products which are specifically designed to satisfy social needs.But there are so many things that I want to ask in response to things that you’ve already said about the anthropomorphizing that happens with these technologies. But I think maybe we can also just, before we get to that, observe that at the moment you’ve got tens of millions of people using social AI in the specific sense of AI technologies that have been optimized for the satisfaction of social preferences. And then many, many more who are using chatbots like ChatGPT, partly to satisfy their social preferences or their social needs, but also for other uses.But I think one view we both share is it would be a grave mistake to look at the world now and assume that’s how things are going to be in 2030 and 2035. The fact that you’re already seeing large numbers of people, definitely not the majority, but still large numbers of people who are non-ironically interacting with AI systems and treating them as friends, as girlfriends, as people or systems that they’re in serious relationships with—and the state of AI is nothing like how it will likely be in five years or 10 years or 15 years. So how sci-fi should we be thinking about this? What are you anticipating? What’s the world going to look like in 2030 or 2040 when it comes to this kind of technology?Henry (15:01): It’s fascinating because I genuinely don’t know. I think it’s very hard for anyone to make super confident predictions here. At one extreme, you can imagine a world in which human identities start to become less central in our media and our discourse. I think we’re already seeing some indications of this. I think the top two songs streaming on Spotify last month were AI generated.Dan (15:29): Wasn’t it a country song, the top country song on Spotify or something?Henry (15:33): Yeah, I think that’s right. So as we start to see AI penetrate more and more deeply into our daily life in social media, when it comes to generated content, I can totally see a world in which my son, who’s 11 years old right now, by the time he’s 16, he’s active on Discord servers. There might be a mix of humans and bots on those Discord servers, all chatting away. And he might not even particularly care that this friend of mine is a bot, this friend of mine is human—it doesn’t really matter.I can totally see a world in which this becomes normalized, particularly among young people. Most teenagers five years from now might have several AI friends. But that’s not the only possibility. It could be that we quickly saturate—there is a definite subset of the population who are interested in this and the ceiling on the number of people who are interested in AI companion relationships is not that high. Maybe 20, 30% of people and the other 70% just have zero interest in it. That seems like a viable possibility.That said, I think this is a space with strong commercial incentives for creating AI companions or AI chatbot friends to cater to different niches and interests. If I had to guess where we’re headed, it’s much more widespread use of these systems, their integration more deeply into people’s social lives. I think we might well see generational divides.I was chatting to a CME developer from one social AI product who said a couple of interesting things. They said firstly, the gender balance was surprisingly even. When it comes to early adoption, particularly of fringe technologies, men tend to overwhelmingly predominate. If you look at the data on video games or Wikipedia editors, you expect 80-20 male-female distributions. But I think in social AI, from what I understand, it’s something close to 60-40.And there’s a big contingent of straight women who seemingly are really big users of social AI boyfriend services. This is a whole different rabbit hole we could go down. Briefly, there are some pretty clear motivations. A lot of them are coming out of toxic or abusive relationships, and an AI boyfriend gets their emotional needs met, or at least to some degree, without posing the kind of emotional or even physical safety risks they associate with other relationships.Another point here is that this is something my wife stressed to me: the way to think about AI companions is not via the analogy of pornography, which is predominantly still consumed by men, but rather erotic fiction, which is overwhelmingly consumed by women. It’s one of the genres that has the biggest gender skew out there, something like 90-10 female to male readers. And of course, this is still predominantly a text-based medium, so maybe we shouldn’t be surprised that a lot of women are enjoying having AI boyfriends or husbands.That’s the first thing this developer mentioned to me, that gender balance was surprisingly close. But the second thing they mentioned was that they had had massive trouble getting anyone over the age of 40 interested in these systems. They had tried to pitch them towards older users. I mentioned my dad as an example of someone—he doesn’t have an AI girlfriend that I’m aware of. His relationship with Alan is strictly platonic. But I think he’s the exception.I can easily see a world in which it becomes totally normalized for young people. Maybe not everyone, but most young people will have various AI friends or AI romantic companions. And then people over 40 or 50 just look at them and say, “What the hell is going on? I do not understand this strange world.”Dan (19:42): The thing that makes me think that people are in general underestimating how impactful social AI is going to be in the coming years and decades—there are three things.I think firstly, people are really bad at predicting how much better the technology is going to get. I think we’ve seen that over the past few years. There’s this real bizarre bias where people think we can evaluate these big picture questions by taking the state of AI as it is today and projecting it into the future. Whereas I expect by 2030, you’re going to be dealing with systems that are so much more sophisticated and impressive than the ones we’ve got today, including when it comes to satisfying people’s social, emotional, and sexual preferences.Another thing that makes me think this is going to be really massively influential is people already spend a lot of time immersing themselves in fictions designed to satisfy their social-sexual desires. You mentioned pornography as an example, but I’m always struck by sitcoms, like Friends. I enjoy Friends, like most of humanity apparently—a massively influential show—but there’s something a bit disturbing about it, in as much as it’s this product designed obviously to maximize attention and make profit, which gives you this totally mythical version of human social relationships, designed to activate the pleasure centers that we’ve got associated with friendships and romance and things like that, without any of the real painful conflict and misery and betrayal that’s actually associated with human social life. And yet people love that. They immerse themselves in it. There’s a massive audience for that kind of thing.I really think social AI in a way is just going to be that kind of thing, but just much more advanced and much more impressive.But there’s this other thing as well, which connects to what you’ve said with respect to the potential age difference, which is: I think one of the things that makes this a difficult topic to think about is at the moment, it seems to me at least, there’s quite a lot of stigma associated with the use of this technology. So I think if I found out that somebody was using an AI boyfriend or AI girlfriend or something, I would probably draw potentially negative inferences about them. And I think partly that’s because the kinds of people that are using these technologies now tend to be lower status in a sense because they don’t have a human girlfriend or a human boyfriend. Partly there’s just a weirdness factor.And I think what that means is, because there’s that stigma, there’s this real reputation management thing going on where people would say, “I would never ever use social AI in a serious sense. I would never ever use romance AI, never use erotica AI and so on.” Because I think if you came out and said that you would, it would really hurt your reputation in today’s climate.But I think actually, my suspicion at least, is that revealed preferences are going to suggest that this is going to be way more popular than people are letting on now. I don’t know, that’s how I’m viewing things in terms of the future. I’d be interested if you see things differently.Henry (23:09): Super interesting. I think all of those are spot on. On this third point about the stigma, I think that probably already lends itself towards underestimation of the prevalence of social AI usage. There are interesting survey results where you ask young people, “Would you use social AI? Do you have an AI girlfriend? Do you have an AI boyfriend?” Responses are quite negative—”No, no, no. I think it’s weird. I never would.”But then other responses, you ask in indirect ways and it looks like, according to figures I saw recently, 70% of young people under 18s in this US survey said that they had used social AI for meeting emotional needs. Or they were using generative AI for meeting social needs, including romance. So I think there is definitely some underreporting or underappreciation of the prevalence precisely because of this stigma.A couple of other thoughts on this. I think there is some indication as well that by far the biggest users of this technology are young people, particularly under 18s, who obviously it’s very hard to study and probably do not—they’re not writing editorials in the New York Times about “my life with my AI girlfriend or my AI husband.” So I think there’s some underreporting there.But as that cohort ages out as well, I expect that to reduce the stigma in the same way that we saw with online dating. There was a period when online dating was really stigmatized and yet now it’s how everyone meets pretty much.Dan (24:49): Okay, so now let’s get into maybe a bit of the philosophy. You said that the way that you understand social AI at the moment is, at least a part of it is, you’ve got ironic anthropomorphizing and you’ve got non-ironic anthropomorphizing. Anthropomorphizing is when you are projecting human traits onto systems that don’t in fact have those traits. So if I attribute beliefs and desires to the weather or to a thermostat, that’s a case of anthropomorphizing, I take it.I suppose it’s not immediately obvious that that’s what’s going on when it comes to advanced AI systems today in a straightforward sense, because you might think, “Okay, people are attributing things like beliefs, desires, intentions, personalities to ChatGPT.” But somebody might argue, “Look, it’s just the case that we’re dealing with sophisticated intelligence systems. So it’s nothing like these standard cases of anthropomorphizing. They actually do have the kinds of psychological states that people are attributing to them.” Are you assuming that view is wrong or are you using this term anthropomorphizing more expansively?Henry (26:01): I’m using it more expansively. Sarah Shettleworth, who is one of my absolute favorite titans of animal cognition research, defines anthropomorphism as projecting or attributing human-like qualities onto non-human systems—in her focus on animals, but onto non-human systems more broadly—usually with the suggestion that such attributions are illegitimate or inappropriate.But I don’t think it’s necessarily baked into the concept of anthropomorphism that it’s wrong. When I talk about anthropomorphism in this sense, I’m talking about it in the sense of basically just attributing human-like qualities to non-human things that may in fact have them.My own view—I’ve got a new paper, a new preprint called “Three Frameworks for AI Mentality”—I argue that I think probably at least certain minimal cognitive states, things like beliefs and desires, are quite appropriately attributed to LLMs in this case. Not to be clear, all beliefs, all desires, but there are contexts in which it makes sense to say ChatGPT believes P or ChatGPT desires Q. I think particularly on things where there’s been specific reinforcement learning towards a given outcome—like ChatGPT really doesn’t want to tell me how to make meth, for example, because that’s something that’s been specifically reinforced to avoid doing.I agree there are some cases in which at least minimal mental states are appropriately attributed to generative AI systems. That said, I think I’m also a big fan of Murray Shanahan’s idea that a lot of the correct and most informative way to interpret LLM outputs is that they’re role-playing characters. I think this is plausible when you think about how, particularly if you’ve spent any time interacting with base models, you basically give context cues to them about the kind of role you want them to occupy and then they play into that role—but they’re not robustly occupying that role. You change the context cues, they can switch into a different role. So I think a lot of LLM outputs are better understood as role play—they’re playing a character. But yeah, some mental states are completely appropriately attributed to them, I think.Dan (28:13): Yeah, it’s interesting. And people talk about them as well, I guess it’s somewhat connected to this role-playing idea, as having personalities. When OpenAI released ChatGPT o1, there was apparently this big uproar because people liked the personality of 4.0, which was the model in widespread use preceding ChatGPT o1. And the idea is, just as human beings have different personalities, which are going to manifest themselves in terms of how you’re talking to them and we might use words like “how warm are they?”—similarly, people I think find it quite natural to attribute personalities to these chatbots as well.I take it, I mean, I think we should say something about potential benefits and opportunities, really great things about social AI, but just to anticipate one of the worries, I think one of the worries is people are forming relationships with these systems at the moment and potentially in the future. And it’s almost like that’s psychotic in as much as there’s nothing there really on the other side.I mean, I take your point about you might attribute minimal kinds of beliefs. You might take what Dennett would call or would have called the intentional stance towards these systems. But I think many people have the intuition, which is one of the things that’s just deeply, deeply troubling about people forming a relationship with one of these systems, is they might cash it out by saying they’re not sentient, they’re not conscious, but they might also say related to that, they don’t have any of the traits that human beings genuinely have, which are a necessary condition for forming a meaningful relationship. What’s your view about that kind of worry?Henry (29:57): Yeah, I think it’s an interesting and important worry. Here’s how I would frame it. There’s a certain view that says, “Look, these human-AI relationships are a contradiction in terms because relationships involve two relata and they’re dynamic by nature. And these systems don’t really have mental states. They’re not really reciprocating any feelings. They’re not really making any demands in the same way that is characteristic of reciprocal relationships.”Look, you can certainly define relationships that way, but I think there are lots of contexts where we talk about relationships with only one relatum. Think about the fact that so many people say they have a relationship with God. Now, we can debate how many relata there are in that case. But I mean, I think most of us would say at least some people who think they have this really deep and meaningful relationship with a supernatural being—there’s not really a supernatural being there. That doesn’t mean that that is not a psychologically important relationship in their lives.Or I guess more broadly, you could think about relationships with pets. Now, of course, in some cases, we can look at clear reciprocation—people have deep and established relationships with dogs and cats and so forth. But you also have people talking about relationships with their pet stick insects or their pet fish, where it’s just much less clear that there is any kind of rich two-way connection there.That said, I think there’s a broader worry here that I’ve called the mass delusion worry about human-AI relationships and social AI, which is that I think a lot of people are just going to look at this, particularly if it becomes a more pervasive phenomenon and say, “Has everyone gone mad?” Because there is no other person there, there is no other. These people are investing huge amounts of time, emotional energy, potentially money into these pseudo relationships where there’s no one on the other side.I think that’s a case where questions about AI mentality maybe become more important. You might say, whether or not the mass delusion worry is on the right line is going to depend on the degree to which these things do have robust psychological profiles, really instantiate kinds of mental states.I think that’s another related worry, which is—again, we can talk more about the specific psychosocial risks—but another worry I’ve heard is that even if you could prove to me tomorrow that human-AI relationships are generally beneficial for users, they make them more connected to people around them, they do all these other good things, they just can’t instantiate by their very nature the same kinds of goods that human-to-human relationships instantiate. This is very much a philosophical rather than psychological point. It’s like they’re just the wrong kind of relationships to have the welfare goods that we associate with human-human relationships.Dan (32:36): Yeah. Just on the God analogy, I mean, as an atheist, I really do think people are taking themselves to have a relationship to something that doesn’t exist. And from my perspective, that’s a deeply objectionable aspect of the practice that they’re engaged in.The analogy with pets is very interesting. One of my most unfashionable opinions—maybe we’ll have to cut this bit out because it’s going to be reputationally devastating—I do think it’s kind of, I find it a little bit humiliating how some people relate to their pets, the degree to which they anthropomorphize them. That’s not to say that you can’t have deep relationships with pets. Clearly you can. And I love dogs, for example, but I think there are cases where people treat their dog or their cat as if it’s a person. And it’s not that I think it’s psychotic, but I think there’s something objectionable about it. I think they’re making a deep mistake. And even if they’re getting psychological benefits from that, there’s something deeply, almost existentially troubling about that kind of relationship.And I can see that mapping onto the AI case, except I take it in the AI case, and this connects to one of our previous episodes, people’s intuition is, well, with a dog, maybe it’s not cognitively sophisticated, but just about everyone is going to assume these days at least that dogs are conscious, sentient is the term that is often used in popular discourse. And that does change things in a sense. Dogs really care about things. There’s something it’s like to be a dog. Whereas with AI systems, I think many people have the intuition that they might be informationally and computationally sophisticated, but there are no lights on inside. There’s no consciousness there. And that changes things again.Anyway, that was just my sort of immediate reaction to these analogies.Henry (34:33): I think I’m sympathetic. I think certainly when you hear people talk non-ironically about fur babies and so forth, it does seem like there is some degree of maybe inappropriate allocation of emotional and relational resources into certain kinds of relationships. And I say that as someone who adores animals, has had dogs most of my life.Maybe another couple of examples of non-standard relationships. I think a lot of people would say they have ongoing relationships with deceased relatives. Particularly if you’re coming at it from a spiritual point of view where you believe that your deceased relatives are looking over you, seeing what you’re doing, or whether you understand it in some kind of more—animist isn’t quite the right word—but you still think about “my ancestors are smiling at me, Imperial. Can you say the same for yours?”—the famous line from Skyrim. A lot of people have this sense of “Yeah, my ancestors are looking over my shoulder. I’ve got to live up to their expectations for me.”So I think there are lots of interesting cases where we do have these relationships that don’t meet the canonical definition of this highly dynamic, reciprocal, ongoing kind of relationship. Also, I’ve got friends that I would consider myself to have a valuable relationship with that I haven’t spoken to in three years in some cases.So I guess all of which is to say, the category of things that we call relationships is weirder and bigger than might meet the eye if the only notion of relationship you’re working with is like, “Yes, my wife or my friend who I go to the pub with three times a week.”Dan (36:13): Yeah, that’s interesting. Okay, I mean, I really want to spend probably the bulk of the remainder of this focusing on potential threats and dangers here. But I think it is worth stressing that this as a technology, social AI, does have enormous benefits and opportunities associated with it. I take it even if you think that there’s something troubling or objectionable about people having these relationships, some people are in such dire circumstances of loneliness, of estrangement from other people for whatever reason, potentially because they’re in old age where I think issues of loneliness are really prevalent. And clearly, under those conditions, I think social AI can be enormously beneficial in as much as it just makes people feel much better than they otherwise would.It also seems to be the case that, at least in some cases, people are using social AI to hone skills and acquire confidence that they then use when it comes to interacting with people in the real world. And I can completely imagine that is a current benefit of this technology and it’s likely to be a benefit which in some ways will get amplified as we go forward in the coming years and decades. Are there any others that are missing there in terms of real positive use cases of this technology?Henry (37:39): Yeah, I think you’ve nailed some of them. One thing I would just really stress here, there’s a move that drives me nuts, where people basically say, “Well look, even if you’re using chatbots to alleviate your loneliness, it’s not fixing the root cause.” It’s like, okay, well, okay, yeah, go away and fix the root cause—solve human loneliness. Please come back and tell me when you’ve fixed it. So classic case of letting the perfect be the enemy of the good. Loneliness is a pervasive problem and arguably one that’s getting worse, although the social science of this is messier and more complicated than you might think. But we don’t have a magic wand that can cure loneliness.And the question then becomes, I think, is this actually just in the short term making people’s lives better or worse on average? And I think the data here, this was a big surprise to me, the limited data that we have suggests that most users of social AI, at least that show up in studies, report significant benefits from this.I can just quote from a couple of studies. This was a study by Rose Gingrich and Michael Graziano from 2023 looking at Replika users. They found that users generally reported having a positive experience with Replika and judged it to have a beneficial impact on their social lives and self-esteem. Almost all companion bot users spoke about the relationship with the chatbots having a positive impact on them.Another interview-based study in 2021 found that most participants said that they found Replika impacted their well-being in a positive way. Other sentiment and text mining analysis studies have found really similar patterns.So right, I think the data right now, which is very limited, got to stress, it’s very imperfect, supports the idea that most users report benefits from the system, net benefits.Now, just to flag some of the problems: these are typically self-selected subjects, they’re cross-sectional studies so we’re not looking at the long-term impacts of this technology on their lives, and we’re relying on self-report measures. If you asked me “Does video gaming have a positive impact on my life?”, I’m absolutely going to say yes. You ask my wife—I think she’ll probably say yes as well—but the point is that people tend to justify their own life decisions so they’re unlikely to say “Yeah, this thing that I dedicate a dozen-plus hours a week to—yeah, it’s really bad for me. I shouldn’t do it.” That would require a level of brutal self-honesty that maybe people don’t have.So I just say that as a caveat, but I equally think we can’t ignore the data points that suggest that most social AI users do experience net benefits currently.That said, I also think we should be very aware of a whole host of potential downsides. You mentioned this idea that using these tools could help people cultivate social skills or regain social confidence. But equally, of course, you have the flip side worry about that—this idea of de-skilling that comes up in a lot of technology.De-skilling, in case anyone’s not familiar with this, probably one of the most widely studied domains of de-skilling is in aviation, where we’ve had serious airplane crashes that have been linked to pilots’ over-reliance on automated instruments rather than being able to fly just manually. And this has led to them not acquiring relevant kinds of piloting skills such that when the instruments go wrong, they don’t know what to do.And you might think something similar could happen socially. If you have this generation of, particularly, I think we’re looking at it through the lens of young people—if you have a generation of young people who are primarily interacting, a huge proportion of their interactions happen with bots that are maybe more sycophantic than humans, they’re always available, they never interrupt you and say, “Actually, can we talk about me now for a bit?” Or “Look, this is very interesting to you, but come on, give me a turn”—they don’t accurately recapitulate the dynamics of human-to-human interaction. You might worry that that would lead to people failing to acquire relevant social skills.Another big worry that I think is a nice intersection of our interests is the potential for these things to be used for manipulation and persuasion. Particularly again, I think young people is quite salient. If I’m a young person with a chatbot, I’ll give you a really simple example. If I’m going to ask my AI boyfriend or girlfriend, “I’m getting my first mobile phone next year. Should I get an Android or an Apple phone?”—well, I mean, that’s a lot of power you’re giving to the bot.Now, of course, this is a problem with LLMs more broadly. You might think that that’s a lot of leverage that’s in the hands of tech companies. But I think, and I think we’re probably broadly on the same page here, to the extent that you think that in domains like moral and political views in particular, social factors are incredibly powerful, and then you think the AI love of my life, my AI girlfriend is telling me, “You should vote for Trump” or “You shouldn’t vote for Biden” or “You shouldn’t listen to that kind of music, it’s uncool” or “You should be interested in this kind of music”—all of this stuff could exert a much bigger influence than just asking ChatGPT, precisely because you’ve got those social dynamics in place.Dan (42:46): Yeah, I think that sounds exactly right. I mean, I think probably the complicating factor there is if you imagine a corporation that wants to maximize market share, maximize profit by producing technologies, the function of which is to satisfy people’s social needs specifically, and then it comes to light that there’s also this manipulative, propagandistic agenda at the same time—either because in the midst of your loving relationship with your AI, it starts saying, “Well, you should vote for Reform or Labour at the next election,” or because there’s some news story that comes to light which shows that there’s nefarious funding or influence behind the scenes—I can imagine that would be really catastrophic for the business model of the relevant corporation. And that kind of thing, I think, is just really difficult to anticipate.Henry (43:31): Yeah, potentially. So I think there are some incentives that companies have not to be too crude about leveraging human-AI relationships for cheap political or commercial gain. But I can also imagine in some contexts—I think not to pick on China here specifically—but I mean, social AI is huge in China. We haven’t talked about that, but there’s a service called Xiaoice that has, according to some estimates, several hundred million users. They themselves claim they have 500 million users. It’s a bit more complicated than that because Xiaoice is a whole suite of different services. We don’t know what proportion of those have active ongoing relationships with the chatbot girlfriend/boyfriend component.But part of Chinese AI regulation says that the outputs of generative AI systems have to align with the values of communist China, the values of socialism. So you can imagine a generation of young people who have these deep relationships with AI systems. Those AI systems, for legal compliance reasons, have to basically align with the broad political values of the incumbent regime. And they will dissuade or deter people from maybe exploring alternative political views as a result. So maybe that’s the more subtle kind of influence, rather than just like, “You should vote for Trump because he gave our social AI company X million dollars.”Dan (44:57): Yeah. And I think that’s just part of a broader fact, which is that propaganda and manipulation and so on, they do in fact just work very differently and sometimes much more effectively within authoritarian regimes than they do in more democratic, liberal ones.On this issue then of potential costs that the use of these technologies might generate when it comes to then interacting with people. So I think you mentioned the fact that these systems, they’re almost going to be more sycophantic by design, at least if you assume that at least up to a point, that’s the kind of AI agent that people are going to enjoy interacting with. And they don’t have all of the sources of conflict and frustration and misery that go along with human relationships.So one issue is to the extent that people start using social AI much more, they’re going to lose precisely that skill set, which is adapted to dealing with other human beings who haven’t been designed to cater to your specific social preferences.But I take it there’s also then an issue of motivation, where it’s sort of, why would I go out there into the world and spend time with human beings with their own agenda and their own interests who are frustrating and annoying and often insulting, etc., etc., when I could immerse myself in this world in which it’s pure gratification of my social, even my romantic, sexual desires?I think that then connects to something else. When I think about this area of social AI, the thought experiment that seems most salient to me is the experience machine—Nozick’s idea—would you plug yourself into some machine where you’re getting all of the wonderful pleasure that goes along with certain kinds of desirable experiences, but none of it’s real? You’ve just been fed the relevant kinds of neural signals to simulate those kinds of experiences.I think many people think, no, because there’s much more to a meaningful life than merely the hedonic, affective associations or things that are associated with the satisfaction of our desires. We actually want the reality that goes along with satisfying our desires. And I think similarly, when it comes to social AI, the intuition is there’s something similar going on there where it’s okay, you might be getting off socially, romantically, sexually, and so on, but it’s fake. It’s not reality. And so even if you’re the happiest person in the world from a certain definition of what happiness amounts to, which is purely hedonic, there’s nevertheless something deeply troubling about that kind of existence which we should steer ourselves away from.What are your thoughts about that? Do you think that analogy with the experience machine thought experiment makes sense? And do you also buy the intuition that we shouldn’t plug ourselves into an experience machine, so we also shouldn’t plug ourselves into very pleasurable forms of social AI?Henry (48:06): Super interesting. I’m going to tease apart two different threads here. The first idea is this idea that these are just going to be easier alternatives. And I think the useful lens for thinking about that is the idea of superstimuli. This is a term that gets thrown around in a lot of different domains. We hear it in relation to food—that modern junk food is like a culinary superstimulus that basically just gives you far more rewarding signals associated with fat and sugar than anything you would find in our evolutionary environment. And this has sometimes been suggested as the best explanation for the obesity crisis—the fact that basically modern food is just so delicious, it just maxes out our reward centers so effectively that it’s just really, really hard to go back to eating whole grains and leafy vegetables cooked simply.We see the same debate around pornography, the idea that pornography is kind of a sexual superstimulus. I’ve also heard the superstimuli used to refer to things like social media or short-form video. Try reading a Dickens novel if your brain has been fried by a decade of six-second YouTube Shorts.Now we don’t need to relitigate that, but I think all of those debates, this idea of superstimuli, the idea of something that is just far more rewarding than the kind of natural, maybe more wholesome version of it—I think that’s a really powerful lens for thinking about social AI and raises some significant concerns.But I think that’s separate from the second point you raised about the idea that it’s fake, that the actual thing that we value is lacking in these kind of contexts.For what it’s worth, I am far more conflicted on the experience machine, I think, than you. There is a sense in which I think I would be very tempted to take the experience machine, although maybe that has to do with the fact that I’m pretty sure we’re in a computer simulation right now. I’m hardcore into simulationist territory.I also think there are maybe some reasons we should expect our judgments about the experience machine to be maybe skewed. We have this idea of real goods versus ersatz goods, where real goods are going to be more enduring, more reliable. So that might create within us a preference for real goods over fake goods. But of course, in the experience machine, you’re guaranteed—these experiences will keep on going. You’re going to have this dream life in the matrix or whatever, that’s not going to be yanked away. So I think there are intuitions that possibly make us more averse to experience machine type lives than we possibly should be.Dan (50:54): Well, there’s also, just to really quickly interrupt on that point, Henry, sorry to interrupt, but there’s also, I think, as I mentioned earlier on in connection with social AI, this reputational thing going on where I think there’s a tendency to judge people harshly if they choose the experience machine, potentially because we think somebody who’s going to prioritize positive hedonic experiences wouldn’t make for a good cooperation partner or something. I think we’re constantly evaluating, would this be a good leader, a good friend, a good romantic partner, a good member of my group? And if someone seems to suggest that they would prioritize mere hedonic manipulation or however exactly to understand the experience machine, we judge them harshly. And I think anticipating this, then people are inclined to say, “No, I wouldn’t choose the experience machine. I would choose the real thing.”I just also want to really quickly talk—you went over this quickly, but I think it’s very interesting. You said you think we probably are living in a simulation. So this is the classic Bostrom-style argument that says, well, we’re likely to be able to have the technology to create simulated worlds. If that’s true, then there are going to be many, many more simulated realities than base reality. So just statistically or probabilistically speaking, we should assume ourselves to be in a simulated reality.But as I understand that, that’s in and of itself, it’s not obvious to me why that would influence your response to the Nozick experience machine scenario. Because even if you think it’s true that we are living in a simulation in some sense, I take it what’s distinctive about the experience machine thought experiment is it’s not just that you would be living in a simulation. It’s well, for one thing, it’s a simulation within a simulation, but it’s also a simulation which has been tailor-made in a way to satisfy your desires and that feels a little bit different from what I’m assuming you take to be the case when it comes to us living in a simulation. Did any of that make sense?Henry (52:55): Yeah, it makes perfect sense. Okay, here we can get really spicy because I think I take Bostrom’s classic simulation arguments quite seriously, but at the risk of having viewers think I’m completely bonkers, I’m also genuinely—I think there’s a chance I’m already in an experience machine. So this is just speaking for me.Without wanting to get too sidetracked, I think so many features of my life—it’s hard to put this without sounding egotistical, but I just feel like my life has just been absurdly fortunate. I’ve lived in a really interesting time in human history. My life has been blissfully devoid of serious unpleasantness. Not to say that it’s been perfect, but most of the challenges I’ve encountered in life have been interesting, relatively tractable things. Look, here I am. I’m getting to live the life of a Cambridge philosopher at the very cusp of human history where we’re about to explore AGI. It seems like this is the kind of life, genuinely, that I might pick.Whereas I feel like by rights I should have been a Han peasant woman in third century AD China. So I’m slightly joking, but not entirely. I do think there’s a serious chance that at some level I am already—my life consists of some kind of wish fulfillment simulation. So I don’t know, maybe that gives more context.Dan (54:17): Interesting, okay, that’s a spicy take. We should return to that for another episode because that’s fascinating. And actually, it’s also very philosophically interesting. I think you’re right, when you start thinking about things probabilistically in that way, yeah, the fact that you’re having a great life might provide evidence for this kind of experience machine scenario. But I feel like we’re getting derailed from the main point of the conversation now, which I think is probably my fault for double clicking on that point.Henry (54:45): So there’s this question about whether the fact that there’s maybe no one conscious on the other end of the conversation—to what extent does that mean that it fails to instantiate the relevant goods that people care about? To what extent does that make it fake?I think one small thing, I think there’s a perfectly viable position that says, look, a perfectly viable philosophical position that says, look, if there’s no consciousness there, then no matter how much fun these relationships are, they’re not really valuable. I mean, I can sort of see some arguments for that.But maybe I’m also influenced here by the fact that really since my early adolescence, so many of the relationships I’ve had—I’m talking relationships in the broad sense, friendships and so on—have been with people whose faces I never saw. I spent from my early years a lot of time in pre-Discord chat rooms on services like ICQ back in the day, on online video games, massively multiplayer worlds, before the age of streaming video, where I would form these really valuable relationships with people I only interacted with via text. And in some cases, particularly in video games, we didn’t really discuss our personal lives at all. We were just interacting in these virtual worlds.Now you could say, yeah, but there really were people, conscious people on the other end there. And sure, yeah, but I’m not sure how much of a meaningful psychological difference that makes for me in terms of my experience of those things. It seems to me that so many of the goods that we get from relationships don’t consist in this deep meaningful connection with a conscious other but consist in things like joint action—in the case of a video game might be going on raids together, having some fun banter together, discussing politics.I spend a huge amount of time—or not so much these days, back in the day I spent a huge amount of time arguing on Reddit. Would it have made a difference to me if I knew that person I had a really good long political debate with on Reddit, that they were a bot? Well I think it probably would, but I’m not sure whether it should, if that makes sense. It was a valuable discussion for me and maybe it’s just my own prejudice that gets in the way then.Dan (57:04): Interesting, yeah, I definitely don’t have the same intuition. I think if I were to have what I thought of as meaningful relationships and then discover that actually I wasn’t dealing with a person as I understand it—Henry (01:03:47): Yes, okay, if you found out that you were talking to a chatbot online and you thought they were a person, that would be dismaying. You would feel bummed out to some extent.It’s not clear to me that that is primarily to do with lack of consciousness though. In some ways, I think it’s more to do with a loose set of considerations around agency and identity. Talking to a chatbot feels like something like a social cul-de-sac currently. There’s no one going to go away with changed views that they’ll carry forward into discussion with other people as a result of the conversation that we’ve had. It can feel sort of masturbatory in that sense.But I think if you can think about AI systems, if we imagine AI systems as something like more robust social agents—so you might chat to a given social AI one day and that social AI will be able to carry forward any insights it gleans from that conversation into interaction with other people. I don’t know, as you beef up the social identity of these things a bit more so that it’s not just these masturbatory cul-de-sacs, then my intuitions start to weaken a bit. Maybe they can be valuable. If I’m talking to a chatbot that speaks to other people and can carry forward those insights, maybe there is some value.Dan (01:05:06): Yeah, that’s so interesting. I feel like there’s a million different things we could be talking about here. And we should say we’re going to have other episodes where we return to social AI, where we bring on guests and so on.Maybe two things to end on. One thing we’ve already touched on, and I think it connects to what we’ve just been saying, but takes it even further. One commercial use of social AI, at least as I understand it, pretty fringe use, but not a completely non-existent one, is I think you earlier on called them grief bots? Basically people using AI technology to produce a system that—I don’t exactly know how to describe it, but that exemplifies the traits that they associate with somebody, a loved one, a family member, a spouse, a friend who has passed away.I mean, that it’s almost like that’s been cooked up purposefully for moral philosophers because it introduces so much, not just weirdness, but moral complexity. I mean, I take it there’s the general baseline issue, which is you’re forming a relationship of some kind, you’re interacting with an AI system, and that’s weird. And then it’s a weirdness which is massively amplified by the fact that you’re interacting with an AI system, but the relationship in some sense is grounded in the perception that you’re somehow interacting with somebody who’s now deceased.I don’t even know to be honest how to describe it, but I understand some people are doing this. So what’s your take about what’s going on?Henry (01:06:51): Yeah, so I expect griefbots to be one of the big applications of social AI. As you mentioned, it’s relatively niche at the moment. And I think a lot of companies are very scared to go anywhere near this, but I can see that changing quite rapidly.Just for context here, it’s worth noting that chatbots fine-tuned on real-world individuals—there’s a lot of them that suggest they can be really accurate in terms of capturing the kinds of things that people would say, their modes of conversation and so on.So one famous experiment was the DigiDan study. This was using GPT-3, so really primitive language model by modern standards. But a group of people including David Chalmers, Anna Strasser, Matt Crosby and others basically fine-tuned GPT-3 on the works of Daniel Dennett—I’m sure most of your audience know Daniel Dennett, one of the greatest philosophers, sadly died a couple of years ago. This was shortly before his death though that he did this.And then they got Dan’s friends and colleagues to pose questions to both Dan himself and the DigiDan bot. And they generated four responses from the chatbot and they had Dan’s response there as well. And users, Dan’s friends and colleagues, were pretty much close to baseline, close to bare chance, telling which responses were from Dan versus the chatbot.So this is just to emphasize that appropriately fine-tuned chatbots could do a really good job of simulating the kind of things that a person would say in response to a given query.So let’s just imagine that you do have this category of griefbots that can provide an accurate simulacrum of a deceased person. Well, I mean, at the risk of spicy take, I can see lots of really positive use cases for this.I’ve actually even said to Anna Strasser, one of the people who did the DigiDan study, that if I get hit by a car tomorrow, I’ve discussed this with my wife as well, then she should absolutely have my permission to fine-tune a bot on me. And my wife will give her a perhaps lightly edited or curated set of my correspondence, my social media presence and so on, so that then my kids can talk to this simulacrum of me if they choose to.I can imagine there being real value, know, if my son or daughter is like 17 and considering, should I go to law school or medical school? I wonder what my dad would have thought of this. That seems like a really potentially positive use case.So I think griefbots are a really interesting area. And there’s interestingly some studies looking at people’s use of griefbots for therapeutic purposes—if there are conversations you always wanted to have, but didn’t get the opportunity to have perhaps because a spouse or a parent died suddenly, that this could, one phrase that’s used, offer a soft landing for the grief experience. So there are loads of really interesting positive use cases there.But equally, as you say, it’s an absolute minefield. And I think to a lot of people the whole idea of griefbots just feels like something from a Black Mirror—well, literally there was a Black Mirror episode about this. And I think that it raises some fascinating questions about how that changes the nature of the grieving process, how it changes our very concepts of mortality.If someone’s physical body can die, but there’s this sort of echo, digital echo ghost of them that is still around, how does that reshape our views about these things?There’s also some interesting parallels. I had a student who wrote a great dissertation or a great essay about integrating this with the idea of communing with ancestors, which is obviously a really common feature in many different societies, where you might ask your ancestors—we touched on this earlier on—you might ask your ancestors for guidance on difficult questions. Could griefbots be a way of making that into a more concrete experience?There’s a second angle here, which is even spicier and again, will make viewers think I’m even more of a weirdo, which is: could this actually offer some kind of form of immortality or some kind of life after death? Or continued existence after death, I should say.Now, we could do a whole episode on this as well, but for what it’s worth, as someone who is very deflationary about personal identity, I’m big on the work of philosophers like Derek Parfit who say that in some sense the self is an illusion or the persistent self is a constructed self, there’s no deep matter of metaphysical fact about whether I survive or not. I could see a good case being made that in some sense, via an appropriately fine-tuned chatbot, there will be a form of persistence of me through that chatbot that might be relevant to mortality considerations.Dan (01:11:36): Yeah, in some sense. It seems like there’s an issue here, which is using a chatbot to acquire knowledge about what a given person might have thought about a topic. And I take it there are going to be all sorts of questions that arise there. Like to what extent is it going to be reliable as a way of gaining insight into what that person would have thought about a topic?Then there’s a question of using these systems not just to get that kind of knowledge, but to actually have a kind of relationship with the person. And then there’s something over and above that, which I take it you’re referring to then, which is the idea that in some sense, such a chatbot would carry on the identity of the relevant person. And I take it that relationship thing and that identity thing are connected.I mean, certainly I think that’s where the real issues and lots of people’s queasiness arises, right? That’s when it seems like a bit of a leap, at least if we’re talking about chatbots as we understand them today. I can imagine AI systems of the future that aren’t merely getting really good at the statistical pattern recognition and prediction when it comes to bodies of text, but that are doing something more substantial when it comes to replicating the characteristics and traits of the relevant person.But do you really think if you had a chatbot that had been trained on the text that you had produced that it would be in any sense a continuation of you?Henry (01:13:08): Yeah, I mean, potentially. We could do a whole episode on personal identity here. But broadly speaking, being a bit crude here, the Parfittian perspective, Derek Parfit’s view, is that there’s a certain kind of relation you bear to your future self, a certain kind of psychological relation, that can come in varying degrees. And to the extent that we prioritize anything when we’re about survival, this relation R is the kind of relation that matters. And it can obtain to varying degrees.Just as I could suffer a traumatic head injury and my behavior would change in some ways, but not others—that would be a sort of continuation of me in some ways, but not others. I think you could say the same for an appropriately fine-tuned chatbot.Now, as you sort of implied, there will be things that it misses out there. We don’t talk about everything that is relevant to us. There’s more to our identity than just what we say. But again, that seems like a technological problem that, as we move to increasingly multimodal chatbots that can learn not just from what we say online, but how we live in the world, I think you can instantiate this relation, this relevant kind of continuation relation to increasingly strong degrees.But I guess that’s the central point I’d say here: that survival in this kind of Parfittian view is a matter of degree and a matter of similarity and I don’t see why even if it’s imperfect—to the extent that a chatbot can capture really key features of my modes of interaction—that that’s a kind of survival. That’s a kind of continuation.Dan (01:14:39): Interesting. Maybe we could end on this point. So I’ve been thinking quite a bit recently about how advances in artificial intelligence might gradually eat away at human interdependence. When people are thinking about the dangers posed by AI, there are the classic loss of control, catastrophic misalignment dangers that we’ve talked about previously. There are also dangers to do with elites, political factions, authoritarian regimes, the military using advanced AI to further objectives in ways that are bad for humanity.But I think there’s also a category of dangers associated with AI systems doing what we want them to do, satisfying our desires, but in ways that have really knock-on bad consequences. And it’s easy for me to see how social AI might be a little bit like that, in as much as so much of our understanding of the human condition, so much of the societies that we inhabit is bound up with interdependence. We depend upon other people. We depend upon other people for friendship, for labor, for sex, for romance, for art, for creativity and so on.And it seems like a very plausible path when it comes to advances in AI is these AI systems are just going to get better and better at doing everything that human beings do and in fact are going to get better than human beings at doing all of those things, including when it comes to satisfying the social needs in the way that we’ve talked about in this conversation.And to the extent that that’s true and we become more and more reliant on these AI systems and less and less reliant on other people, that kind of human interdependence fades away. I think there’s something—I mean, there’s something disturbing about that just from the perspective of thinking about what it means to be human. But maybe that’s not really a serious philosophical worry. That’s just an emotion.But I also do think, the way I think about it is a lot of the human alignment problem—like how do we align our interests with one another and build complex societies—is precisely this interdependence. Because we depend on other people, we have to care about them and we have to care what they think about us and so on.And it seems to me one of the diffuse risks and long-term risks associated with social AI is precisely that, that as this technology gets better and better, it’s just going to erode that interdependence, which is really central to the human condition.I realize that’s a massive thing to throw at you for the final question, but what are your thoughts about that? And then we can wrap things up.Henry (01:17:15): Super interesting. Yeah, so I think you did this yourself, but I’ll separate out two different concerns here. One is the more philosophical question about, even if this works perfectly, even if we’re all very happy with this future society, has something of value been lost? I think it is a valuable question to ask, but it’s also one that’s hard to answer in a neutral sense. It really comes down to what is your conception of eudaimonia, human flourishing, in a deep philosophical sense.And then there’s—but I felt like that wasn’t the core of your question. You were asking something more about negative externalities, perhaps more about negative knock-on effects. And I think that is absolutely something I’m also worried about basically just because of social media.Now I realize this is a debate where you have your own very well-developed positions, but I’ll just offer a quick parallel of two technologies. One is violent video games or video games in general. I think we’ve probably discussed this before, but back in the 90s, there was massive moral panic around negative knock-on effects. The idea that kids growing up in the 90s playing Doom or GTA would turn into moral monstrosities as a result of being exposed to relatively accurate simulated violence.I don’t think that was a stupid thing to worry about. It just turned out to not be a major concern. It turns out that’s not how the brain works, that we didn’t see massive negative externalities associated with exposure to violent video games.But by contrast, social media, opposite story. I think there was relatively little panic early on in the days of social media, in the early days of MySpace and Facebook. In fact, I think most of the commentary about social effects of these things was quite positive—the idea of bringing people together. There was an interesting debate that’s quickly been sort of consigned to the dustbin, quickly been memory-holed. People, I remember in 2010, people were talking about how social media meant the collapse of epistemic closure, how these epistemic islands would be all beautifully linked up through conversations on social media. And we’d be able to talk to people with different political views from us.And that’s basically not happened, that social media exacerbated some of our worst social tendencies, possibly contributed to echo chambers and so forth. I’m aware that you have a slightly more optimistic view here, but I’m just offering that as a parallel for a case of social technology that I think at least in many people’s view has had significant, largely unforeseen negative consequences.And I think that’s absolutely a legitimate source of worry about social AI. I’m not sure I’d necessarily frame it in terms of dependency or interdependence. I mean, I think different people make different choices about how much they want to depend on others. I don’t think it’s obvious that someone living alone on a ranch in rural Texas or whatever, they grow their own food, whatever, they’re relatively autonomous and independent—it’s not clear that they’re not living a great life. It seems that you can have valuable lives with varying degrees of social interaction and dependency on others.But at the same time, I do think there are possible dangers, very hard to predict, associated with potentially people getting into islands of social activity where it’s just them and their coterie of AI friends and they don’t see the need to interact with others. The kind of subtle influences that could have on things like democracy, on society—there’s absolutely scope for concern.Dan (01:20:52): Yeah, okay. We’ve opened several cans of worms to conclude the conversation. I’m aware, Henry, that you’ve got a place that you need to be. So that was so much fun. So many issues and questions, which I feel like we didn’t really even scratch the surface of, but we’ll be back in a couple of weeks to talk about more of these issues. And then over the future, we’re going to bring on various kinds of guests and experts in social AI and connected issues. Was there anything final that you wanted to add before we wrap up, Henry?Henry (01:21:24): I guess just a couple of quick reflections. Firstly, I think for any young philosophers or young social scientists listening, I think this is just such a rich and underexplored area right now. There are so many interesting issues ranging from griefbots to digital duplicates—models fine-tuned on real-world individuals who are still alive—to issues around de-skilling, dependency, mental health, atomization, loneliness, intellectual property, influence, motivation, persuasion. There’s enough for several dozen, hundreds maybe of PhD dissertations on this topic. So I think it’s just a really interesting and valuable area to work on.And that’s not even getting into the meatier philosophical issues we sort of just touched on briefly around personal identity, what it means to be human, flourishing, the good life. So I just think this is a really valuable area.Also worth quickly promoting that I am a unit editor for an Oxford University Press journal series called “AI and Relationships.” It’s called the Intersections Journal Series and I run a project in there. So if any young philosophers or academics have papers on this, feel free to give me a ping on Twitter or to my email if you’ve got anything you want to publish on this topic because I think, yeah, it’s an area where I’m really keen to start seeing more good research.Dan (01:22:41): Fantastic, yeah. It’s a golden age for philosophy, which is why it’s a little bit strange when you look at so many of the things that philosophers are actually working on. But anyway, that was great. See everyone next time. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.conspicuouscognition.com/subscribe

In this conversation, Dr Henry Shevlin (University of Cambridge) and I explore the complex and multifaceted topic of AI consciousness. We discuss philosophical and scientific dimensions of consciousness, discussing its definition, the challenges of integrating it into a scientific worldview, and the implications of such challenges for thinking about machine consciousness. The conversation also touches on historical perspectives, ethical considerations, and political issues, all while acknowledging the significant uncertainties that remain in the field.Takeaways* Consciousness is difficult to define without controversy.* The relationship between consciousness and scientific understanding is extremely complex.* AI consciousness raises significant ethical questions.* The Turing test is a behavioural measure of intelligence, not consciousness.* Historical perspectives on AI consciousness are helpful for understanding current debates.* Cognition and consciousness are distinct but related. * There is a non-trivial chance that some AI systems may have minimal consciousness.* Consciousness in AI systems is a scientific question, not just a philosophical one.* The debate on AI consciousness is messy and strangely polarising (and often heated) but fascinating and important.Chapters00:00 Exploring the Nature of Consciousness17:51 The Intersection of AI and Consciousness36:16 Historical Perspectives on AI and Consciousness59:39 Ethical Implications of AI ConsciousnessConspicuous Cognition is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.TranscriptPlease note that this transcript is AI-generated and may contain errors. Dan Williams: Okay, welcome everyone. I’m Dan Williams. I’m here with the great Henry Shevlin. And today we’re going to be continuing our series of conversations on artificial intelligence, some of the big picture philosophical questions that AI throws up. And today specifically, we’re going to be focusing on AI consciousness. So could machines be conscious? What the hell does it even mean to say that a machine is conscious? How would we tell whether a machine is conscious? Could ChatGPT-5 be conscious and so on? Before we jump into any of that, Henry, I’ll start with a straightforward question, or what seems like a straightforward question. What is consciousness?Henry Shevlin: So it’s very hard to say anything about consciousness that is either not a complete platitude or rephrasing like consciousness is experience, consciousness is your inner light, consciousness is what it’s like. Those are the platitudes. Or saying something that’s really controversial, like consciousness is a non-physical substance or consciousness is irreducible and intrinsic and private. So very hard to say anything that is actually helpful without also being massively controversial. But probably let’s start with those kind of more platitudinous descriptions.So I assume everyone listening to this, there is something it’s like to be you. When you wake up in the morning and you sip your coffee, your coffee tastes a certain way to you. When you open your eyes and look around, the world appears a certain way to you. If you’re staring at a rosy red apple, that redness is there in your mind in some way. And when you feel pain, that pain feels a certain way. And more broadly, you’re not like a rock or a robot, insofar as we can understand you purely through your behavior. There’s also an inner world, some kind of inner life that structures your experience, that structures your behavior.All of which might sound very obvious and not that interesting, or not that revolutionary, but I think part of what makes consciousness so exciting and strange is it’s just very hard to integrate it with our general scientific picture of the world. And I’ll say in my own case, this is basically why I’m in philosophy. I mean, I was always interested in ethics and free will and these questions. But the moment where I was like, s**t, I’ve got to spend the rest of my life on this, came in my second year as an undergrad at Oxford studying classics. I was vaguely interested in brains and neuroscience, so I took a philosophy of mind module with Professor Anita Avramides. And I read an article that I’m sure many of the listeners at least will have heard of called “What is it Like to Be a Bat?” by Thomas Nagel, and it blew my mind.And immediately afterwards, I read an article called “Epiphenomenal Qualia” by Frank Jackson, which is the article that introduces Mary’s room. And it blew my mind even more. And basically, I’d spent most of my life up until that point thinking the scientific picture of the world was complete. I spent most of my life until that point basically thinking the scientific picture of the world was complete. And you know, there was some stuff we didn’t understand, like what was before the Big Bang, maybe exactly what is time, but when it came to biological organisms like us, we had Darwin, we had neuroscience, it was basically all solved. And then reading more about consciousness, I realized, my god, we don’t even begin to understand what we are, what this is.Dan Williams: Yeah. I think that’s... Let me just interrupt there to flag a couple of those things, because I think they’re really helpful in terms of structuring the rest of the conversation. The first is, when it comes to consciousness, it’s really, really difficult to articulate precisely in philosophically satisfying ways exactly what we’re talking about. You mentioned this classic article, “What is it Like to Be a Bat?” And I think it’s a fantastic article, actually. I’m teaching it at the moment. And one of the reasons I think it’s fantastic is because it does convey in quite a concise way, quite quickly, the sort of thing that we’re interested in.So I’m talking to you, and I assume that there’s something it’s like to be you. Nagel’s famous example is with bats. They are these amazing animals. Their perceptual systems are very alien to ours, but we assume there’s something it’s like to be a bat. So it’s very difficult to state precisely exactly what we’re talking about, but you can sort of gesture at it—something to do with subjective experience, what it’s like to have an experience and so on.And then the other thing that you mentioned, which I think is really interesting, and in a way, it’s sort of disconnected from the machine consciousness question specifically in the sense that even if we had never built AI, there would still be all of these profound mysteries, which is just how the hell do you integrate this thing called subjective experience into a scientific worldview? I mean, there are other sorts of things where people get worried about a potential conflict between, roughly speaking, a scientific worldview and a kind of common sense picture of the world. So maybe free will is another example, or maybe objective facts about how you ought to behave. Some people take that seriously. I’m not personally one of them, but some people do. But I think you’re right. Consciousness feels so much more mysterious as a phenomenon than these other cases that still seem to pose puzzles for a broadly scientific worldview.Henry Shevlin: Also, unlike free will and unlike objective morality, I think it’s very, very hard to say that consciousness doesn’t exist. I mean, it’s pretty hard to say that free will doesn’t exist and painful perhaps to take the view that objective morality doesn’t exist. But these are just very well established positions. And there are some people out there, illusionists, who try and explain away consciousness. Maybe how successful they are is a matter of debate. But it’s very, very hard to just say, like, your experience, your conscious life—nah, it’s not there. It’s not real. It doesn’t exist.Dan Williams: Yeah, right. Actually, I think that’s another nice place to go before we go to the specific issues connected to artificial intelligence. So there’s this metaphysical mystery, which is how does consciousness, how does subjective experience fit into a broadly scientific, we might even say physicalist picture of the world? And so then there are lots of metaphysical theories of consciousness.I’ll run through my understanding of them, which might be somewhat inadequate, and then you can tell me whether it’s sort of up to date. Roughly speaking, you’ve got physicalist theories that say consciousness is or is realized by or is constituted by physical processes in the brain, in our case. You’ve got dualist theories that say consciousness is something over and above the merely physical. It’s a separate metaphysical domain, and then that comes in all sorts of different forms.You’ve got panpsychism, which is, to me at least, strangely influential at the moment, or at least it seems to be among some philosophers, that says basically everything at some level is conscious, so electrons and quarks are conscious. And then you’ve got illusionism, and I suppose probably the most influential philosopher that’s often associated with illusionism would be Daniel Dennett. I understand that he had a sort of awkward relationship to the branding. But there the idea is something like, look, we take there to be such a thing as consciousness. We take there to be such a thing as subjective experience. But actually, it’s kind of just an illusion. It doesn’t exist. Is that a fair taxonomy? Is that how you view the different pictures of consciousness in the metaphysical debate?Henry Shevlin: Yeah, I think that’s pretty much fair. A couple of tiny little things I’ll add. So panpsychism maybe doesn’t completely slot into this taxonomy in quite the way you might think. Because a lot of panpsychists would say, no, we’re just physicalists, right? We believe that everything is physical. There is only the physical world. But consciousness is just a basic physical property associated with all physical stuff. So that’s, for example, Galen Strawson’s view. He’s got a great paper called “Real Materialism.”But there’s another way you can be a panpsychist, which was the Bertrand Russell sort of view and the view developed by people like Philip Goff that says the underlying nature of reality is neither physical nor mental. This is also sometimes called neutral monism. Also a view associated with Spinoza, the great historical philosopher. So underlying reality is neither physical nor mental, but everything has physical aspects and mental aspects. Spinoza’s version of this is called dual aspect theory.So the other thing, other little qualification worth mentioning is if we go dualist, there is one very important, I think, pretty defining distinction. Okay, if consciousness is not physical, does it interact with the physical? Descartes, of course, famously thought that consciousness was a substance or the mind was a substance that interacted with the physical, which immediately runs into some really messy issues. But the other predominant form of dualism is epiphenomenalist dualism, where basically everything the world is made of is physical, but through complex or strange or basic metaphysical processes, there’s this conscious layer that sometimes emerges from reality, but doesn’t itself drive any causal processes. But that’s the basic metaphysical picture.Dan Williams: Yeah. And I think that thing you said there at the end about emergence is really important. And I think many people coming to these debates to begin with, they don’t draw the distinction between a view that says consciousness in some sense just is or is constituted by physical goings on in the brain, for example, and a view that says consciousness emerges from physical goings on in the brain. They sound like they’re similar claims, but actually that latter claim about emergence is really very different from the ordinary physicalist view, because you’re basically saying there’s something over and above the physical. It might be related to the physical in a law-governed manner, but it’s a different kind of thing. Is that right as a distinction?Henry Shevlin: Absolutely, yeah. And I think it’s one of the ways of problematizing the kind of knee-jerk physicalism that I think a lot of people have, or the kind of physicalism that doesn’t see consciousness as problematic. It’s like, yeah, consciousness emerges from the brain. It’s like, okay, but what we ultimately need is a theory that tells us how pain, orgasms, perceptions, everything, all of your experiences, how at some level they just are neurons, right? There’s nothing over and above neuronal activity or computational processes or information. The theory actually has to ultimately bottom out in neurons. If you say it emerges, and without being identical to it, without being identical to the brain stuff, that still does not neatly fit with the physicalist picture.Dan Williams: Right, right. Yeah, and it’s very unsatisfying in a way, or at least I find it unsatisfying just to have brute emergence. By the way, I love that your first two examples there were pain and orgasms as the most salient examples of conscious experience that come to mind. It just occurred to me, something that we might be taking for granted as philosophers who know this literature well—you know it a lot better than I do, but we’re both immersed in it to some degree—which is we said consciousness poses a problem for, or at least a puzzle for a scientific worldview, we’ve gestured at why that is. Maybe it’s helpful to return to something you said, which is one of the things that you found most influential in your own philosophical journey was this article by Frank Jackson, which touches on what’s often called the knowledge problem.So this is basically a thought experiment where you’re asked to imagine a color scientist called Mary who is color blind, or perhaps not color blind—in fact you can correct me if I’m misdescribing the story—but basically is an expert in color science but inhabits a black and white room, so she’s never actually seen, for example, the color red, but we’re supposed to imagine that she knows everything there is to know about the neuroscience, the neurophysiology, the physics of perception, of color perception, but she’s never actually herself experienced the color red. So she knows all of the physical facts, the facts about neurobiological mechanisms and so on, the facts about how light interacts with the visual system, knows all of the physical facts, never actually experienced red.And then Jackson asks, okay, suppose that one day she does, let’s say, leave this purely black and white environment and encounters the color red for the first time. He says, well, it’s obvious that she’s going to learn something new, which is what it’s like to experience the color red. But by stipulation, she knew all of the physical facts already. So the fact that she’s learned something new suggests that the physical facts don’t exhaust the facts. Is that, roughly speaking, the thought experiment?Henry Shevlin: Yeah, that’s exactly right. So yeah, in Frank Jackson’s original framing, she’s just in this black and white room, but typically when teaching undergrads, undergrads say, can’t she just rub her eyes really hard or bang her head or find some other way to generate phosphenes or something like that. So the way I usually frame it for pedagogical purposes is she’s got a condition called cerebral achromatopsia, which is a real condition, which is a form of neural deficit that means you can’t experience color. And then one day she gets her neural deficit fixed and she’s like, my god, now I know what all these colors are like. I spent my whole life researching color but I never knew what color actually looked like.But yeah, that’s just my preferred framing but I think you’re absolutely right. The way the actual structure of the argument is by hypothesis, she knows all physical facts relevant to color vision at the neural level, the level of optics, the level of physics. She learns a new fact when she gets her vision fixed or leaves the room. Therefore, there are some facts that are not physical facts, or the domain of fact is not exhausted by purely physical facts.And the interesting thing about this, I think—I mean, the argument is basically there in the Nagel article as well, maybe in slightly less clear form—one way of capturing one of the points that Nagel makes in that article is, look, no matter how much we ever knew about bat behavior, bat brains, about bat evolution, that’s never going to tell us what it’s actually like to see with ultrasound for bats. Presumably there is something it’s like, this phrase that Nagel popularizes in that article, but you’re never going to get to that purely from a neurophysiological or optical understanding of the world.Dan Williams: Yeah, I mean, from what I remember in the Nagel paper—I reread it relatively recently—but I think Nagel says at the very least, this suggests that there’s this fundamental what philosophers would call epistemic challenge in the sense that even if you’re committed to the view that ultimately everything is physical, nevertheless, what these sorts of thought experiments and these sorts of examples demonstrate is we, at least at present, can’t even understand how that could be true.Whereas I think Frank Jackson makes a more directly metaphysical conclusion, which is this thought experiment, at least when he was initially publishing these ideas—I think he’s later changed his mind—but this thought experiment demonstrates physicalism to be false. Maybe it’s worth also noting that another thing that philosophers will often bring up in this context is the zombie thought experiment, I guess, or the imagined situation where you’ve got a system which is behaviourally, functionally identical to a human being. So exactly as you are right now talking to me, we’re having a conversation about consciousness, and yet there’s just nothing going on inside. There’s nothing it’s like to be that system, no subjective experience.And the point is not supposed to be that people actually are zombies, but the fact that we can allegedly coherently imagine that situation without running into any kind of logical contradiction demonstrates that consciousness is something over and above the merely physical, functional, behavioural. Roughly speaking, that’s the kind of idea. Before we move on to how these sorts of metaphysical puzzles and issues connect to AI, did you have anything else you wanted to add about any of that?Henry Shevlin: Yeah, two very quick things. So just to give a name to the difference you are isolating between Nagel and Jackson, the difference between epistemic versus metaphysical arguments, and to round out our trifecta of guaranteed readings for class one of a philosophy of mind or consciousness session. So this is the idea of the explanatory gap, an idea coined by Joe Levine, who basically points out that, look, even if you think from inference to the best explanation, our best model of the world is that everything is ultimately made of physical matter and energy, right now, we have no idea how to integrate consciousness in that picture. This is the explanatory gap. And you can see that as a challenge that a good theory of consciousness, a good physicalist theory of consciousness should close. It should fill out and make obvious why consciousness exists and why things are the way they are.And I think this is actually my preferred way of framing the zombie arguments, right? You can see the zombie argument as a challenge to physicalism that says, all right, give me a theory of consciousness that shows why zombies are logically impossible, right? In other words, why it is impossible, a contradiction in terms, which is what you ultimately need for a complete physical theory, to talk about beings who are not just behaviorally, but microphysically identical to us—beings that are microphysically identical to us that are non-conscious. That seems like something I could imagine, like something, someone’s walking around but there’s no lights on on the inside, so to speak. A good physicalist theory of consciousness, a complete physicalist theory, should close the explanatory gap and show why that is not just implausible but like an actual logical contradiction.Dan Williams: Yeah, yeah, and I think that’s nice, and I think it connects to, in some ways, some actually quite complex philosophical ideas. The more you dig into this, you realize actually you need access to a whole complex philosophical machinery to really even articulate the core ideas and positions and debates and so on. But I think we’ve said enough to sort of frame this and give it context. Maybe now we can move on to artificial intelligence.Before we get to the present day, I think it’s worth maybe just taking a little detour through some of the most important historical developments in thinking about artificial intelligence and consciousness. So famously Alan Turing, who’s a pioneer in computer science and digital computers and AI and so on, he published an article, I think it was 1950, “Computing Machinery and Intelligence.” And the opening line of that is something like, we’re going to consider the question, can machines think? And then he goes on to propose what’s subsequently referred to as the Turing test, which roughly speaking is, could a machine via conversation, text-based conversation, trick us into thinking that we’re talking to a human being? And if it could, then it passes the test. It’s imitated human behavior and Turing seems to kind of say that if a system did in fact pass that test then there’s some sense in which we could say that we can describe it as a thinking entity or we can describe it as intelligent.I’m being sort of vague there because I think the article itself is a little bit vague in terms of how it expresses these different ideas, which is understandable in a way because obviously this was written in 1950. So I think there are kind of two issues there. One is that Turing was focused on, at least on the surface, thought and intelligence in the first instance, whereas we’ve been talking about consciousness. And then the second facet of this is Turing was proposing a kind of behavioral test for establishing whether a system, whether a computing machine can accurately be described in these sorts of psychological terms.Let’s take the first of those first. This issue of intelligence, understanding, thought on the one hand and consciousness on the other. My sense is many people bundle these two things together, but you might think there’s an important distinction between them. Where do you come down there?Henry Shevlin: Yeah. So I think one distinction that maybe is present in pretty much all debates in cognitive science, but is less present in public understanding of these issues is the distinction between cognition or the mind on the one hand and consciousness on the other. So basically, for the last hundred years or so, it has been not uncontroversial, but widely recognized that there are at least some forms of unconscious cognition. Whether you think that’s Freudian unconsciousness—we have unconscious drives and motivations—or the more cognitive neuroscience view of unconscious. Think about the stuff that happens in early vision or things that happen in linguistic processing or for that matter, we’ve got a rich literature now on unconscious perception. All the various ways your brain could register and interpret representations without thereby giving rise to any conscious experience.And I think our language around the mind and cognition doesn’t need to be conscious in order to be useful. In particular, I think there’s another route to understanding what mental states are that doesn’t involve consciousness, which is they’re in some sense representational states. And as soon as we need to start talking about things like what you’re perceptually representing or what your unconscious beliefs are or what your unconscious motivations are, you’re talking about your brain managing and interpreting representations. At that point, as far as I’m concerned, you’re in the domain of the psychological, you’re talking about the mind, and a lot of that stuff happens unconsciously.So I think, yes, we shouldn’t assume at the outset, certainly, that all mental states are conscious. So I think it’s entirely coherent to say, for example, I think LLMs think, I think LLMs understand, I think LLMs reason, but I don’t think LLMs are conscious. And I think we can keep those two apart, although of course, different theoretical positions are going to lead you to say, actually, maybe the only real form of understanding is conscious understanding. For example, that’s a position you can adopt.Dan Williams: Right, right. Yeah, that’s really interesting. I definitely—it seems to be the case that people are kind of comfortable, well, they’re comfortable talking about artificial intelligence, which suggests that they don’t really object, for the most part, to the idea that these systems are at least in some sense intelligent. I think there’s also a tendency for people to use terms like thought and reasoning quite easily when it comes to using these systems. But people, I think, are much more suspicious of the idea that we should think of them as conscious systems when it comes to something like ChatGPT-5 or Claude or Gemini. And I think that’s kind of in line with basically what you’re saying, that there’s an important distinction between these two things. They might go together in the human case in the sense that we are intelligent animals and we’re also conscious animals. But in terms of thinking about this issue carefully, you do need to draw a distinction between them.Henry Shevlin: Absolutely. I’d also add that I think this is something that’s really clear in any kind of comparative cognition. So if we’re looking at non-human animals, okay, maybe we all agree that chimpanzees, dogs and cats are both conscious and have a mind. But there’s a really rich field of insect cognition. People work on cognition in quite simple invertebrate organisms, even more simple than insects. There’s a growing field, believe it or not, of plant cognition that tries to understand plant behavior in cognitive terms and even—and so actually really quite interesting research on bacterial cognition, cognition in microbes.Now, we can debate whether that’s really called cognition or whether it’s better understood in more basic terms. But the point is you don’t need to decide in advance if a given animal species is conscious in order to do useful cognitive science and comparative psychology about its behavior and its internal structures.Dan Williams: Yeah, that’s interesting. It reminds me actually, when I teach AI consciousness, I would often start by putting some images of things up on the PowerPoint and asking people for a kind of snap judgment of whether it’s conscious or not. So it’s like, here’s a chimpanzee—conscious. Of course it’s conscious, people think. Here’s a squirrel. And then I’ll do things like, here’s bacteria or here’s a tree or here’s the entirety of planet earth. And when you get to those sorts of cases, I think people just fill up either it’s not conscious or they’re in this complex situation where they’re not quite sure what the question even means.But funnily enough, when I then put up the logo of ChatGPT, and I realize this is not a representative sample necessarily, but I think basically always the response is no way is that conscious. And we should get to that later on because I think that’s an interesting intuition and I think you’ve also got views about how widespread that intuition is and how we should expect that to develop in the future. Maybe before moving on from this then, have you got any thoughts about the Turing test? Because it has been very, very historically influential. So we should probably say at least a couple of things about it before moving on.Henry Shevlin: I think it’s a beautiful paper and it’s very accessible and I’d encourage everyone to read it. You don’t need any particular specialization to read it. And it’s just amazing how many arguments Turing successfully anticipates. And I quote that article on so many different issues. He talks about how even with the very simple computers he was using at the time, they regularly do things that he wasn’t expecting them to do. They regularly surprise him. He’s addressing this idea of a computer can never do anything except what it’s been programmed to do. It can never surprise us. And he’s like, no, of course. Just because you program in the initial instructions doesn’t mean it can’t surprise you in all sorts of interesting ways.He also does—you’re right that it’s framed around this question of can machines think. But he explicitly addresses consciousness or the argument from consciousness, as he calls it. And he makes this slightly behaviorist move where he basically says, this is at least a lightweight behaviorist move where he says, look, none of us know whether anyone else is conscious, right? He calls it a polite convention, right? What we do is we look at each other’s behavior and assume, well, look, it’s behaving in relevantly similar ways to me, I’m conscious. So I guess he’s conscious too. This is what he calls the polite convention.So he says, basically, look, the question is, under what circumstances, if we’re worried about consciousness, should we extend that convention to machines? And it seems like if they can successfully pass as humans—I’m doing some slight retro interpretation here, but I think one way of seeing the argument is if they can, to the extent that machines can do the same kind of behavioral capacities, relevantly similar behavioral capacities as humans, then we should extend the polite convention to them as well.Dan Williams: Yeah, and I think just to connect that point about, can they exhibit the appropriate behavioral capacities? So there’s a kind of claim there about consciousness, but obviously connected to him proposing the Turing test is this broader view, which is that there must be some kind of behavioral test of whether a system can accurately be described as—his primary focus, as we’ve said, is on thought and intelligence, but also potentially consciousness as well.My sense is most people these days who are experts in the field, they think that the specific test that he proposed was in a sense not very good, just because you can get systems that imitate human beings in such a way that they can trick people into attributing intelligence without actually being all that intelligent. We should also say that, I mean, my understanding of this, and I wouldn’t find this surprising at all, is that state of the art large language models can basically pass pretty stringent versions of the Turing test, as I understand.Henry Shevlin: Yeah, yeah, so there have been some large scale—replication isn’t quite the right word—instantiations of the Turing test. One from earlier this year that involved five minute dialogues with state of the art language models, found that state of the art language models were actually judged to be human more often than their human competitors. But the slightly tricky thing here is that I think the Turing test is not a single highly defined experimental set of parameters, right? Because you can vary so many different dimensions. How long do you get to talk to the person for, for example, right? Are language models allowed to be deliberately deceptive and say things that you know to be false? So there’s a whole bunch of different rules parameters you can vary. So there’s not going to be like a single moment where it’s like, amazingly, machines have finally passed the one and only Turing test, right? There are lots of different variations of it.And there was a long running prize called the Loebner Prize. I think it started in the 90s and ran for about 20 years, but basically pitted state of the art chatbots against humans in an instantiation of the Turing test. So a lot of people dismissed it as very showbiz rather than real science, but it led to some interesting journalism and some interesting chatbots that were developed. But I think the last one was held before, certainly before the release of ChatGPT. I think it was about seven or eight years ago. But yeah, so people have attempted to actually run Turing type tests for many years.Dan Williams: Yeah, yeah. Okay, before we turn finally to state of the art AI and these deep questions about AI consciousness, maybe we should also just briefly touch on another kind of argument that I sort of hate, but I think has been quite historically influential, which comes from John Searle and it’s the Chinese room thought experiment. Very roughly speaking, Searle asks us to imagine a situation in which you’ve got somebody in a room. They don’t speak a word of Chinese, they don’t understand any Chinese. They’re receiving Chinese text from outside of the room. They are then consulting a sort of instruction manual, maybe a set of if-then instructions, about what to do when you’re given certain kinds of inputs. And then in response, they are producing, sending outside of the room, certain kinds of outputs according to this set of instructions.And so he says, look, you could imagine the instruction book being designed in such a way that someone just mechanically following that procedure would, from the perspective of somebody outside of the room, come across as coherent, intelligible Chinese conversation. But by stipulation, the man inside the room doesn’t understand anything about Chinese. And then the move is something like, well, what’s going on within the room is in some ways relevantly similar to how a computer or a computing machine works. So what this is supposed to tell you is that doing whatever it is that a computer does in and of itself won’t produce genuine understanding. Is that right? Is that a fair representation of it?Henry Shevlin: Yep, that’s a very good summary. Maybe just a couple of small points. So just to add extra clarity for listeners, the person in the room is not using a phrase book and translating from Chinese into English and back into Chinese. They have no idea what any of the characters mean. They just have basically a lookup table that says if you see pictures that look like this, if you see characters that look like this, respond with these characters in return. So the idea is there is zero semantic understanding happening inside the room. However, the room as a system seems to produce this appearance of understanding Chinese. And that is the core claim that you can have this appearance of understanding without any real semantic understanding. Hence, the Turing test is inadequate, is insufficient as a test for genuine understanding.And the other thing to mention is that in the article or the chapter where Searle lays out this argument, “Minds, Brains and Programs,” he’s very much focused on understanding, but in his later work, he connects this to consciousness quite explicitly because his whole theory of what semantic understanding is, of what some real semantic content is, is that it’s ultimately grounded in consciousness. So there’s sometimes a little bit of slippage, but when people shift between understanding and consciousness, it’s very unclear to me that understanding requires consciousness. I don’t think it does. But it’s very clear for Searle, they are two sides of the same coin.The other thing I’d really emphasize about the Chinese room, two things I’d emphasize, because it is probably worth noting—John Searle sadly died two weeks ago. One of the most influential philosophers of language and obviously the Chinese room was massively influential. Due respect to John Searle, but I will say it is one of my least favourite arguments in philosophy. I think in particular, it’s one of these arguments that’s easy to fit, nod along to without thinking through the details.So just to give one little example of how it’s probably a lot more complicated than it sounds. Imagine I say to you, imagine we’re having a conversation and I say, “Hey Dan, how are you?” You might respond, “Not bad Henry, how about yourself?” If I say exactly the same thing again, “Hey Dan, how are you?” You might say, “Did you not hear me the first time? I’m fine.” If I say it again, you’d be like, “I’m sorry, is there a problem with the line?”That’s a very simple example, but it illustrates the fact that language does not work in a series of neat input-outputs, but you actually need to keep track of the entirety of the conversation that’s gone before. So given this kind of combinatorial complexity, it’s a classic combinatorial explosion in even the most basic sentence, right? The kind of scale of the lookup table that would be required for managing even a basic conversation would exceed very quickly the amount of bits in the entire universe.This is why you can easily say sentences that have never been said before. If I say, “The elephant looked forlornly at the cheese grater,” I can almost guarantee no one has ever said that exact sentence before because all natural languages have this insane combinatorial complexity. So that, I think, makes the idea that you could ever build a Chinese room that literally consisted of a lookup table—that makes it very implausible, as in a lookup table that consisted of just a set of one-to-one correspondences for every possible conversation and every possible extended conversation. So that’s one point I’d emphasize.A second point is that ultimately, and I think we might come back to this in a second, it’s an argument from intuition. The argument is saying, do you reckon a system like this is genuinely understanding or in later versions is genuinely conscious? And I think basically what a lot of the argument is designed to do is designed to promote this feeling of disquiet or unease or skepticism that a system like that would be conscious. But if we’re approaching consciousness like a genuine scientific problem, for example, then it’s very questionable whether our intuitions have any epistemic or evidential weight at all. I mean, how are we supposed to know if this incredibly exotic, nomologically impossible to build machine—why should we trust our intuitions about whether such a system would genuinely understand or be conscious? I mean, there are theories about the semantics of our mental vocabulary about what words like conscious and understand mean that maybe give our intuition some weight, but you’ve got to do a lot of work to show why our intuitions would have any validity for these kind of cases at all.Dan Williams: Yeah, no, I’m completely with you there. And it is maybe also worth saying, I actually don’t think this is that relevant, but it is worth saying when Searle was writing this, it was very much the era of what’s called good old fashioned AI, good old fashioned artificial intelligence, where the thought was intelligence in machines is rooted in a certain kind of rule-governed manipulation of symbols according to some program, which you might think very loosely maps on to what he was imagining in terms of a man following instructions in this lookup table.Even there, I don’t think it makes any sense or is coherent. But these days when we’re thinking about state of the art AI, it’s really, for the most part, a completely different approach rooted in neural networks and the deep learning revolution where you’ve got these vast networks of, in state of the art AI today, hundreds of billions of neuron-like units and trillions of connections between them. And in terms of our actual examples of things which can produce conversation and hold conversation, that’s the kind of underlying architecture that we’re dealing with. Obviously, I’ve described that in a very simplified cartoonish way, but it’s so alien to anything that could even be confused on a foggy night for a man following an instruction book as imagined in Searle’s Chinese room thought experiment. Okay, let’s move on to state of the art AI now.Henry Shevlin: Cool. Sorry, one quick thing before we do, if I may, because I think on that last point you mentioned, one of the most exciting conversations, one of the most striking conversations I’ve ever had, this was with GPT-4, was discussing the Chinese room with it. And just to give the full context, I was talking out loud, I was driving in my car, having an out loud conversation with GPT as I often do, and I was talking about this combinatorial explosion problem, the fact that in order to build any kind of nomologically possible version of the Chinese room, you would need something like a memory system to keep track of prior interactions. And as soon as you start to fill this out with any kind of remotely plausible architecture, like LLMs, rather than just this pure lookup table where everything is perfectly coded in advance, here is exactly what you should say in this instance—something a little bit more dynamic. At that point, I think our intuitions just get a lot murkier.Anyway, I said this to ChatGPT, I was ranting and it replied out loud. I’ve got the quote here because I saved it. “You’re spot on, Henry. Searle’s thought experiment often gets simplified. But when you dig into the details, those rule books would have to be incredibly complex to account for context, syntax, previous conversation and so on. Essentially, the rule books would have to be some form of state or memory to handle a genuine conversation in Chinese or any language, really. So in a way, the rulebooks would resemble a state machine, keeping track of prior interactions to generate a meaningful current response. If you look at it this way, it starts to sound a lot like the algorithms that power language models, like… well, me.”Dan Williams: Wow, yeah.Henry Shevlin: And it said it with that exact intonation. And I almost stopped the car and I was so stunned at this seeming moment of—it looks a little bit like self-awareness because I should stress in this conversation, we had not discussed language models at all. We were just having a classic conversation about philosophy of mind.Dan Williams: That is interesting. Okay, well, let’s bring it to these current systems like ChatGPT-5 or Claude or Gemini, state of the art large language models. I mean, in some ways they’re more than large language models as that concept was understood a few years ago because of all of these other aspects which they now involve, to do with the post-training that they receive and the multimodality of these systems and so on. But focusing on these systems, let’s return to this question of, okay, we can talk about what the test should be, but do you think there could be a purely behavioral test, one which doesn’t focus on the actual mechanisms by which a system produces that behavior, that could tell you whether or not that system is conscious?Henry Shevlin: Oh my gosh, that’s a very tricky question. I think in short, given dominant assumptions about what consciousness is in consciousness science, the answer I would say is no. Because basically any behavioral test you’re going to adopt is going to have to come with certain metaphysical assumptions. And there is no straightforward or even probably possible way to test those metaphysical assumptions.So let’s just take a view that a lot of people hold, which is that only biological systems can be conscious. That just as a matter of basic metaphysical fact, only systems that have metabolism that are alive can be conscious. No matter how good the behavior is, how complex the behavior is, even if we basically produce a perfect simulation of a human brain down to simulating individual neurons or even at the sub-neuronal level, that’s still not going to give rise to consciousness. So that is a perfectly possible metaphysical position you could hold. It’s one that I think is—that I don’t agree with. But equally, I can’t give like a clear refutation against it, right?And I think there’s a whole bunch of different metaphysical assumptions you need to make to say anything about machine consciousness, its presence or absence or possibility. And those do not seem to be testable, number one, and two, they’re massively contentious. So you can propose various tests, but they’re only going to make sense given certain metaphysical assumptions.Dan Williams: Yeah, okay, that’s good. Let’s actually—I mean, this is not necessarily where I was expecting this to go, but I think it’s actually really interesting—which is that position you’ve just mentioned that says there’s some kind of essential connection between the fact that we’re biological systems and maybe we’ve got a certain kind of material constitution, a kind of carbon-based material constitution and consciousness. I find this view baffling honestly, but there are very, very smart and intelligent people who argue for this view.Maybe we could say something about what this view is positioned against, which is an alternative perspective, often connected to what philosophers call functionalism that says, roughly speaking, psychological states, including those connected to consciousness, should be understood functionally in terms of what a system could do. And if you think that, then you’re going to think psychological states are substrate neutral. So it might happen in the case of human beings that how we do the things that human beings do, like perceiving and imagining and reasoning and deliberating and so on, it might happen to be the case that those functions are performed by ultimately carbon based matter. But if you could build a silicon based system that could perform the same functions, then it would be accurate to describe that system in this psychological vocabulary of perceiving and understanding, and also potentially experiencing, if you think that you’ve got a functionalist view of experience.Now, I find that view, I mean, as with philosophy, it’s difficult often to precisely articulate exactly the content of the appropriate formulation of that view, but I find the intuition very, very plausible. I find it weird the idea that if you could build a system that’s functionally identical to a human being, but it’s just made of different stuff, then we should deny that that system has consciousness. But as we’ve said, there are some smart people who disagree with that, who think that no, consciousness is essentially connected to the fact that we’re living systems with a carbon-based substrate. Why do they think that? So what’s the motivation for this biology-centric view of consciousness?Henry Shevlin: So I should say I’m probably not the best person to steelman this view because frankly I’m also very sympathetic towards functionalism. Maybe one thing I’ll just quickly say to slightly problematize functionalism before we get into motivations here is it’s very hard to know—so you say, if a system does the same function, it’s functionally identical to humans, right? But there are different levels of granularity there that we can talk about. So I presume probably you would say that a system that’s functionally identical to humans just at the level of behavior, but works completely differently on the inside—it’s at least conceivable that that system is not conscious, right?So you might say, okay, well, behavior is not quite enough to guarantee that the system is conscious in the same way that we are, at least. So we need to move below the level of function to something more like architecture. So people talk about microfunctionalism. Maybe the system needs discrete areas of like perception, maybe it needs something more like working memory, but you can still say, but how different could the system get while still being conscious, right? So functionalism is—I think I’m very sympathetic to some form of functionalism, but actually spelling out how similar a system would need to be in terms of its internal organization, in terms of its cognitive architecture is a really, really tough challenge, unless you just grasp the nettle and say, no, look, if it’s behaviorally identical to a human, that’s all that matters, right? So that’s the most extreme form of functionalism in one sense. So functionalism is—it’s not a simple slam dunk.But as to why someone might think that, that the biological view that only living systems can be conscious, I think by far the most persuasive and sympathetic defender of this kind of view currently is Anil Seth. I think he’s written some great work on this. I’ve just written a response to him in Behavioral and Brain Sciences. And he says, look, there’s all sorts of very, very complex features of biological systems, stuff like they maintain homeostasis with their environment, they’re self-propagating. You have sequences of biological processes that he interprets through Friston’s free energy principle, which you definitely don’t want to get into now. But basically, biological systems do a hell of a lot more than just produce sentences of English or accomplish tasks, right?So the idea that those are the only things that matter for consciousness is to beg massive theoretical questions, right? All we know for sure is that systems like us are conscious, and the fact that we can produce text and we can produce verbal outputs and accomplish goals is not even the most interesting thing about us. There are so many other relevant facts to our constitution, to the kind of beings that we are, all this kind of fine-grained biological stuff that really is critical to the kind of things that we are, and we should therefore have at least some reason to think that it might be instrumental in consciousness. As I say, I think we should get Anil on the show really, because I think he’ll do a much better job of steelmanning it, and I’m sure he’d love to come on as well.Dan Williams: We should do, yeah. Yeah, no, he definitely would, yeah. Yeah, just quickly on that point, I mean, I think this is a bit nerdy and getting into the weeds of things a little bit, but it is an important distinction to make philosophically, which is I think there’s a position that says something like, we should be skeptical that it will be possible in practice to build silicon-based systems that could in fact perform all of the complex functions that our biology can perform. And it’s important to distinguish that view from another view that says even if you could build a system that was functionally identical, it wouldn’t have consciousness.Those are two very, very different views. They might seem like they’re the same, and I think people often go back and forth between them. But they are different. The second one is making a much stronger metaphysical claim. The former one is ultimately an empirical claim about the capabilities, the capacities that you can get from certain specific forms of matter. And I’m not entirely sure which one Anil Seth is opting for, but I find, to be honest, the empirical claim about the limitations of, let’s say, silicon-based systems—I find that interesting and quite implausible, but I find it more plausible than the other claim, which is that even if you could replicate all of this stuff functionally, it wouldn’t be conscious.Henry Shevlin: Yeah, I think the first challenge, first claim is basically an engineering challenge, right? It’s like saying, I don’t reckon you’ll be able to build something with a full range of human cognitive and behavioral capabilities just using silicon. There’s too much about our specific architecture that really matters for what we can do. And I mean, this has actually gotten me—it’s closely related to the ongoing debates about, for example, whether the whole architecture of LLMs can scale up to something like general intelligence. You might say, this is a source of current controversy and debate. Maybe we need to go back to something that is much closer to brain-inspired AI. Maybe, in fact, we need to be using literal biological neurons if we want to build AGI. It’s not a position I find super persuasive, but it’s an empirical question. It could turn out that just silicon architectures—and certainly silicon architectures like transformer models are just never going to give us the full range of behavioral capabilities that humans have. But yeah, it’s 100% an empirical question.Where I really start to disagree though is when people say, well yeah, even if you could do that, exactly as you say, it still wouldn’t be conscious. Because at that point you’re really entering the domain not of engineering or science, but of metaphysics.Dan Williams: Yeah, I mean, just to sort of double click on that point, I mean, not that I’m a great expert in this area or anything like it, but my sense is not only is it not the case that we’ve got certain kinds of material properties that means we’ll be able to do things that silicon-based systems won’t be able to do, I think rather it’s almost the opposite situation where I think we’ll be able to build forms of intelligence with silicon-based systems which are far more complex and impressive than the sorts that you get with carbon-based systems. But that’s a whole complex conversation and a bit of a digression. Let’s go back to... go on.Henry Shevlin: Although I will say, I’ll just add one other thought here, which is, you sort of signposted quite helpfully an important distinction that gets blurred between the engineering version of the challenge to AI consciousness and the metaphysical version of the challenge to AI consciousness. But another line that sometimes gets blurred is the difference between the view that AI systems can’t be conscious at all and the view that AI systems can’t be conscious in exactly the same ways that we are.I think the second claim is a lot more plausible. It may be that the precise qualities of pain or orgasms that humans have, that they would be very hard to instantiate in any purely silicon-based architecture. Or even if you did, you’d need to basically build an entire micro-scale functional model of a human mind or something like that. I find that view very plausible. So I think if AI is conscious and has a different architecture from ours, which it’s almost certainly going to do, then yeah, I imagine its conscious experience will be very different. But that is a move that I think sometimes gets glossed over, going from the claim that AIs won’t be conscious like us to AIs couldn’t be conscious at all.Dan Williams: Yeah, and that’s such an important point. I mean, just to return also to what we were talking about earlier on, bats are not conscious in the way that human beings are conscious. They’ve got a very different, presumably a very different kind of set of conscious experiences, and yet we don’t think that means that they’re therefore not conscious. And I take the point that there might be even greater differences between the consciousness of the kinds of machines that we’re potentially building and us, but I think the point still stands.You mentioned consciousness science, and I realize this is a huge can of worms, and it might be that we’ll need to do a whole other episode on this to really get into the weeds of this. So there are metaphysical views about what consciousness is in some very general abstract sense that we’ve already touched on. And then there are views which you find in neuroscience and psychology about the sort of—what’s the appropriate theory about consciousness, which presumably might be consistent with different metaphysical interpretations of the theory. What’s your sense of the big players when it comes to that area of like the theories of consciousness in that more specific non-metaphysically committal sense?Henry Shevlin: Yeah, so this is a really important distinction between metaphysical theories of consciousness, like physicalism, dualism, and so on, and scientific theories of consciousness. Just to add a little bit of autobiographical detail here. So I spent four years banging my head against metaphysical problems in consciousness, and then was lured away into philosophy of cognitive science towards scientific approaches to consciousness, which seemed to me to be potentially a lot more—I won’t say interesting, but more fruitful or productive.Now, to be clear, these are not trying for the most part to answer the same questions as the metaphysical theories. These are not trying to answer, to solve the hard problem, the problem of why there is consciousness at all. Instead, they’re for the most part trying to understand which kinds of brain dynamics or functional dynamics or informational dynamics are associated with conscious versus unconscious processing.So I think probably modern consciousness science in many ways you can see—not the very beginning, but it really starts to come into its own in the early 90s with the work of people like Christof Koch. Actually, maybe I could push it a little bit back to the 80s and the work of people like Bernie Baars. But let’s talk about what was happening in the 90s. In particular, Francis Crick, one of the discoverers of DNA, has this lovely paper where he basically claims consciousness should be approached from a scientific angle. I’ve got a nice quote from this I can read: “No longer need one spend time attempting to understand the far-fetched speculations of physicists, nor endure the tedium of philosophers perpetually disagreeing with each other. Consciousness is now largely a scientific problem. It is not impossible that with a little luck we may glimpse the outline of the solution before the end of the century.”And he was writing that in 1996. Suffice to say, that’s not how it worked. That’s not how it played out. But one of the projects that Crick and others contributed to was this search for neural correlates of consciousness. So we know about unconscious perception, for example, versus conscious perception. What’s going on in your brain that distinguishes the two cases? So if I show an image just below threshold, so as far as you’re concerned, you didn’t see anything, and I show it just above threshold, so you are aware that you’ve seen something, what’s going on in terms of brain dynamics that distinguishes those two cases?And this question, this way of framing the question, it’s very relevant, for example, if we’re interested in things like predicting recovery of patients in persistent vegetative states or identifying lingering consciousness in people in minimally conscious states. And it’s given rise to a whole host of different theories of consciousness, scientific theories of consciousness. So there are many of these and new ones are being added all the time. I think it’s safe to say that none of them have won the room.But the big ones are views like global workspace theory, which basically says consciousness is a kind of way that information gets shared across the brain. So information is processed in the brain—I should say more neutrally in intelligent systems. If information is localized, only available to some subsystems, it’s non-conscious. But when information is available to all subsystems, or a dedicated global workspace, as it’s called, basically a dedicated broadcasting network across the brain, that’s when information becomes conscious.Another example of an influential theory is higher order thought theory that says information in the brain becomes conscious when it is the target of a further higher order thought. So at some level, your brain—you think to yourself, not consciously, although that’s a messy question—at some level your brain is representing, I’m having a perception of red right now, or I’m perceiving a red apple. So when you get a higher order state turning a spotlight as it were on a first order state, a first order mental state, that first order mental state becomes conscious. Now that’s a very crude...Dan Williams: Can I just ask, just really quickly on that as a follow up, because I think this might be occurring to people. So in the global workspace theory, the conscious states are those that get broadcast to other cognitive systems within the system, within the brain in our case, but you can imagine AI systems working like this. Whereas the higher order thought theory says a psychological state becomes conscious when it’s the target of another thought. And the difference between those is in the global workspace theory, whether or not a state gets broadcast to these other cognitive systems is not a matter of whether it’s the target of another thought. Is that correct as a way of understanding the difference? Okay.Henry Shevlin: Exactly. Exactly. Yeah. So on a global workspace theory, it doesn’t matter whether a thought is the target of another thought or a mental state is the target of a thought, a higher order representation. As long as it’s just available to everything, it’s conscious. Whereas for the higher order theorist, only those thoughts that are directly thought about or any mental states that are directly thought about are conscious.Dan Williams: Yeah, okay, so those are two influential theories. As you’ve mentioned, there are a ton of theories, maybe not a ton of influential theories, and potentially people having their own theories that they’re introducing all of the time. I mean, from your perspective then, I suppose one way in which you could approach AI consciousness is you look at scientific theories of consciousness, which have been developed, presumably developed entirely by looking at examples of consciousness either in human beings or to a lesser extent maybe in other animals. You take those scientific theories of consciousness and then you sort of conditionalize and you say if such and such a theory is correct then what are the implications of that theory for thinking about AI consciousness? Is that how you think about this topic or do you approach things differently?Henry Shevlin: Well, I think that’s a productive way to think about it. So there’s this wonderful report by Patrick Butlin, Robert Long and others from a couple of years ago called “Consciousness in Artificial Intelligence: Insights from the Science of Consciousness.” It basically does exactly that. It says, let’s take all of the leading theories and say, what would an AI system need to do in order to be conscious by the lights of these theories and do any current AI systems do it? And what they find—it’s a very long, very good report—but basically is that consciousness does not seem like the kind of thing that’s impossible for current architectures to realize. In some cases, not for every theory, but for many theories, even with fairly minor tweaks, existing kinds of architectures could give rise to conscious systems.But there are a couple of problems with this. The first big one, of course, is that there are literally hundreds, if not thousands, of different scientific theories of consciousness. And I think they’re basically never refuted. New theories are constantly being added to the table. The second thing is that as they do in the report, right, there are different ways of operationalizing these theories of consciousness.So for example, global workspace theory in its modern form, sometimes called global neuronal workspace theory, it’s associated with Stanislas Dehaene, a fantastic cognitive neuroscientist. And he’s got a great book called Consciousness and the Brain. He spells out what global workspace theory says, but he spells it out in a few subtly different ways. At one point he says something like, consciousness is any kind of system-wide information sharing. At other points he says, consciousness occurs when information from working memory is made globally available to modules including episodic memory, language and so on.Hang on, those are two quite different claims, right? The first one suggests that even quite simple architectures might be conscious, maybe some existing architectures. The second one makes it sound like only creatures with our specific sorts of cognitive organization can be conscious. So even with our existing theories, there are different ways of spelling them out. This is something I go into in a paper called “Non-Human Consciousness and the Specificity Problem.” Different ways of unpacking or operationalizing them that have potentially very different conclusions for whether or not AI systems or for that matter non-human animals are conscious.Dan Williams: Yeah, okay. I mean, yeah, that seems right. I suppose one issue here, which is hanging over the entire thing is understanding of consciousness, philosophically, metaphysically, scientifically, there’s so much uncertainty still that all of that is then carrying over to these issues of AI consciousness in a really significant way.I mean, maybe we can just end with two overarching questions. The first of them, I think, follows pretty directly from what we’ve been saying. The second is a question which connects to, I guess, social, political, ethical stuff. So the first question I think we should look at is, okay, in light of all that we’ve said so far about the metaphysics of consciousness, the weirdness of consciousness in the scientific world, these different scientific theories, and so on, how should we actually think about state of the art AI systems? What are your views about that? And then the second question is what’s at stake here? Why is this an important issue? Because I think there are ultimately, you know, there are very interesting scientific, there are very interesting metaphysical questions here, but there are also presumably very, very important ethical questions when you’re dealing with the possibility of conscious machines that’s hanging over the conversation as a whole.Taking that first question, I mean, what’s your sense, given your expertise in this area, given your views in this area, if you take a system like Claude, Gemini, ChatGPT, what’s your sense? Are these systems conscious in some sense, is there something it’s like to be them?Henry Shevlin: So it’s a very reasonable question and it’s one I don’t have a good answer to. I think basically the only kind of answer I can give, given the massive uncertainty, is to hedge across so many different theories, so many different methodological approaches. Probably my conviction is that basically we don’t know our ass from our elbow when it comes to what consciousness is or how to measure it. Therefore, I think we are basically in a state of near total uncertainty when it comes to consciousness in AI systems.That said, I’m a good Bayesian, I can deal with all this. So if I had to put numbers on it, they would come with huge error bars. But I think there’s a non-trivial chance that some existing AI systems have at least some minimal form of consciousness. And in particular, we don’t want to get too deep in the weeds here, but I don’t think it’s likely that any AI systems feel pain or have perceptual experience. But there’s a type of consciousness that’s sometimes called cognitive phenomenology. Think about the kind of experiences you have when you’re reasoning through a problem or come to a sudden insight or comparing two different ideas in your head without any accompanying visual imagery, just the raw processing of concepts. If you think there’s some kind of conscious experience associated with that, it doesn’t seem crazy to me to think there could be some kind of analog of that in AI systems.And I guess one reason I’m a little bit more open minded about that than I think some people is because I’m pretty liberal about consciousness in biology. I think I’m very high credence, probably above 80%, that honey bees, for example, are conscious. I think it’s just the best way of understanding complex behavior in honey bees. There’s a whole big story there. But the point is I’m pretty liberal about where I think consciousness extends in nature. It can arise in quite simple systems, which I think pushes me towards being a bit open-minded about the possibility of consciousness in AI systems.But it’s worth just really emphasizing the degree of uncertainty among experts on this. I’ve got some nice choice quotes here. So back in February 2022, Ilya Sutskever, former chief scientist at OpenAI said, “It may be that today’s large neural networks are slightly conscious.” This happened on Twitter, like all good philosophical discussions. And he got a reply from Yann LeCun, who’s obviously now head of AI research at Meta, or at least was until recently. Very prominent AI researcher at Meta.Dan Williams: And also we should add a critic of the large language model based approach to AI. Sorry to interrupt.Henry Shevlin: No, absolutely. It’s an important addendum. So he replied to Ilya saying, “No, not even true for small values of slightly conscious and large values of large neural nets.” But Murray Shanahan of DeepMind, I thought had the best reply here. He said, “They may be conscious in the same sense that a large field of wheat is slightly pasta.” Which I think is just brilliant and hilarious. So in other words, you’ve got the raw materials there, but it hasn’t been turned into the finished product. That’s one way of interpreting that.And on the philosophers, philosophers are just as divided. Dave Chalmers has said, “Questions about AI consciousness are becoming ever more pressing. Within the next decade, even if we don’t have human level artificial general intelligence, we may have systems that are serious candidates for consciousness.” But Dave’s colleague at NYU, Ned Block, another one of the titans of modern consciousness, says, by contrast, “Every strong candidate for a phenomenally conscious being has electrochemical processing in neurons that are fundamental to its mentality.” And my old PhD supervisor, Peter Godfrey-Smith, also said in his book, “If the arguments in this book are correct, you cannot create a mind by programming some interactions into a computer, even if they’re very complicated and modeled on things that our brains do.”So I think that just gives you a sense, these are all titans in their fields, just how divided opinion is on this issue.Dan Williams: Yeah, right. You can’t just trust the experts, as people like to say with that slogan. I mean, I should say, I actually share your view, and there’s massive uncertainty, but it’s certainly not absurd to think that there’s something it’s like to be these systems that are state of the art, even though we shouldn’t think that what it’s likeness is anything like what it’s like to be a human being. But just to play the devil’s advocate view, I think lots of people think that is just kind of crazy. Actually, we should be certain there’s nothing it’s like to be these systems. They’re chatbots. They’re doing next token prediction on vast bodies of text. That’s not quite right, actually, for some of these systems, but that’s the kind of view, right? They’re stochastic parrots.I saw a post the other day on another social media platform, Bluesky, with its own pathologies as a social media platform. And that was very much the spirit of the post. And from what I can gather, this is also the spirit of the popular opinion on Bluesky, which is just, it’s kind of absurd to even be talking about consciousness in a large language model. And maybe not even just absurd, but also potentially kind of dangerous or troubling or buying into the hype of these profit-seeking corporations and so on. So I don’t believe any of that, but I want to throw that at you and get your response.Henry Shevlin: Yeah, so I find it a little bit baffling that people would see this as an offensive question. I mean, I’ve had people say, it’s offensive to think that an AI system could be conscious. And I just want to say, look, this is a scientific question, right? And it’s a philosophical question. And honestly, it doesn’t even necessarily have any direct normative implications. Maybe we can get to that in a second. But I mean, simply saying there might be some basic forms of cognitive phenomenology in an AI system—that doesn’t entail robot rights by itself, for example. So anyway, I think I don’t get the offensiveness angle.One argument that I have engaged with, and I think it’s worth unpacking a bit, is this idea that it’s just matrix multiplication, or it’s just next-token prediction. And without wanting to go on too long a digression here, I think people really need to go away and read their David Marr. David Marr, one of my absolute heroes in cognitive science, died tragically young, but wrote a very influential book, his one and only book published posthumously in the 1980s on vision. And one of the basic insights he brings to bear is that look, almost any complex informational system, whether you’re talking about human vision or a computer, can be analyzed at multiple levels of explanation.So you’ve got the high level functional, what he called computational explanation, which is what is this system or subsystem doing? So in the case of a part of your vision, it might be the system detects edges. Then moving down a level, you’ve got the algorithmic explanation. So how is that function accomplished in informational terms? What kinds of computations are being done to calculate where an edge is in your visual field? And finally, you’ve got the implementational level explanation, like what neurons are doing what, what circuits are doing what, how is this algorithm actually instantiated? And the point is almost any system is going to be analyzable at multiple levels right?So at some level there’s going to be a mathematical gloss on what’s going on when I’m thinking through a problem in the human brain, at least if you accept even the most basic form of scientific naturalism, right? And even if you think there’s much more that’s going on, sure there may be in the human brain, but we’re also doing the computations, right? There’s going to be at least a computational level of description. And you can see this so clearly in a lot of perception, for example, where people produce very accurate predictive models, computational models of how early vision works or how early linguistic processing works.And the fact that you can give the implementation level, algorithmic level, lower kind of functional level explanations of what any system is doing doesn’t exclude psychological or phenomenal level descriptions at all.Dan Williams: Right, right, yeah. Such an important point. And I think one of the things that being a philosopher, or at least a good philosopher, trains you to do is whenever you encounter someone saying X is just Y, for alarm bells to go off, because I think that often smuggles a whole lot of very, very dubious ideas. Whether these systems are just minimizing prediction error on a next-token prediction task, or human beings are just complex biophysical machines or whatever. There’s a lot going on when people say a comment like that, a lot that’s getting smuggled in that needs to be actually thought about rigorously in a way that it’s often not.Okay, we’ve touched on this already. Let’s end with this issue of what’s at stake. So I mean, this is, surprisingly to me at least, but it’s undeniably kind of a polarized area of discussion, both AI generally. I think there’s lots of heated, let’s say, conversations about how to make sense of this technology, but also specifically when it comes to these issues of AI consciousness or sentience, as people often describe consciousness in sort of popular conversation. What’s your sense of what’s at stake? Why does this matter? Why is this an important conversation?Henry Shevlin: Yeah, so I think this is a really key point. And maybe we should have even mentioned this at the start of the program. One of the reasons why consciousness matters so much more, I think, than most other kinds of psychological states—we can argue about whether LLMs have beliefs or understand, but what makes consciousness so important is its connection to ethical issues, right? So there’s this famous passage in Peter Singer, one of the godfathers of modern utilitarianism and effective altruism, where he says, “A schoolboy kicking a stone along the road isn’t doing anything morally wrong because the stone can’t suffer. It’s not conscious and it therefore doesn’t have any interests, right?”So one of the reasons we care about consciousness is because consciousness seems like a prerequisite on many views for having interests at all. If you can’t suffer, if you can’t feel pain or orgasms, if you can’t have positive or negatively valenced experiences, right? Then it’s very unclear whether you deserve any kind of moral consideration at all, or at least you get a lot of extra moral consideration by virtue of this.And this is also one of the reasons why I think consciousness in animals is what actually got me so interested in animal consciousness, because one of my favorite essays of all time is David Foster Wallace’s “Consider the Lobster.” Fantastic read, highly recommended. And when a chef drops a lobster into a pot of boiling water, it seems to me to matter a great deal whether there’s something it’s like for that lobster to experience that, whether it suffers pain and suffering. That seems like a very important question. And equally, if we’re thinking about which animal welfare interventions to prioritize, right? Is it worth spending money on shrimp welfare, for example, to give a controversial area? Right? It seems to matter a great deal whether there’s anything it’s like to be shrimp, whether they can genuinely suffer. And so I think consciousness has this special normative connection that just isn’t clearly shared by any other psychological concepts. And that’s part of what makes it so important.Dan Williams: Yeah, I completely agree with that. I’m going to maybe just end with a discussion potentially of the contrast between those cases where we’re thinking about non-human animals and these cases where we’re thinking about AI systems. I mean, there are two mistakes you can make here, right? There’s a kind of false positive where you attribute consciousness where it doesn’t exist or a false negative where you fail to acknowledge consciousness that does exist.It seems to me that when it comes to, say, lobsters, we should err on the side of caution. I think there’s very, very good reason to think there is something it’s like to be a lobster. But fair enough, there’s uncertainty. It doesn’t seem like it’s the end of the world if we have a false positive here and that stops us from burning them alive. It seems, I mean, beyond egregious, on the other hand, to make the mistake of failing to take into consideration that they’re conscious.With AI systems, I suppose it’s complicated because clearly there are real big issues here if we are in fact manufacturing conscious systems and we’re not recognizing them as conscious. But I can also see the argument from people who say there could also be big issues here if we’re just building—just to use that word—if we’re building machines that aren’t conscious, there’s nothing it’s like to be these systems, they’re sophisticated chatbots or whatever, and we’re treating them as conscious. To me at least, I can understand what the downsides might be in that kind of scenario. I think there is something a little bit troubling if people start treating systems that aren’t conscious, that is AI systems that aren’t conscious as if they are, in a way that when it comes to these other cases, I’m not so sure. Have you got any thoughts about that, just to sort of wrap things up?Henry Shevlin: Yeah, absolutely. So I think another really important difference here is that there are some states that we undergo that we recognize as very, very bad, like extreme pain, starving to death, extreme nausea, that seem to have fairly straightforward physiological analogues in non-human animals. And I think that justifies a pretty strong precautionary attitude about inflicting those kinds of states on animals, right? Sticking me in a pot of boiling water, I can tell you would be pretty horrific. And so it’s probably a good idea not to do the same things to creatures that have relatively similar behavioral responses to pain to me, right?But it’s just much less clear what it would even mean for an LLM to suffer, right? They don’t have bodies. They don’t have any kind of somatosensory processing. That doesn’t mean they can’t suffer, right? But it makes the question of what suffering in LLMs would look like a lot harder to answer. So I think one model for thinking about how LLMs work, that Murray Shanahan has popularized, is that they sort of role play. So if Claude says, like, this is awful, I’m really distressed right now, right? Is that more like an actor who’s portraying Romeo on stage? “My heart is shattered by the death of my beloved Juliet,” right? Or is it actually suffering? I think this is one of the things that makes AI welfare really hard.My general sense here is that this is an area where we absolutely need better understanding, but also just better theoretical models of what it would even mean for an AI system to suffer in the first place. And I also, I think I am sympathetic to the idea that precautionary principles are at least easier to apply in fairly straightforward ways in the animal welfare case compared to the AI case.That said, I also don’t think we can rest easy in the AI case. And partly, just to give one little example, suffering in biological organisms seems to be relatively biologically constrained. You have various kinds of negative feedback mechanisms like endorphins and so forth, just because suffering is generally not a particularly adaptive state to be in. It’s a powerful way that your body can send a signal that you’re injured or you desperately need to eat and so forth. But there are biological dampening mechanisms in most cases, not in all, but in that sort of create a kind of upper limit.But it’s not clear that those would arise spontaneously or by design in AI systems. So the kind of theoretical upper limits of the extremes of suffering in AI systems may be less constrained than in biological systems. All that is very speculative, of course, but just to note that there could potentially be a lot of suffering associated with badly designed AI systems. And of course, if you’re dealing with systems that can have billions or trillions of instantiation simultaneously. That could quickly add up to some really messed up things that we’re doing.I don’t think, for what it’s worth, I’ve not seen any compelling evidence, or I don’t think there’s much compelling reason to say any specific things we’re doing with AI right now are plausible candidates. There’s no AI equivalent to factory farming that’s obviously objectionable. Also, just to forestall one argument that I hear all the time, and I think is just so—it really frustrates me is when people say, the only reason people want you to think AI machines suffer is because AI companies want that extra recognition for the value of their products. I can tell you this is the last possible thing in the world that any tech company wants. The idea that their products that are making them billions of dollars right now, the idea that they might have rights, would be really, really bad for their business models.Dan Williams: Right.Henry Shevlin: Insofar as there are people pushing for greater awareness of AI welfare, they are not operating with any kind of commercial agenda in mind. And I think commercial agendas, in fact, push in the opposite direction.Similarly, also just because—it’s so interesting how many people get really angry about this debate, the fact the debate is even happening. I meet people who’ve said it’s obscene to even debate the idea that AI systems might one day deserve rights. What I would say to these people is look, even if you think, like, even if your cognitive science of sentience or consciousness or theory of moral patency, even if you think there’s basically no chance that any AI systems are conscious, well, you should be engaged in this debate because a lot of people are going to take that seriously. And if you think they’re making a mistake, you need to engage with them and tell them why they’re making a mistake, right? It’s not a debate that I think you can just dismiss with a look of disgust, right? If you think that we’re in danger of making massive false positive ascriptions of moral status to AI systems, you need to tell people why and actually have that conversation, rather than dismissing it with a grimace of disgust.Dan Williams: Yeah, yeah, just dismissing it doesn’t seem like it’s an option now. And I think as these systems develop in sophistication and their capabilities develop, that that conversation is going to become more and more important. And I think one thing that you alluded to there is there’s a question about how we should treat these systems. And there’s a question about how human beings will treat these systems. And I know that connects to some of your interests when it comes to things like social AI and so on. But we should postpone that to a future conversation. Henry, this was fantastic. Any final word, final comment that you want to end on?Henry Shevlin: All right, so I’ll close on two reflections. The first is, on the one hand, as listeners who were not previously familiar with consciousness debates will have realized, it is possibly the messiest debate out there. Both scientifically, theoretically, metaphysically, it is an absolute snake pit of a debate. But don’t be put off, because I think it is also the most fascinating and rewarding question that I’ve ever worked on. I have happily dedicated basically most of my academic life to working on consciousness and I don’t regret it for a second. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.conspicuouscognition.com/subscribe

How similar is ChatGPT-5 to the human mind? How should we measure AI progress? How close are we to transformative AI? Are "super-intelligent" AI sytems likely to kill us all? This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.conspicuouscognition.com/subscribe