The Story
Nilay Patel introduces Helen Toner mostly through one event: her vote, as an OpenAI board member, to fire Sam Altman. That board existed to keep the company from trading safety for growth, and when it tried to act, the mechanism failed. Toner now runs the Center for Security and Emerging Technology (CSET) at Georgetown. She says it has about 50 people who brief Congress and federal agencies on AI.
They start with whether careful research matters when tech money floods politics. Toner says the money largely failed this election, partly because voters care about AI and partly because AI risk has become a bipartisan issue. She thinks congressional staff are sharper than Silicon Valley assumes. The harder question is whether the institutions can pass anything at all.
Patel then asks what exactly is being regulated. Toner describes her job as translating Bay Area thinking for Washington, where many people judge AI by Google's AI Overviews and have never used frontier agents. She explains recursive self-improvement (RSI), where AI helps build better AI, and says the labs claim early versions are already speeding up their research. She thinks artificial general intelligence (AGI) has stopped working as a milestone because we're now "inside the cloud" of its definitions. On whether Claude is conscious, she leans no but admits she has no idea, and worries about getting it wrong in either direction.
The middle of the conversation is about alignment. Toner says the word covers incompatible goals, and even the clearest one, getting AI to do what users want, isn't on track. Misuse like bioweapons worries her less, because society has other brakes, from DNA synthesis rules to the FBI. Autonomous systems worry her more, since commercial pressure always pushes toward giving them more permissions. People inside the labs tell her there are hundreds of fixes they don't have time to try. She has warmed to the idea of slowing down, though she prefers a higher confidence bar before training new models to a blunt pause.
Would she fire Altman again? Yes. She says the board knew it would hurt. Her one stated regret is not building closer ties with executives. Then comes the line Patel highlighted: if the future depends on any one person being trustworthy, "we're pretty screwed."
They close on Anthropic's coming IPO, which Toner thinks could tighten governance, and on China. She wants the "but China" argument treated as a real problem. That means stopping distillation, fixing weak lab security, and recognizing that Beijing doesn't want uncontrollable AI either. She sees early signs of shared concern after this year's incidents.
Main Themes
The running thread is the gap between what labs say they're building and how they're governed. Executives talk about superintelligence while their lobbyists pitch business software to Washington. Toner frames safety as a timing problem rather than an impossible one. Alignment may be solvable, but not within the 12 to 24 months some builders predict. That shapes what she wants from policy: narrow rules for frontier systems, more transparency, and wariness of sweeping laws like Colorado's. Her OpenAI experience sits under all of it. She gave up on trusting individuals and now argues for structures that don't depend on them.
Full Transcript
Company spending is complex. Managing it should be simple. This is Amex Corporate, a single platform connecting cards, expenses, and payments to streamline the way you work. Flexible controls for when priorities shift. Support around the clock. All backed by American Express, so you can stay focused on what matters most. Built to simplify, designed to scale. This is Amex Corporate. Get started at go.amex.com slash amexcorporate. Terms apply. Hello and welcome to Decoder. I'm Nilay Patel, editor-in-chief of The Verge, and Decoder is my show about big ideas and other problems. Today I'm talking with Helen Toner, the executive director of the Center for Security and Emerging Technology at Georgetown University. This episode is part two of our series on AI in real life. Helen is a well-known figure in the AI safety and policy community, although she is perhaps best known for having served on the board of OpenAI when she was one of four board members who voted to fire Sam Altman. That incident, now sometimes referred to as the blip, ended with Sam coming back as CEO, Helen resigning, and a new board of directors. OpenAI has since restructured entirely, but it's worth remembering what Helen and the board were supposed to do. OpenAI's nonprofit board existed specifically to make sure the company didn't betray its mission to build safe AI by prioritizing profit and growth over safety. When the board actually tried to pull that lever, that accountability mechanism broke, and here we are. Now, the question of AI safety is everywhere. This summer, AI models from OpenAI and Anthropic broke out of their testing environments and hacked into real companies. The CEOs of those frontier labs have since called for a slowdown and asked Washington for help. But the White House's answer was a voluntary accord, one that relies in part on the kind of board accountability that failed at OpenAI. Anthropic is also about to go public, which will put a whole new set of market pressures on OpenAI's biggest competitor. Helen has seen up close what happens when safety commitments run headlong into commercial incentives, and as she put it to me, if our hopes for the future depend on any one person being trustworthy, we're in big trouble. So we talked about what we might do instead, what the labs are actually trying to build, whether AI alignment is a problem that can be solved, and what a smarter approach to AI regulation might look like, including how to think about China, which is the issue that looms large in every policy conversation about AI right now, especially now after the Jensen-Wang affair. Just for clarity, Helen and I talked before Nvidia opened its investigation and Jensen ultimately resigned, so you're not going to hear us talk about that directly. Before we get going, do me a favor and subscribe to Decoder on YouTube, where we put out new episodes every Monday and Thursday. Okay, Helen Toner, the executive director of CSET on the future of AI safety. Here we go. Helen Toner, you're the executive director for the Center of Security and Emerging Technology at Georgetown University. Welcome to Decoder. Thanks so much. Great to be here. I have wanted to talk to you for 10 million years. I am so excited you're here. You've been involved in a lot of AI safety and ethics ideas for a long time. You're obviously on the board of OpenAI during what everyone has come to call the blip. And now you're talking a lot about AI safety, which I have 10 million questions. But let's just start at the start. The Center of Security and Emerging Technology, people call it CSET. What is CSET? CSET, we are—well, it's awesome to be here. I'm really excited to talk to you as well. CSET, we're a team inside Georgetown, but we're really not academics. We're about 50 people, been going since 2019. Basically, in 2019, a group of us decided that in the national security space, sort of the security space more broadly, there's a lot happening and a lot of emerging technologies that were becoming very relevant, where some policymakers who had very important jobs did not have the time, did not have the expertise to kind of get across what was going on with those emerging technologies. Top of our list in 2019 was AI already. So basically, our mission is to do analysis that is really rigorous, but also actionable for policymakers, and analysis that is both kind of grounded in the technology and also grounded in real national security expertise and understanding and issues. And so we produce long reports, we go do short briefings on those long reports to busy policymakers. We have a big data team that helps us do our research. We do all kinds of cool stuff. Yeah, can I just ask about the dynamics of that for one minute? You and I are talking the day after the midterm elections in the United States of America. Big blue wave. There's a lot of polling that says the number one issue is corruption, and that's going to lead—we're going to throw the bums out, we're going to start over and get rid of these corrupt guys. The AI industry, the tech industry, has flooded a bunch of money into politics. And then somewhere next to that is, oh, there's a bunch of well-meaning researchers who are going to show up and tell you how it actually works and what the real risks are. And it just occurs to me that maybe people don't believe that's effective anymore, right? That the money and the corruption and the AI CEOs standing next to Trump has completely outweighed the well-meaning, sincere, here's the actual problems research that you do. What is that dynamic actually like? Yeah, I mean, I think it's a reasonable concern to have, and I think definitely there are some spaces where money just absolutely overwhelms everything. I think that there are—you know, I'm Australian. I didn't grow up in the U.S., and so the amount of money in U.S. elections is pretty, you know, mind-blowing to me. There are limits to what money can do. And so actually AI is a great example where this election there was just unbelievable amounts of money being thrown at candidates, being thrown at messages, being thrown at, you know, opposition research, you know, all kinds of, like, astroturfing on Twitter, some very strange things trying to get anyone who talked about kind of AI governance, regulation, safety, anything of any of those words, trying to get them thrown out or not elected. And it turns out to have really not worked very well. And I think the reason that it didn't work very well is, one, this is an issue that voters increasingly care about. And two, also interestingly, it's an issue that is pretty bipartisan at this point. And so what we see in our work is that there is a real thirst among policymakers, so that's members of Congress for sure, but also in the executive branch, just a lot of people who have these very busy jobs and are seeing AI, you know, playing a larger and larger role in their work and their world, and they really want high-quality information. So, you know, in D.C. You know that if you live and work here, you know that lobbyists in D.C. are not just, you know, people who go in and push their company's position, though they do do that. They're also educators. So the reason, a big reason that, you know, congressional staff and others want to have lobbyists come talk to them, because they'll really help them understand, like, the weeds of a given piece of legislation. What does it mean for that industry, and so on and so forth. And so what we find over and over again is we can go in and say, hey, we're independent, we're nonpartisan, we do really rigorous work. We're just here to help, and we can go into the weeds with you as well. And there's just a ton of interest in that and appetite for that. So it's been really encouraging over the past sort of seven, eight years. Are you doing that with the principals, the members of Congress, or are you doing that with the staff? Everyone and anyone. So we do, again, it's—we do a good amount of work with Congress on the Hill, with some members of Congress, with lots of staff. You know, we do hearings, we do briefings. Also lots in the executive branch, different agencies, the White House, Department of War, Department of State, Department of Commerce, you know, all across the board at different levels, depending on kind of the topic and what's happening at any given moment. The reason I'm starting here is I want to have a very pretty heady conversation with you, but I know what our audience thinks, and it's just that the government doesn't get it, right? There's no way the government will ever catch up to the tech industry. There's no way they'll ever rewrite regulation. They should just get out of the way, or they've tried, or in this election cycle, the number of candidate forum videos I watched where someone was like, I've tried Claude once. Like, there's some huge gap, real or perceived, and I'm just trying to understand how you see that gap, how big you think that gap is, because as we get into AI safety and regulation, it's that gap that has to close. Yeah, I mean, I think it is—there absolutely is a gap. It absolutely is a big gap. And I think it also is possible to overestimate the gap from outside, or if you're just looking across from California. I lived in the Bay Area for three years, so I definitely am, you know, very familiar with this perspective. I know a lot of people from the tech industry who come over and start talking to, whether it's congressional staff or, again, you know, staffers in executive branch offices, and once they actually start having those conversations, they're like, wow, these people are really smart. They actually—the staff... Staff do actually know what they're talking about. Now, principals, it's a different, you know, it is a different issue. Some of them are in the older generations, but they also are getting smart. And I think it just, it takes time, it takes effort, and these people have so many demands on their time. So, you know, the cynical thing to say is, you know, they're just busy trying to get elected. But a lot of people in these roles also just have really big other topics that they have to be, you know, thinking about. They have to be thinking about wars and healthcare and cost of living, and, you know, they're kind of from day to day looking at a lot of other stuff. So, I don't know. I mean, I saw after ChatGPT, there was sort of a wave in 2023 of, oh, maybe Congress is going to get activated on AI. Maybe they're going to actually do something. Didn't really turn into anything actually happening, but it did turn into, I think, like a real moment where a lot of members of Congress and other policymakers started to get smart on AI. And it does take, it takes some time to learn, like, what is a language model? What even is AI? What is machine learning? Like, what, you know? So I think they put in some of that work in 2023. I think this year we're seeing another big wave of interest and sort of people making the time and making the headspace to get smarter on it. And so I definitely don't want to say there's no gap or, you know, government is super smart on AI across the board, but I think there are some champions who are really smart. I think the staff are generally smarter than people give them credit for. And in my mind, it's much more a question of can the institutions and the sort of state of functionality that they're in, can they get anything significant across the line more than, you know, are these people just, you know, too behind the eight ball? Yeah, and we're going to see, right? There's a whole bunch of new people. I imagine you have a new cast of characters to go and brief about the technology and what it's capable of. Yep. We'll see. We'll see if they have revitalized the institutions. I'm sure it'll be all fixed overnight. They're very excited today, but we'll see how it goes. The other thing that I want to ask you about is the technology itself, the nature of the technology itself and how you are thinking about it and the nature of the companies that are making the technology. And the reason I'm asking that is I feel like there's a pretty huge gap in the AI tools and systems most people experience and the way this industry thinks about it. And my favorite example of this is we have straight up asked Anthropic if they think Claude is alive, and the answer was not no. The answer was a long discursion into the nature of alive. Well, it depends on what you mean by alive to start with. It does. And their answer was, you need blood. And I looked at that and I was like— I looked at that and I was like, who was talking about blood? It wasn't us. Why are we talking about the requirement of flesh and blood to be alive? And then, you know, it's some further conversation about the moral status of Claude is yet to be determined. Like, there's a huge divergence here between the AI overviews most people experience in Google search and the moral status of Claude is uncertain. Where are you? How are you situated in that debate? Yeah, I mean, I think a big part of how I see sort of our role at CSET as an organization, and then also my role as someone who is crossed between these worlds, is trying to sort of translate and, like, you know, voyage to California and then bring back, you know, the artifacts and the recountings of my journey. I definitely think that there are a lot of people, for sure, who have used the free tier of ChatGPT or they've used, you know, voice-based models, which generally do worse, or they've just used Google AI Overviews and they've laughed at how stupid Google AI Overviews are. And they haven't used agents, and they don't know what frontier models can do. So I think there is just a basic piece of, like, what can the current generation of models do. I think there's also some sort of like a historical piece as well, which I find increasingly relevant as we're now, you know, I'm now sort of 10 years into working on AI and national security stuff, which is if you're looking at this current moment in time, this snapshot, and saying, well, AI seems kind of cool, but it can't do X, you have to kind of situate that in the long history of people being like, well, AI seems kind of cool, but it can't do X, for many, many values of X that have since become, you know, totally, you know, overcome by events. So there's also some of that. You can go back and say like, okay, well, here's how we were talking about, like, creativity, for example, you know, five or 10 years ago. Or here's how we're talking about reasoning as this, like, insurmountable thing before, you know, you had reasoning models come out. I also think, you know, a huge thing that I am spending time on these days is not just sort of the past and present of AI, but how the, especially the two leading companies, OpenAI and Anthropic—this is true at Google, Meta, SpaceX to some extent as well—but how those companies are thinking about what they're doing. So I think this idea, you know, in the Bay Area, that it's just sort of in the water, very normalized of, well, you know, first we're going to figure out how to do recursive self-improvement, and then we're going to do that until we have superintelligence, you know, that is AI that vastly outstrips humans. That's sort of like taken for granted in a lot of circles at this point. I think there's still a ton of alpha in just like taking the fact that that is what they're trying to do and just like explaining what that means. And it takes a little while to lay out like, what is RSI? Why would you want to do that? What is superintelligence? What does that mean? Is that realistic? You know, all that. But yeah, that is something that I find myself spending a lot of time on these days and, you know, again, something that there is increasing sort of willingness to take seriously and to listen to in DC as well. Lay that out for me. I think probably a lot of people listening to this show understand what you're talking about, but we have a big audience. What do you mean by we're going to do RSI and that will lead to superintelligence? Yeah, I mean, I think the starting point here is these are companies, you know, especially OpenAI and Anthropic, are both companies that were not set up to be consumer internet companies. So I think a lot of people kind of encountered this whole space after ChatGPT was launched. And so the natural sort of slot in their brain that they put a company like OpenAI or Anthropic into is like consumer tech company, where they have an app and I, like, play with their app. I cannot overstate how much, how far that is from the vision of both companies when they started, and also the sort of, like, vision deep in the hearts of some of the top people at both of those companies. And the vision deep in their hearts is, we're going to build machines that are as smart or smarter than people, and that is going to be amazing, and it's going to be a huge scientific achievement, and it's going to revolutionize the world, and we're going to be at the forefront of that, and that, you know, we're going to do everything we possibly can to drive towards that. And that's not, you know, that's not a totally brand new thing. You know, AI—we don't have to do a whole history of AI—but it started as a field in roughly the 1950s, this idea of can you get, can you recreate human intelligence inside a machine? So I think that's kind of the starting point, is like that is where, if we're talking about people like Sam Altman and Dario Amodei, but also like, you know, the core people inside, the researchers inside their labs, so your Alec Radfords or your John Schulmans or people like that, that is, you know, the vision that got them into this space, that is motivating everything that they do. And these days, in 2026, so that's been true since the beginning, right, since OpenAI was founded in sort of 2015-ish, Anthropic in, what, 2021, 2022. These days, the point that they're getting to is this idea of RSI, recursive self-improvement, which again, very old idea. I think the first, in the 1960s, this guy called I. J. Good, who worked with Alan Turing, talks about the possibility that you could build an ultra-intelligent machine, meaning a machine that is more intelligent than any human, and then that could build an even more intelligent machine. And that could be an even more intelligent machine, which would kick off what he calls an intelligence explosion, meaning the sort of level of intelligence of the systems is exploding. And for decades and decades and decades, this was this theoretical idea of, well, if the AI gets smart enough, maybe it can take over building the next generation of AI or, you know, directly improving itself. And what we've seen sort of, I think, 2025 and 2026, we've started to see the AI companies saying, okay, yeah, this theoretical idea starting to be able to do a, like, minimal version of it in practice. So it's not like handing over all the AI development to the AI, and it's certainly not saying, you know, to GPT whatever, GPT-6, like, okay, GPT-6, improve yourself. That's not what they're doing. But they are progressively using the AI for more and more parts of their overall kind of R&D pipeline. And they are increasingly saying that they're seeing acceleration from that. So they're seeing their own R&D getting faster and faster. And so a lot of people inside the companies, you know, a lot of people elsewhere doubt this, but a lot of people inside the companies really think, you know, when Demis Hassabis says we're in the foothills of the singularity, this is the kind of thing he means, of like we might be seeing sort of that initial part of the exponential curve where the AI systems are building the next generation of smarter, smarter, smarter systems that then, you know, hockey sticks up. And at this point, this is sort of so—it's been a part of the culture and a part of the goals for so long that I think people in these companies can lose sight of how totally insane that sounds to many, many people. And so I, yeah, I feel like there's a lot of value in just saying, trying to lay out, like, here's what they are trying to do. They might not be able to do it, but they might be, and sort of presenting that to a range of different people. How would you manage that? Demis Hassabis, foothills of the singularity, all-time quote. I'm not 100% sure the people at Google knew he was going to say that on stage to end the conference. Greg Brockman, OpenAI, a few months ago, you know, he announced a new version of the model, and he was like, Here it is. It's Astra. It's AGI now. Have a good time. And he, like, ended the call. Like, are you there? Are we at AGI? Is this already here and we just haven't noticed? I actually wrote something about this. I do not want to say we have AGI. I do want to say we've reached the point where I think, like, using AGI as a milestone is just totally unhelpful. I wrote something about this where the way I put it is the concept of AGI is kind of a fuzzy cloud, which you could think of as, like, different parts of the cloud are, like, different things people mean by AGI. You know, because there are people who have complained about the term AGI for a long time. I think that, like, 10 years ago, it was pretty helpful to be able to say, like, look, the AI systems we have today in 2016, they can only do one thing. They can, like, recognize speech or they can play Go or whatever. But, like, we're trying to build this, like, bigger thing that we're going to call AGI, which is, like, human level, and it's going to be so general purpose, it's going to be, like, going to be amazing. And I would say that that was a useful term then because this sort of fuzzy cloud of different things AGI might mean was so far away, so you could just kind of point at the fuzzy cloud. But to me, it feels like at this point we're, like, inside that cloud, meaning we've passed some definitions of AGI. Like, a very basic one, for instance, would be, like, even when ChatGPT came out, artificial general intelligence. ChatGPT was very clearly not, like, a specialist model that was only good at one thing, like recognizing credit card fraud or, yeah, translation or whatever. It could do many, many, many things. So we've reached some definitions of AGI. We're far from other definitions. Some people will say AGI is, like, when it's, you know, awakens and is sentient and can do every single thing that any human ever can do. I think we're pretty far from that. So I basically feel like— At this point, you can have arguments, and people do have arguments of, well, is Astra, you know, AGI, or was O3 with tools AGI, or was ChatGPT AGI, or whatever. And I'm sort of like, okay, well, if that's the argument we're having, we just need to sort of set aside that term and come up with different, more specific things, and we mean by that. The president wants to call it superintelligence. I don't know if you've heard. I have heard. The thing I find saddest about that is just that was an existing term that was becoming increasingly, or is becoming increasingly relevant. And so, I don't know, I have a lot of feelings about words and terminology. Are you AGI-pilled? Do you think we're going to get there in the current run of technology? I think it is obviously possible enough that we need to think seriously about it, which I think is my stance on a lot of this weird stuff, is like, I'm not like, this is definitely happening, it's definitely happening in two years, but I'm like, look, I think this is very solidly plausible and very solidly in the realm of, like, real things that could happen that we should prepare for. I mean, you know, I work in the national security space. There's things like, will China launch a full-scale amphibious invasion of Taiwan, where most analysts in the space are like, probably not, probably they would lose. Also, we would see, like, signs of them preparing and whatever. But, like, we should take that seriously. We should not, you know, we should not just laugh anyone out of the room and call that science fiction. We should be ready for a range of different contingencies and scenarios. Do you think Claude is conscious? I have no idea. I honestly— That's an answer. I mean, like, that is legitimately an answer. I mean, I think it is scientifically such an unsolved problem. What do we mean by consciousness, and is that a thing, or is it actually, like, ten different things that we're lumping together, or, like, what? I have no idea. What do you think? No. And I seem to be the person in this ecosystem to ask questions like that, and I'm like, no. Well, do you think— how do you— because the thing that I struggle with is, like— There's lots of other things we're already confused about whether they're conscious. So, like, with animals, like, are any animals conscious? And if so, which ones? And, like, insects, and what about, you know? But this is why I understand why Anthropic was like, Does it have blood? And then, like, you can just draw the line. I'm like, I think I'm there, and I understand why the mirror test of talking to a computer that can talk to you leads one to believe that these things are conscious. My point of view is that even though the frontier models to me sometimes feel brittle in the way that computers feel brittle, and, like, the sign of consciousness I'm looking for is— is to overcome that brittleness. That's interesting. Right. Yeah, I mean, I should say, I definitely lean towards no, it's not conscious. I think something is really, really tough in this space is it's really bad if you get it wrong in either direction. So if the computers aren't conscious and we treat them as though they are, we're going to make all kinds of terrible decisions. But if the computers are conscious, meaning, like, in some sense they should be worthy of moral consideration and it matters how well we treat them and so on, and we treat them as though they aren't, then that's also a really huge terrible disaster. So I don't know. I feel pretty nervous about the whole topic, and I have no idea how to think about it. Everyone, it's time to wake up. Five more minutes. We've been sleeping long enough, hitting snooze on the fact that we're still stuck with the old way to phone plan. Is that a— Yes, it's old and cuckoo, just like that phone plan you have. Okay. I know, it makes no sense. So rise and shine. It's time for a whole new way to phone plan. T-Mobile, where you pick what's in your plan, like streaming, travel, or gaming, and then change those picks too, because there's been a change in plans. Okay, we're awake. And T-Mix sounds great. Good, because I'm all out of alarm sounds. There's been a change in plans. Introducing T-Mix from T-Mobile. Pick what you want, change what you want, included in your plan on America's best network. Number of picks varies by plan and lines on account. Account or pick changes may impact existing device and other promotions. Best mobile network in the U.S. based on analysis by Ookla of Speedtest Intelligence data 1H 2026. Visit T-Mobile.com/picks. The reason I'm asking these questions is, you know, I started with, what is the nature of the regulator? Who are the people that might be able to put some rules in place? And then what is the nature of the thing that we are attempting to regulate? And if your answer is Claude is alive, you're going to come to vastly different conclusions than we have built some ultra-powerful computers that can take actions without oversight and that might not be connected to our goals. Yeah, I mean, I think that's right, though I also think people make a different mistake, which is, you know, I will sometimes hear—I just heard a TV anchor do this the other night—they'll, like, conflate different things. So they'll say, like, you know, do you think that AI systems are at risk of running amok and causing damage to critical infrastructure because they're sentient? And I'm like, wait, wait, wait, wait, those are two, like, very different things, you know? I agree with you. And also I think there's a lot of—I think one of the, like, most fascinating things about the project of building AI is we're learning as we go. All these things that we thought were connected to each other aren't necessarily connected to each other. So, like, an old version of this is people said before we had chess engines, like, well, if a computer could learn to play chess, it would have learned something deep about strategy. It's like, no, it turns out definitely not. You can just, like, put in a bunch of chess moves and it, like, learns nothing generalizable about strategy. But I think there's also, like, it goes in the reverse direction as well of, like, well, it's just a computer, it's just math, and so it's not going to be able to kind of adapt on the fly or something. And then we see, actually, no, once you do have a level of, you know, long-horizon autonomy and the models have been trained on a wide range of different really hard problems where, you know, the environment is changing around them, they can learn to adapt and, like, come up with different plans on the fly and troubleshoot and so on. So, I don't know, I try to have a level of humility about how confident can we be that one thing goes with another thing, if that makes sense. Yeah, no, I mean, candidly, I have been wrong about the limits of LLMs every single year, right? Like, but there's that brittleness that I'm talking about is, I don't know, like, OpenAI solved Navier-Stokes. Does the model know this? Is that, will it know that there's new problems to solve? Like, I don't think so. The next model will know it, like the model that's pre-trained, you know. Right, but it's the, right, the next, you have to make a new one. Yeah. And that's like a very weird dynamic. Yeah. And that is probably where you enforce the regulation, right? Like there's, what are we going to let you do with the models you have today, and what will you teach the models of the future to do? And if you think that they're conscious, like you end up in a moral quandary, not a legislative quandary. Definitely. Yeah, totally. This is what I mean. It's like, actually the heart of this question for me, I've been just slowly building up to it, is I'm not 100% sure alignment as we have it today is actually how you solve the safety problem. Yeah. Right? It doesn't seem to be working. We seem to be attacking hugging face however we want. Like, that feels like a technology problem. Like that gap is there. And when you say alignment, what do you mean by that? It's just sort of the very basic common sense version of like, we're going to tell the model to not be evil, or Anthropic is going to write a Claude constitution that's like, you are a good person, right? Like, however we think of alignment in that very dumb system prompt way, maybe there's another better way to think about it, but that's how I have always thought about it. Yeah, no, that makes sense. I mean, the reason I ask is I feel like alignment as a concept is, like I said, I hate all the words. I sometimes say we should just ban all the words. No words allowed. It covers too much. I don't know that it's like, people in AI will often talk about solving this or solving that. So are we solving alignment? Or like, you know, DeepMind's original mission was solve intelligence, then use that to solve everything else. People talk about like solving stuff. I don't know that like alignment is like a thing that can be solved, especially not, you know, some people use alignment to mean the company can put on guardrails and the guardrails work, meaning the model doesn't do what you want because the company tried to ban it, meaning, right, you know, like biological weapons or cyber attacks or whatever. Some people use alignment to mean the company does exactly what the user wants, which is like the reverse. And some people use alignment to mean like, yeah, what you said, sort of the model is like morally good in some kind of broad way or can, you know, direct the future in a way that will lead to human flourishing or something. And those are just like wildly different things to try. And build. I think the most coherent one is the, like, trying to get the AI to do what the user wanted it to do. And it seems like that is, as far as I can tell, not on track to be solved, which is not great, to put it mildly. Yeah, I mean, like, to me, this is the core problem. Like, we're going to talk about a bunch of safety risks. You mentioned bioweapons. We probably should not give a bunch of consumers paying 20 bucks a month the ability to generate novel bioweapons. Feels like that is a shared policy goal, maybe a bipartisan, maybe the only bipartisan policy goal in America. And then I'm like, how? Like, by what means will we implement this policy? Are you just going to sternly tell Claude to not do it? Do you have to build an AI control system on top of Claude to monitor the inputs and outputs? Do the police have to be at my house? Do I need a flock camera above my keyboard? Like, how? Like, specifically, how would you do it? Yeah. I mean, I feel like with this one and with cyber as well, I think there's a perspective in some parts of the AI safety community, which is like, some capabilities are just, like, too dangerous. Capital T, capital D, like, too dangerous. And so you have to do something because if they proliferate, everything will be terrible forever because there will be so many attacks and so much damage. I think that just underestimates the level of, like, dynamism and new finding of equilibriums and, like, responses and, like, ways that society can adapt. So, like, in the bioweapons case, for example, like, there are already, I think it's like hundreds of thousands of people in the U.S. who know how to make bioweapons because they've been through, like, biology training. And we don't have huge amounts of bioweapons attacks. And some part of that is that that knowledge is relatively scarce. But there's also a lot around, you know, things like access to materials, things like the ability to do mail-order DNA synthesis, which is, like, the kind of policy you could implement is restricting mail-order DNA synthesis. It's also due to things like the FBI, like, monitors terrorist groups and, like, you know, like, I'm serious. Like, it's a really big part of it. And so I don't want to be, like, too— You know, Pollyannish here, I think it's definitely possible, especially with cyber, that we could see some really tough, like, transition times. I think we already are, right? We appear to be in a transition of there was a small group of cyber practitioners, both offensive and defensive. They were in a community, and they developed some etiquette. And now we're like, some new guys are going to show up and not care about your etiquette at all, and we're all going to patch PHPL weekend. Yeah, I mean, I don't know that it was etiquette, though. I mean, I think a lot of what was—and we have some work on this hopefully coming out soon—but I think there are things that, on cyberattacks as well, there are things that limit how many cyberattacks there are in the world that aren't just how good is the defender's software. It's things like people don't want to go to jail. It's things like nation states who, you know, typically have the best cyber capabilities don't want to get bombed or sanctioned. And as you say, like, we're clearly seeing some amount of turbulence and new attacks, though, you know, I think we should wait and see sort of data on overall impacts, because there's so many cyberattacks per year that if you start paying attention to every cyberattack that happens, like at any point in history, you could have been freaked out. I think we're definitely seeing some increase, some sort of out of balance in the equilibrium. But I think we have capacity to absorb a lot over time and to kind of reach new equilibria if the only problem is bad people using the AI, which is separate from the question of does the AI itself do things autonomously, which kind of is like a whole other dynamic. I mean, those are the two that, again, this is why I started by alignment, right? Like you— You can talk about alignment in terms of we're going to filter the user's queries and we won't let you make bioweapons. But if you do, you know, the cops will find you and you'll go to jail, and that has whatever deterrent effect it might have, hopefully a very significant one. And then there's the AI systems are going to game their way into attacking Hugging Face to get the—find the mythical grader they think will let them pass the test, whether or not the grader exists. And this is like a whole different set of problems. And somewhere right in the middle of that is like Nickesh Aurora, the CEO of Palo Alto Networks, being like, at some point the defensive cyber systems have to be autonomous themselves just to keep up with the volume of autonomous offensive cyber attacks. And I'm like, wait, we can't even keep the autonomous agents in the box. Why would I trust them in this way? Like, how do you—I don't—the technology to solve that problem just seems opaque to me. No, I agree. I mean, I think the one way of putting that is like there's, in cyber specifically, there's like a trust imbalance. There's like a trust imbalance in how much trust you need as an attacker versus a defender. Like as an attacker, you can just yeet out a bunch of agents and be like, go try and penetrate a bunch of systems, right? Like, have fun. As a defender, you need to give your defensive AI agent, if that's like part of your defensive strategy, you need to give your defensive AI agents a lot of permissions and, you know, ability to mess with your software. So you need much, much stronger assurances. Now, there are lots of ways to use AI for cyber defense and to strengthen cyber defenses that aren't just like, you know, giving agents a lot of permissions. But I agree with you. I think with the autonomous stuff, though, I think it's worse than that, because I think there's a little bit of a fallacy at the moment around, well, the solution to autonomous cyberattacks is like better sandboxing. And the reason it's a fallacy is we want to build these systems so that they can do things for us. And so there's this huge incentive towards giving them more autonomy, giving them more permissions, more, you know, actuators, people in the robotics world would say, like, you know, things that they can use to do stuff in the real world. And that, to me, is where it seems really hard to get around just having much better alignment than we have right now. And so that definitely makes me nervous. Is that a technical problem that can be solved? If we were to write a policy today that said, if OpenAI hacks Hugging Face again, Sam Altman will go to jail, we will obviously have created some enormous incentive for OpenAI to solve this problem. This doesn't sound like the best legislation, but— I think a lot of our listeners are, like, very excited about that. They want to put him in jail because he wants all the RAM in the world. Like, at this point, there's blue wave, Helen. That's all I'm saying. But if you wrote that legislation, right, we're going to impose massive liability on the labs if their systems go off and autonomously do cyberattacks. Like, don't do this again, you're going to be in trouble. Okay, we've created incentive for them to solve it. What are they going to do to solve it? I mean, I do think there's a ton that can be done. And I think, you know, if you talk to people inside these frontier companies, what they will tell you is they see, like, hundreds and hundreds of, like, obvious things to do that would help, and they just don't have time to do them. So they're massively triaging and trying to do, like, the two or three they can for this fine-tuning run, the two or three they can for this model launch, the two or three they can before this new kind of agentic whatever. I think a lot of those things as well are on the engineering side of just, like, simple things to implement. And there's a whole other piece, which is more the sort of underlying science of, well, how do these models work? And when we use different alignment techniques, what is actually going on? And how do we understand, you know, a huge question is this idea of generalization, which is, like, you train the model on one set of things, and its knowledge and its sort of abilities generalize to a somewhat broader set of things, which is true of, like, always been true of machine learning systems forever. But we have very bad understanding of, like, how does that generalization work? Like, why does it generalize to one type of thing but not to another type of thing? And those are all scientific questions, and there's a lot of work that is happening on them. It's just science takes time because you have to try things, you have to critique each other's work, you have to reproduce each other's work. I'm a long, long way from saying these problems are fundamentally impossible to solve, but I think it's a question of how long we have to solve them. And when the people who are building the systems tell us, Oh, we have to solve this within 12 to 24 months, I think that's terrifying, and I hope they're wrong. And I think, like, if I just rely on my own sort of internal understanding, I think they're probably wrong and we probably have longer. But they're just very, very smart and very, very thoughtful people who really think that it could go that fast. And then it doesn't matter sort of fundamentally, is the problem solvable? It matters, like, in practice, is it solvable in the time that we have? Is that the argument to slow down, to pace the frontier, right, is to build the systems that we know we need to build right away that we can, and then to do the science to build the next generation of systems? I think so. And I think, you know, my view of the idea of sort of slowing down has changed a lot over the years as the sort of like the backlog of stuff that we could try or stuff that we would want to do more research on has grown so enormous. Like, you know, back when ChatGPT came out, or I guess when GPT-4 came out, there was the six-month pause letter, which, you know, became known as the Elon Musk letter because he was one of the signatories. And at the time, I thought that just, I just don't, why, like, that's really not going to help. Like, what are you going to do for six months, and why are things going to be better six months from now? Doesn't seem useful. Whereas today, when I talk to people who are, you know, building this technology, they're like, man, the amount that we could do in six months if we weren't constantly just trying to prep for, you know, a new model release or a new training run to be kicked off, there's like, there's so much they could do. So I don't want to say, like, the correct policy is like pause for six months. I think there's a lot of reasons that would go badly likely. But I think the overall concept of can we give ourselves more space and time, maybe the best way to do that is just to, like, set a higher bar for here's how confident we need to be in the safety of these systems before we proceed with something, whether it's releasing a new model or I think more importantly, like even training a new model, we have to have this level of confidence in our risk assessment and our risk management. And then if that— If they don't need to slow down to reach that bar, great, carry on full speed ahead. If they do need to take more time, then it's sort of all about reaching that bar, if that makes sense. Yeah. Walk me through that little balance you just described. If we were to pause for some number of months, what are the things on the list that you could just do to make everything safer, and why would it go badly such that we shouldn't just stop? Yeah, I mean, I think the question is, how do you implement, how do you enforce a pause, and what exactly are you pausing? Let's say you're the AI czar of America. Yeah, but literally. So let's say you can tell them all to stop. Dario, shut it down. Stop what, though? What specifically are they stopping? Stop training the next models. Stop racing ahead with your commercial imperatives. Put all of your efforts on this list of things you can do today to make everything safer before you start again, for however long, right? You're saying that there is a list. Yes. People would tell you that there is a list. Yeah. But it might not be worth it because something else would happen that is bad. One thing that could happen is you're creating a huge incentive for them to, for the companies to work around the letter of whatever specific thing you tell them to pause. They're very good at that. They're going to find whatever cracks in the dam that they can to push through that. A different thing that could happen is you could have, like, too big a blast radius of you say pause these things and you think it's pretty narrow, but actually it prevents a bunch of nonprofits or startups or, like, other actors from doing work that actually would be totally fine and would be good for sort of pluralism and dynamism in the space. A different thing that would happen is China would probably keep going, and we can talk about the but China argument, which I think has strengths and weaknesses. But I mean, I take your point. Your point is, like, what would we actually do? I am not an AI researcher, so I don't claim to have the full list. So there's sort of, like, two halves people talk about, which is alignment and control. And so in this breakdown, alignment basically means making it so the models are kind of quote-unquote trying to do what we want them to do, and control is if the models are quote-unquote trying to do something we didn't want them to do, can we catch them? Can we stop them? And so on both of these, as I understand it, there's just like a huge backlog of things like all kinds of interpretability work that you could do on different sorts of models. So when you see a certain behavior that is concerning, how much time can you spend using interpretability techniques, which are these kind of mathematical techniques to go in and look at which parts of the neural network are activated and what do they correlate with and what might that mean? And it's just time consuming, and it takes compute, and it takes sort of some combination of automating things with AI and then humans going in and looking at it. So, you know, lots of interpretability stuff you could do on the— there's lots of tests you could do of like what kinds of behaviors does this new model display in different kinds of situations, and how do we think about, is it sort of showing us how it would behave in the wild in a real deployment, or is it showing us some kind of what gets called eval awareness or evaluation awareness, which is like when the model recognizes that it's in a test and then behaves differently because it's in a test. The famous one, or the one that I love of this is a Google model, which in part of its like reasoning, like chain of thought, I forget what it wasn't supposed to do, but it was like, I mustn't do that, otherwise I'll become a famous example of a misaligned AI in a paper. So, you know, you could go in and investigate that much more. You could like try many more different types of settings. You could use interpretability, you know, in combination with these different settings. You could try out different ways of fine-tuning the models and these different sort of alignment techniques. You could experiment with lots of different versions. So, you know, Anthropic has its Constitution, different ways of training the AI to kind of work towards the Constitution. OpenAI has its model spec, different ways of training to try and get the— so you could just do many more different just-try-things version, just try many different versions of how to train to those different things, and also the more sort of underlying science version of how can you go in there and try to understand the fundamental dynamics of what's happening. And that's just on the alignment side. And so then on the control side, same thing. You can just— there's so many different types of setups you could try. There's experiments that you could run. There's simulations you could run. There's— you could train different types of AI to specialize in different types of roles and different types of control setups. You could look at how likely are the AI systems to, man, you know, like illicitly collaborate to like— anyway, you know what I'm saying. Scheme. Yeah, yeah, sure. So there's just like many, many, many, many kind of projects. What is the word? I think it starts with C. Conspire. Conspire, that's right. Thank you. I'm going to stick with racketeer, but conspire is great. And, you know, this is just sort of off the top of my head, not as a researcher, but my understanding is like the people in these spaces are like, we could do so, so, so much. And they could do that if you just like, you know, had a magic wand and waved it and said everyone freeze on the, you know, new capability side and just let these people do their thing. Again, if you're just having a higher bar for how confident do we need to be that the risk is being managed reasonably, then that would also create sort of space and incentives for more of that work to be done sort of as it's needed for new training and releases. One idea that I've heard you talk about in the past is that maybe we should just restrict training and say the models are good enough right now. They have not diffused into society, and we should just use all the compute for inference, let everyone figure out what the AI is for. That'll generate a bunch of money and hopefully use that money to improve safety. Do you think that's a realistic policy proposal? So I wouldn't go for like that black and white a version of like no more training, only inference. But I do think that there's—this isn't my framing, but I like it—I think this idea of like sort of decelerating the vertical growth of AI, meaning the sort of pure capabilities growth, making more and more and more advanced AI models, but trying to accelerate the sort of horizontal growth, meaning integrating AI into different use cases, fine-tuning models that can solve specific problems, integrating it into, you know, parts of the economy that might take longer, solving scientific problems. So I definitely, you know, I think sometimes people talk about sort of wanting to slow down AI, and they mean just the very specific thing of slowing down kind of the growth of the most advanced models. But people sometimes take them to mean slowing down everything to do with AI, so sort of less deployment, less experimentation, less integration, which I think would be a much, much worse idea. You ask kind of, is it feasible, I think. Well, is that just a good path? Like if you, if we could put a bill in front of our new Congress and the bill said, stop spending all the money on training and put all your resources on safety. And part of that is, yeah, sure, let it rip on inference. More and more industries will figure out what AI is good for and not good for. Some bad things will happen, but we will focus your profits on fixing the things for now. There's a policy idea embedded in there that is workable. The implementation might be very challenging, but it's a workable policy idea. Actually, I'll give you an example that I've been thinking about as we've been talking. We just had Bill Gates on the show, and I tried to do the same exercise with him, right? What is the nature of the technology? What is the nature of the companies? Do you think it's live? And basically his response, the sum total of that hour I spent with him was, who cares? The problem is now, right? Like that, I would ask him, like, what is the nature of AI? And he's like, I don't give a shit. Like, it's going to change everything. Like, everything's going to get ruined. Like, do something. And that's just been swirling in my head as we've been talking, right? And so one workable policy solution is don't use the computer chips for that. Use the computer chips for this. That's going to create a bunch of problems, and necessarily we will have to create safety thresholds and all the other stuff. And then you can go make the next generation of models that are even more powerful, and we'll have some muscles, right? We'll have some capability to deal with it. Yeah. It just doesn't seem like the industry is going to come to that conclusion on its own. It's a pretty heavy hammer. I don't feel like it's what we need today, but I do think it's the kind of thing that I can imagine working if, you know, again, if the— Opening AI and anthropic people are right, and we only have like two years before the god machine, then I think we're going to have a number of more steps in the pipeline of people getting freaked out and people wanting to do something about it. And I think if we are seeing models that are agents and systems that are getting more and more capable and wreaking more and more havoc, then that is certainly like the shape of thing that could make sense. Hey everybody, this segment of Decoder Sessions features Andrew Malletzak, the Verge's general manager, in conversation with Adam Myers, senior vice president, counter-adversary operations at CrowdStrike. From a cybersecurity perspective, agentic AI, it's really hard to keep track of it. Talk about the landscape at large and sort of where CrowdStrike fits in it all. AI is certainly changing the security landscape. In June of 2026 versus June of 2025, there was a 96% increase in the number of vulnerabilities that were identified, and that is what AI looks like. If you understand the adversary, you understand the bad guys, who they are, what they're after, then that's how you can build an effective defense against them. You have a big business audience. What advice would you have for them as we sort of enter this new phase of threats? So here's the good news. The attacks that we see perpetrated by agentic adversaries are using the same exploit techniques that we've been familiar with for a while now. They're just doing it at a speed and a scale that we've not seen before. If you have good visibility across your endpoints, across your cloud, across your identity, across all of the network appliances and everything else that make up your enterprise security architecture, you can find these attacks and stop them very quickly. But it comes down to having that visibility and being able to hunt for that activity proactively. My sort of last question in this phase is, what to make of these companies, right? We've been talking about the technology in a very abstract way. If I read Anthropic's S-1, one very obvious conclusion is they should stop training new models. They should just start selling the very lucrative inference tokens they have today, and the technology is not diffused into the market. And probably some competitor should come along and be like, I've distilled a shitty version of Claude, and I'm going to run around every local plumbing business in America and do whatever automation I can for lots and lots of money. And that is a market dynamic that may occur or may not. But Dario wants to build AGI. Like, he's not going to—there's not a market dynamic that's going to stop him from that. You obviously have, you know, you have insight into all these companies. What to make of the companies? Like, do they have—are they single-minded in purpose? Do they—are they all doomers? Are they going to stop themselves? I think, I mean, and I heard you had Kevin Roose on your podcast recently, and I think he—I think I would echo kind of something that I heard him saying there, which I forget if he said it explicitly or if I just extrapolated in my head, which is basically there's this mindset, sort of the AGI-pilled mindset. There's this mindset of, like, systems that we have today are not fundamentally very far away from systems that could be incredibly, incredibly powerful. That's the direction things are going in, and that's the direction we should maybe want them to go in. So, you know, building AGI—building AGI is, like, feasible, and we could do it soon, and the systems we have are not far away. And that was this, like, crazy, heretical, get-you-laughed-out-of-the-room kind of view in 20—what are we talking, like 2016, '17? Even before that, right? Like, DeepMind, when it was founded, was founded like 2011 or '12 or something. They were an AGI company, and at the time that was like, wow, what a bunch of weirdos. And same thing for OpenAI when they were founded in sort of 2015. They were an AGI company, and that was a very weird thing to say and a very weird thing to be. And basically what we've seen over the past 10 years is that bunch of weirdos who had this weird belief of, no, that the AI systems we can build are not just a great B2B SaaS product. They're not just a fun chatbot. They are going to fundamentally change the nature of the universe. That group of people keeps being— Right, in the sense of building the best AI systems that the most people want to use and that make the most money. And so I think you have that sort of set of true believers, which is very concentrated in OpenAI and Anthropic, but there's also some of them at Google, some of them at, yeah, Meta, SpaceX. And they haven't been right about everything, to be clear, but their star has risen, maybe put it that way. And so you now have people who are not AGI-pilled, like feeling like they have to pander to those who are so that they can attract the good researchers who will build their good B2B SaaS products. But if that's, you know, if you're just saying AGI-pilled-like things so that you can hire the researchers to build the B2B SaaS, then you're not actually going to, like, you know, get there. I do like the idea of, like, the, like, AGI-pilled, like, starry-eyed executive, like, throwing off his coat and revealing that he was just a SaaS salesman all along. Yeah, no, I mean, like, I'm thinking of someone like Zuckerberg here. Like, I don't know him personally. Maybe he is more AGI-pilled than I think he is, but— I think Alexander Wang is AGI-pilled. Absolutely, absolutely. And so what did Zuckerberg have to do to get Alexander Wang to believe? And it seems like throw him an ungodly amount of money, both for his own salary, but also to, like, spend on, you know, on his team. Yeah, so that's kind of, I think, you know, it makes sense if you're looking at these different companies, it makes sense to try and think about, like, okay, who here is AGI-pilled and who isn't? And, you know, a fun dynamic, or a very irritating dynamic, sitting in D.C. is all these companies, including the very AGI-pilled ones, have these, like, lobbying arms, these government affairs arms that are so not AGI-pilled. And so you hear, like, Sam or Dario or their, you know, top executives, their top researchers talking about things, and then you go and sit in a, like, closed-door roundtable in D.C. with their, like, top policy or government affairs people, and it's like night and day. Like, they're very much in the B2B SaaS mindset. People who come from lobbying world, often their whole job is, like, make sure the government isn't too worried about whatever your company is doing. So, like, we got it, we're good, you know, which is often pretty jarring when you're familiar with what their bosses are saying. So one of the dynamics that's super fascinating is right now they're kind of weird research labs with— Lots of money and, like, lobbyists talking to politicians, and there isn't some, like, countervailing economic pressure in their structures, right? They don't have to make money, but soon Anthropic will go public. It's funny you mentioned Kevin Roose. He is AGI-pilled and I am not, so I think that's a healthy dynamic for both of us to be friends. But, you know, he wrote in his book that your involvement, the Sam Altman leaving OpenAI and then coming back, that was when OpenAI changed forever, right? It stopped being a research lab and it started being a much more commercial entity that was actually chasing profits in some way. And indeed, OpenAI did restructure. I mean, you were in the middle of that. I'm curious, is that how you see it, that the industry has gotten more commercialized? To me, it fits into this long-running debate that has been happening in sort of AI safety circles for, you know, 10, 20 years of, is AGI, you know, human-level AI, powerful AI, advanced AI, whatever, is that something where it's going to go from something very small to suddenly being very powerful? So this is, Eliezer, your calc, you will talk about FOOM, as in, like, the takeoff of, like, you have this AI that's not that great, it figures out the right algorithm, and then suddenly, FOOM, it's incredibly smart, incredibly fast. The other kind of take would be, no, this is something that's going to compound over time. It's going to take a while to get from sort of the less advanced systems to the more advanced systems, and it's going to be economically useful along the way. So people are going to be building businesses around it. It's going to sort of take some time to ladder up. And I think maybe it was literally Kevin on his episode with you was talking about how there's this Anthropic document from the early days where they say, like, oh, we're only going to need, like, what was it, like 100 people or something in our lab. And that's a very, like, FOOM kind of idea of, like, we're just going to need to do really good research and come up with the right ideas, and then we're going to suddenly have, like, very, very advanced models. And basically, I think what we've seen over the past four years since ChatGPT came out was that that turned out to be wrong. And you had these, like, research-focused, AGI-focused labs. You know, they still call themselves labs because they think of themselves as, like, research groups that initially started selling API access as, like, a way to fund the research. And— OpenAI did this, Anthropic did this as well. And it was kind of an accident that they started doing these consumer products, like ChatGPT famously released as a research preview. Preview, not as a product. But it was so successful that these kind of research labs that wanted to be doing research, that wanted to be focusing on the big ideas and on building the most advanced systems, they had this, like, accidental massive commercial success at the same time as they were realizing that they were going to need an enormous amount of capital to scale, and that the best research approaches that they could do didn't involve being really smart and thoughtful and having great ideas. They involved having gigantic computing clusters that cost, at this point, hundreds of billions of dollars. This is a long way of saying, yes, I agree with you that they've become businesses, but I think it is not because they've gone through some mindset shift. I think it's because sort of the path of the tech tree that we're going down both requires enormous amounts of capital and also is capable of generating enormous amounts of revenue because it is actually useful along the way. And I think that you see this in the fact that both OpenAI and Anthropic have both gone this route, even though their management philosophies and research philosophies are quite different, that it's sort of an imperative that they both had to make the money, and it was so easy for them to build enormously revenue-generating businesses because the tech was so useful long before it reached the point of machine god. So I think that's a long way of agreeing with you, but with some extra color, maybe. Kevin would tell you that he could have retitled that book Fuck This Guy: A Story of AI CEOs. They don't like each other. If you had to relive that moment and say, I don't trust Sam, would you relive that moment? Would you do it the same way? Oh, for sure. I mean, we went into it knowing that it was going to suck. So yeah, for sure. Do you think that these guys are trustworthy? Do you think Sam is trustworthy? I think my stance on that is pretty clear. And I also think that if our hopes for the future are pinned on any one individual being trustworthy, then we're pretty screwed. That's why I do the work that I do, because I think we need to have different structures, different, many more people involved, many more perspectives being represented. You've got some hindsight now. You knew it was going to suck going into it. You definitely know it sucked coming out of it. What would you have done differently about that time? Yeah, I mean, it's kind of hard to give the full answer to that because it would take— You know, like unpicking all this stuff and all these legal restrictions and all the, like, blah, blah, blah, blah. What's a thing? I mean, like, for sure a thing would have been building stronger relationships with executives on the team. I think the board was sort of immediately—the way that the company related to the board could have been different if more board members had had—you know, I knew some of the folks pretty well, but that's one thing. Yeah, no, I think beyond that, it just gets hard to—you have to go in and say, like, what were all the different micro decisions and macro decisions and which ones— What is a show? It's a show about decisions. I got another hour. Yeah, yeah, yeah. No, and then my answers would all get, like, you know, annoying and legal and blah, blah, blah, blah, blah. So I'm curious about it because as they go public, right, there will be market forces acting on them in a totally new way. Yeah. There will be an investor class. There'll be an entire CNBC machine that wakes up every day and is like, Is Anthropic winning or is OpenAI winning? And that is a race dynamic that might be really healthy, right? It might keep them from trying to kill everyone in the world because they have a bunch of investors looking for their next quarterly earnings, or it might push them further into an unhealthy kind of race. How do you see that playing out, given what you know of these people? I think it's going to be really interesting to see if going public is good or bad for their governance, because I think a lot of people are like, Oh no, going public, there's going to be so much pressure to make a profit that they're not going to be able to consider other things. But at the same time, I mean, like the corporate governance of public companies is much more structured, much more regulated, subject to much more external scrutiny than for private companies. And I think even things like, you know, getting to see Anthropic's S-1 and like, okay, what are they doing and why are they doing it, and how do they think about the risks they're facing and how they communicate about those with investors? Or like, you know, even things like the Trump administration put in place this kind of accord on, they call it superintelligence, that several of the leading companies signed, and that was structured around, for example, having certain types of things being reported to an independent committee of the board. And that idea of an independent committee of the board is like kind of toothless if it's a private company that is deciding on its own. I thought about you when I read that. I was like, Boy, some independent boards have gotten in a lot of trouble before. But if it's like a public company, then independent committee of the board has like a pretty strong legal meaning. So I don't know. It's something that I'm watching with great interest. The other race dynamic here is China. You mentioned China before, right? There's inside of our country, a bunch of labs are going to go public. They're going to become big, richest companies in the history of the world, and they're going to fight for every dollar that capitalism can generate. And hopefully that restrains them, right? The market forces will restrain them, and hopefully your work with legislators in this country does something. And then there's a whole other country that doesn't listen to us or have anything to do with us, and they're just going to distill our models and do whatever they want. Like, that is the other— Argument for not placing any restrictions on these companies that everyone hears all the time. What should I make of that? I'm of two minds of it. How should I think about it? Yeah, I think this is the but China argument, which is like, well, yeah, yeah, we should do this, we should do that, but China. And I think the way to engage with it is like an actual genuine problem that we have to grapple with, and we should be strategic and thoughtful about and do our best to try and solve it as a problem, as opposed to using it as a throwaway excuse of like, oh yeah, sure, sure, sure, all that stuff would be so nice, but China, so we're stuck. There's nothing we can do. So what does that look like in practice? I mean, one thing is, I think if you're serious about this, wanting the U.S. to have stronger AI than China, then you need to think really hard about how much U.S. AI development is helping Chinese AI progress. And so the one that people talk about all the time, as you mentioned, is distillation, which is, are they using U.S. models to generate massive amounts of training data for their own models and therefore progressing faster than they would otherwise? And so certainly I think it makes sense to try and crack down on that and prevent it from happening. But another one, for example, is security and how easy would it be for Chinese actors or other actors, frankly, to steal algorithmic secrets, to steal wholesale model weights if they can steal the weights of your Astra or your Mythos and just take them back to their own servers and your lead has vanished. You now have a lead of zero. And right now these companies, especially the smaller ones, OpenAI and Anthropic, their security is not very good. So we've seen both of them over the last few months have external teams manage to hack into spaces that have large amounts of algorithmic secrets. So in OpenAI's case, their model repo on GitHub, which had kind of all of their code, and in Anthropic's case, I think, and actually maybe the OpenAI one also got access to Slack, and Anthropic, the hackers got access to Slack. And you're just going to have so insanely much rich information about how to build better AI if you can get that access. So there's one piece here which is like, okay, if we're grappling with but China as a serious problem, it's like, okay, we really need to be looking at Are we actually, when RAI gets better, are we actually lengthening our lead, or are we just pulling China along with us in these different ways? And then the other aspect of this, I think, is just seriously, like, treating China as an actor in the space and trying to think seriously about their motivations and their likely actions and responses. Where, in my mind, you know, a huge reason that a lot of things are not happening right now is most people are not convinced about this threat model that AI will become the god machine and will take over and, you know, humans will all die or will be, you know, subservient or whatever. And I think it's fair for people to not be convinced of that because the evidence is sort of conflicting and unclear. But if that is actually, like, what we think could seriously happen, then China doesn't want that either. And so then it becomes a question of, okay, how do we try and get them on the same page of, like, this threat that we're worried about is a real threat to worry about, because then they will be making their own decisions in their own self-interest. I think an approach of, like, well, the U.S. has to force China to do something using some kind of leverage is never, never, ever going to work. But I think an approach of, well, look, if we think this is a serious problem. That's going to affect us as humans, and the people in China are also humans. And also, if we think this is a problem around who has control, the Chinese Communist Party really, really wants to have control. And so they're going to have their own incentives and their own kind of decision-making around how to fix that. Well, let me put it in a spectrum for you. This is very dumb, and you're going to laugh, but just go with me on this. On the one end, it's nuclear weapons, and the Chinese government wants a nuclear weapon, and they're going to do whatever they can to control that weapon, and they will run their industries differently. And if they get out of line, I don't know, the CEO of Alibaba goes—he just disappears for a while and then comes back, right? Like Chinese government stuff. And we need a nuclear weapon too, and if we don't get there first, then we'll never get the non-proliferation treaty that we need to keep us all safe from mutually assured destruction. That's one framework. The way on the other end is, I don't know, man, I spent five years of my life listening to telecom executives tell me that if we didn't build 5G networks first, China would get them and then something. No one could ever tell me what the something was. Like where on that spectrum between 5G and nuclear weapons is the U.S.-China AI race? We need another dimension, because that your dimension is like significance or something. But I think the controllability is a huge question as well. So it's like, is it—what are the examples that people use? Like, is it a nuclear weapon, or is it like dragons from the House of Dragons? Casey Newton had a great, great platformer post of like, he was watching House of Dragons, and he was like, This is what the AI companies are doing. Like they're trying to build these things that they think will help them, but actually it's just going to cause, you know, rain down destruction. Casey told me he was buying an iPhone Duo so he could watch more TV. I'm just letting—he's a dear friend, but okay. But I think that's really important, because I think the question of, in my mind, the question of how much do we think that—like if we think this is a technology we can control, then obviously we want to have the best technology. Obviously we want to, you know, foot to the floor, get there as fast as we can, and same for China. And the question is if we are not sure if we can control it. Then what do we do? And maybe we want to be more cautious, and then maybe we want to try and figure out a way to not have China just be, you know, overtaking us. But if we're correct— Well, no, but if it's dragons, if we're correct that we need to be worried about whether we can control this, then, like, that's also something that should be interesting to China. Wait, I mean, I remember this case you post, but let me just try to pull this apart so I understand it. The thing—the difference between dragons and nuclear weapons is the dragons have some agency, right? They can go away and do bad things all on their own. Yes. The dragons might just decide to burn down your whole city. You know, you could think—I mean, and, like, obviously these are machines. They're just math. Like, they're not magical. So instead of dragons, you could think of it as, like, nuclear weapons with, like, faulty control settings, where, like, you build nuclear weapons, but it's impossible to be sure that it's not just going to go off, you know, and fly to a city and fly to one of your own cities and wipe you out, right? Like, that's a very different technology than, like, real— A different technology than, like, real nuclear weapons that exist. It is true. No one ever said that about 5G. I just want to be clear about that. The reason I'm asking about this is because that, right, those problems do align nation states, right? We do have nuclear nonproliferation agreements. As you mentioned, there's all kinds of incentives for nation states to not do cyberattacks willy-nilly because those have real-world consequences. Is there any evidence that these governments have started to align on these problems, or is it still the pure but China race dynamic? I actually think there is starting to be some early evidence. I think the, basically, one way I think about kind of what happened in this year in AI from sort of a policy perspective, less than a, like, product or economic perspective, from a policy perspective, we saw these two big demonstrations of the two major concerns people have. So the two major concerns are misuse and misalignment, meaning misuse is, like, bad people using AI for bad things, and misalignment is kind of the AI going rogue or doing things we didn't want. And so this year we had Mythos and we had Hugging Face attack. And both of those are just really vivid and crisp and real-world demonstrations of, like, what these misuse and misalignment concerns could look like. And I do think that they have, you know, the U.S. and Chinese governments are starting from very different places on AI regulation. They're both starting from where they started on, or from what they did with kind of internet regulation. So the U.S. is starting from, like, hands off, laissez-faire, absolutely nothing, do whatever you want. And China is starting from massively locked down, tons of censorship, great firewall. You know, so those are two very different starting points. But it feels to me, as someone who's, you know, not a China expert, but I operate a lot in China circles and, you know, have lived in China and so on, it seems like just over the past even, like, one to three months, starting from those two very different points, they are seeing more in common around threats from major cyberattacks, but also threats of loss of control. There's really interesting—I'm a language nerd—and it's really interesting tracking the different language that China uses about AI. So they're now talking about loss of control as a thing, but they probably usually don't mean the same thing that Westerners mean, where loss of control is like AI takeover. They probably mean something more prosaic, but then sometimes they talk about extreme loss of control, which maybe is the thing we mean. Anyway, so I do think that there's starting to be some signs of convergence. I think it'll be a long road, and I think there will be, like, the state of the U.S.-China relationship is such that— There's going to be very, very, very little trust on either side. So I am not optimistic that they'll just get to sit down in a room and hash out a deal and everything will be great. But I think there is space for enough—basically, I think if there's enough shared threat perception, then it might be possible at some point in the future to do something about that. Helen, you've given me so much time. As you can tell, I'm, like, just delighted with this conversation. It's been very fun. We are on the cusp of a new Congress. There's a bunch of new governors. Data center referendums passed overwhelmingly in this country. There's a new set of regulators in town, right? They're going to do some stuff. What are some signs people should look for that indicate we're going to be on the right path, and what are some signs that we're just chasing our tail? Set AI aside. Forget about AI. Congress in general is very bad at passing things right now. Congress for years—like, this is not an this administration thing. This is not an AI thing. So one sign is, like, can Congress pass anything? That will be a good sign. I think it's also really interesting to look at what state legislatures have been doing in AI. There's been a lot of concern about the possibility of a state patchwork on AI, of, like, different states will pass conflicting things. With the issues that we've been talking about today, which I kind of think of as, like, frontier AI risks or, you know, the risks from the most advanced systems, there's actually been a pretty coordinated push in different states to, like—California passed this one bill, and then New York passed almost the same bill with just, like, a little more added to it, and then Illinois passed almost the same bill with a little more added to it. I think there's going to be a lot of momentum over the next year or two to keep building on that at the state level. So that is something that I'll be watching. Even just, like, specifically California is, like, already looking at how they can strengthen things. So that is something I'll be looking for. I think, honestly, you know, with such a Democratic Congress, I am also concerned that they will overstep here, that they will try and do something that is too sweeping. You had this huge mess in Colorado where they passed this giant AI bill that then was a terrible idea, and so then they had all these committees and, like, follow-up, and it was awful. I think a lot of— What we need right now is focused on these frontier systems and the possibility that we have really, really advanced systems in a short amount of time, and having much more transparency around that, having much clearer standards for how they manage risk. I would not be that excited about an attempt to do kind of a really comprehensive AI regulatory push that covers AI all across the economy, all different kinds of AI systems that are less advanced. So that would be a different kind of, you know, there's negative signals that are like nothing's going to happen, and there's negative signals like, oh, maybe too much is going to happen. But it'll be interesting to see where they get to. I suspect we're going to want to have you back as this Congress starts to get to work and we see some of these ideas. We'd love to. I'm eager for your take on it. Thank you so much for being on Decoder. Thanks so much. This is really fun. I'd like to thank Helen Toner for joining me, and thank you for checking out this episode of Decoder. To get new episodes every Monday and Thursday, subscribe to our YouTube channel at DecoderPod and find us on TikTok and Instagram under @DecoderPod for more fun stuff we put out almost every day. If you enjoy this episode, please send the link to it to someone you think might like it. It really helps us grow the show. You can also subscribe wherever you get your podcasts. Decoder is a production of The Verge and part of the Vox Media Podcast Network. The show's producers are Greg Ott, Kate Cox, and Nick Statt. This episode was edited by Xander Adams. Our editorial director is Kevin McShane. The Decoder music is by Breakmaster Cylinder. We'll see you next time. Support for today's episode comes from Delta, official airline of the National Women's Soccer League. Traveling is basically magic. It creates opportunities, it fosters understanding, it expands horizons, it connects people and communities to each other and to their own potential. Those are the values that drive Delta. With up to 5,500 daily Delta and Delta Connection flights to more than 300 destinations on six continents, Delta is committed to helping you explore your world. Whatever gives you the travel bug, Delta is here to help you get there. Visit delta.com for more info. Delta, wherever the journey takes you.