AI broke the cybersecurity world. Now what?

Support for this show comes from Anthropic, the team behind Claude. Every business wants to move faster. Few can afford to be wrong. That's why nine of Fortune 10 now run on Claude for the work they want to get right. Safety and accuracy are not just line items for Anthropic. That's their whole mission. See what's next at claude.ai slash vox. Hello and welcome to Decoder. I'm Nilay Patel, editor-in-chief of The Verge, and Decoder is my show about big ideas and other problems. Today I'm talking with Alex Stamos, chief security officer at Pognition and a longtime cybersecurity expert, about what a mess security on the internet is right now. This episode is part one of a special series looking at AI in real life. I've always enjoyed talking to Alex because he's so direct, and you're going to hear him say that the multibillion-dollar cybersecurity industry as we've known it is essentially over due to the unrelenting AI-boosted waves of hacks, breaches, and attacks. And all of those are only the beginning. Both script kiddie hackers and also nation states like Russia and North Korea are breaking into systems all over, thanks to AI helping them both detect and exploit vulnerabilities all at the same time. It's got a real frontier feel to it. You're going to hear Alex say it feels like the '90s all over again, when the whole idea of an always-online world was brand new. That much disruption means there are a lot of places where bad guys can get in while everyone else is left scrambling. Alex isn't a doomer in the long run, but he does think we're in for a, quote, dark age while we try to figure out how to adapt to this new era. There's a lot going on in this conversation, weaving all the way to how AI-powered misinformation and slop might be managed on our social networks. Before we get going, do me a favor and subscribe to Decoder on YouTube, where we put out new episodes every Monday and Thursday. Okay, Alex Stamos and how AI is upending cybersecurity. Here we go. Alex Stamos, you're the chief security officer at Cognition. You're also a lecturer at Stanford University's Cyber Policy Center. Welcome back to Decoder. Hey, thanks for having me, man. I'm excited to talk to you. It's been a long time since you and I chatted in the, what you might call the content moderation era of internet policy discourse. I feel like you and I talked a lot about how all that was working, about election security, all those sorts of things. We're in a new era of big problems on the internet, particularly around cyber defense, cybersecurity. You've been thinking about this stuff a lot. You referred to the past few weeks as Security Awareness Month. Tell me what's going on. Just give me the lay of the land right now. October is, you know, traditionally Security Awareness Month, right? Like lots of companies run events and, you know, both internally and then tell people, you know, change your password. Use multiple passwords. Use a password manager. And what has happened over the last month is we've had, you know, a couple of these big bugs, PHP, Node, and such. What's been less discussed is a huge number of companies have been breached. You have a series of AI-related companies, the Neo Clouds, have been breached in succession. You have the Binance hack, which is the largest heist of anything in human history. You have the Cursor incident, where, you know, one of the things that's going on here is these hacks are being chained together. So it looks like, from what we know right now, one of the companies that was broken into was Fireworks, which is a prominent provider of AI services to a number of companies. From the Fireworks hack, API keys were stolen that allowed people to break into Cursor. Once they got into Cursor, they then pushed a bad plugin, which went to 5,000 more companies, and that caused 5,000 potential breaches. So it's just this kind of permanent, continuous breach that is going on throughout the tech industry. And one of my friends said, a CISO friend of mine said to a bunch of us, Is anybody else tired? And everybody said yes, because effectively we have been at DEF CON 2 for all of October and now going into November. So I'm going to ask you some really dumb questions here. Who is behind all these attacks? Is it Russia? Is it North Korea? Do we know? It's a variety of different groups. It is not just because adversaries have decided, but what has happened is, from my perspective, we are at the start of the takeoff of adversary capability enabled by AI. That the open-weight models have now gotten to the point at which you can automate most of the kill chain. You know, so this kill chain term is something we stole from the military, right? In the military, they mean to actually kill something. There are the steps you do to blow something up. In cyber, the kill chain is the things you do to have an effect on a target. So to steal information, to turn something off, right? To steal money, to steal Bitcoin. Yeah. Traditionally, you, you know, do research up front. You look at a piece of software. You find a vulnerability. You write exploits, and you do that all way ahead of time. And then you have a human being go and take that stuff you built and deploy it against either one target, because maybe you're a government and you're going against Lockheed Martin, or you're going against another government, or you're a ransomware actor and you take all that work you did and you deploy it against hundreds or thousands of targets really fast. Well, now with AI, what's happening is this kill chain has inverted, where you can run an AI system that just goes hog wild, and it looks at the targets and it finds the bugs in real time, and it writes the exploits in real time, and it breaks into these companies. So that's one of the things that's happening, is these systems are going and they're so good at finding bugs that they are finding the bugs in real time and writing the exploits in real time. And so the system that everybody's talking about is not the only one, but the system everybody's talking about is a thing called Artex, which is a Chinese open source tool that anybody can go look at. There's an English translation on GitHub that you can check out, and if you power it on the back end by something like an obliterated GLM-5-3, it will let you break into a shocking number of organizations. This started with three South Korean banks. This week, about 200 more companies have been hit. I've been helping a couple of companies. Cognition has been kind of, you know, we're not an incident response company, but we do go in and, like, help people fix their bugs afterwards by using our tools. It is shocking the companies that are getting hit by something you could just download. It has been a long time since you've been able to go and just download an open source tool and break into multibillion-dollar companies. It's like the '90s, right? Yeah, that's one thing, like Windows in the late '90s, early 2000s, when it was a security disaster. I started saying this actually in the beginning of this year, which is like, it's an incredibly exciting and terrifying time, because the last time it felt like this was when I was a much younger man, and I would go to the very early DEF CONs. DEF CON these days is like 40, 50, 60,000 people. Back then it was a couple hundred people, and it was one track, and you'd go in and somebody would give a talk, and you'd leave the room and you'd be like, Every company in the world is vulnerable to this attack. The difference was is the entire world didn't run on the internet, right? So it's like you could go do bad things, but you couldn't ruin the world, right? And you certainly couldn't automate it in the way you can now. And it's back to those days where just like tools you could just go get and build yourself can go do crazy things. It's back to the Wild West days, except we actually, back in those days, there's like a couple hundred people who had those skills, and we were all in one room in Las Vegas, and most of them were actually good guys. They didn't want to be criminals. And now you have a much larger ecosystem of people, many of whom live in places where the law enforcement can't touch, and who have a big economic incentive and have access to these tools. And the other big thing is they no longer have to use frontier models that have protections, that have logs that are available to law enforcement. They can—I have a M5, my M5... Mac Ultra Studio finally showed up, and I've been running, like, a quantized, obliterated GLM 5.3, and oh my God, like, what you can do is incredible. You mentioned the economics of frontier models, just the economy of cyber is one of those things that has gotten sophisticated over the years, right? So you're in these early days, bunch of people in a room. One of the things that our systems got safer is you could pay the number of people at DEF CON a whole bunch of money to protect all the systems, and it was economically smart to pay them to be defenders instead of attackers. And then I'm just rewinding my own brain through this history. There was, like, a long conversation in our industry about security through obscurity by hiding all the bugs and hiding all the code versus open source and being really transparent. There was an etiquette that developed about how to reveal the bugs and when to give people notification. There was transparency reports and bug bounties. Like, we built—you could call it an etiquette, you could call it an economy—we built a culture around cybersecurity and cyber defense and how the attackers and defenders would operate. What you are describing changes it all for a whole number of reasons, but the one that sticks out to me is the economics of it, right? I can go buy a Mac Studio, I can get an open-weight model, and I can just start. And that does not place me inside of the culture. There's not going to be some old man, there's not going to be Obi-Wan Kenobi being like, Here's how you act like a cybersecurity Jedi. Like, I'm just going to start doing stuff. Tell me how that's changing right now. Effectively, the last era of security, from my perspective, started with the Microsoft, the Bill Gates trustworthy computing memo in the early 2000s, and that is when we started designing how we do software security, and— These big companies, Microsoft and, you know, Oracle, and then later Amazon and Salesforce, and these folks started hiring ex-hackers. And I was part of that generation of the ex-hackers who were, like, at DEF CON and Black Hat, got hired by the companies to go break their stuff. And we started coming up with responsible disclosure, bug bounties, consulting companies, so the Foundstone and AtStake, and then Mandiant and CrowdStrike. These companies grow up, and we get this huge industry that is now RSA and Black Hat, and this multibillion-dollar security industry grew up in the 2000s and 2010s. That era is over, like you said, because it was based upon a couple of assumptions. One was that there was a hierarchy of skill sets of people who could do this stuff, that you had the script kiddies who could download tools to do things, and they're then the next level up of the people who could write those tools, and that that was a more rarefied skill set. And then there was the kind of vulnerability researchers. And then there's kind of the gods. And that god tier was in the hundreds of people, and that is the people who could do the really, really good vulnerability research, who could write the really, really good shellcode, who could reverse engineer patches from a Microsoft or somebody like that. Also, the assumption that there's a limited number of people who could be operational in an attack. And so if you wanted to build an attack team, that you have a small R&D component, and then you have a much larger operational component. And so if you are a professional attack team at U.S. Cyber Command, at Russia SVR, at Chinese MSS, that you build this structure of your R&D team of elite people, and then your less elite operational people. And then that was copied then by the Russian ransomware actors, and that you have to have a conspiracy of people to kind of do this stuff. A lot of this stuff also fell apart with the invention of Bitcoin, because one of the assumptions was that you had to have people who could launder money, and the Bitcoin meant that nerds could launder money. So that started to change the economics of the entire thing. All this stuff is falling apart because of AI, because the skill set issue is that now script kiddies have a bunch of the skill set of the DevIts and the other cyber gods who could do these things. That with this Mac that's sitting on my desktop, I can reverse engineer a Microsoft patch and walk through and then write stable exploit code. It's not quite there yet to be automatic. You still have to have a bunch of skills, but we're really, really close. And then what's really concerning, and what you see with the APT stuff, is you can now kind of let loose on the operational side. We were never secure. It was all like a fantasy, right? These companies, these vulnerabilities have always existed. These companies have never been safe. And that is what we have seen over the last decade, is these mostly Russian ransomware actors nailing kind of the midsize companies that most people have never heard of, but are actually really important, like some midsize industrial company that makes some widget that we all depend upon. Depend upon. I once did work for, like, the largest distributor of medical and dental supplies in the country, and they got really hit badly by a Russian ransomware actor. And then, like, dental offices across the country were, you know, canceling appointments. Didn't even make the news, right? But because they couldn't get gloves, because they couldn't get supplies. And it's because, you know, a company like that's got, like, one security guy and a 10-person, 12-person IT team, and they're going up against a 20-person Russian ransomware team. They're toast, right? But it's just because there's only a handful of Russian teams that could do that. Now two or three Russian kids can have a whole bank of Macs, or they have the money, they can buy Nvidia H100s, and they can run a ton of agents who do this all day, and that is what is changing. So, I'm sorry, you asked who is doing this. For Binance, it's the North Koreans, and they have an entire team that all they do is steal Bitcoin. Other people have gone out of it because it's, like, really hard to compete against a foreign intelligence agency that recruits their smartest kids and all they do is steal Bitcoin. They're extremely good at it. And for the Binance hack, we can talk about it. There's some, probably some AI component, but mostly that was actually human intelligence and very, very smart social engineering. And the North Koreans are—I mean, you have to give it to them—they're very, very good at that. But for the other stuff, it's— It's, it looks like for the people who are breaking into the AI companies is what's called the Chinese cottage access industry, which are small Chinese contractors who break into companies and then sell the information to much larger Chinese companies. That diffusion of skill changing the sort of economics of cyber is really fascinating to me. We ran a headline ages ago called Attack of the Killer Script Kiddies, like the very beginning of all this, and I feel bad because it is much more suited to now than it was to then, because then it was just a lot of, like, automated mischief. Because you wasted the headline. You wish you held it back. You know, every now and then you get a lightning bolt of a headline. You gotta, you gotta get it out there. It's a great piece of art. You know, we'll put it in the show notes. But it feels like the diffusion you're talking about is not, oh, a bunch of teenagers are going to be even more malicious. It's a bunch of bad actors with real incentives to do harm are now vastly more capable than they were before. And that doesn't seem like you can rejigger the economics of that unless you somehow make it really expensive to do the attacks. I think it's going to be extremely hard to make it expensive to do the attacks. We're going to have to lower the cost of defense. And you're completely right to look at this economic problem, because what's happening is attacks are becoming agentic. The cost per token for— Going towards zero, and the cost for defenders is staying quite high. And so it is very important for us to find ways for defenders to have cheaper options. And I think that is critical. So that's something I am personally extremely focused on. And I think, to be frank, very few people are. There's a couple of problems here. One, there's a ton of focus on the defensive side of AI on finding and fixing bugs. That's great. It's important. But software security as a CISO, like I work for an AI software company, Cognition. Even as a CISO of a software product company, software security is 25% of my job, right? I have all kinds of other responsibilities. And software security is the part of my job that has been the most both revolutionized and improved by AI. The other 75% has been helped a bit and in some cases made much worse. So there's so much more we need to do than just hill climb on finding and fixing bugs. There's been so much less hill climbing on all these other things. And the next area needs to be the operational security side. Give me an example of the other things, the 75%. Okay, so like the operational security side is watching your logs, looking at, ah, somebody is running our techs against us. I am now going to trigger an investigation. I am going to look to see what is going on, and I'm going to make an intelligent decision of what I'm going to do to respond to this without shutting down the entire company. That is SOC work, that security operations center work. Companies that want to survive this era are going to have to replace human beings who do that work with the first tier and then eventually the second tier. You can do that with current LLMs. The first tier actually has to be not an LLM. It has to be something much dumber because you're looking at terabytes or petabytes of logs, and it is just completely uneconomical to do that with actually an LLM. We have other things, TF-IDFs, ML-BERTs, traditional classifiers that can operate at much faster. But then that's a soft alarm, and then you send it to a small LLM, and then the small LLM calls its bigger brother, and that might call an even bigger brother, right? But for those models, we need to do a lot more work to optimize them, and we need to make them cheap, right? And I have a colleague who works at, like, one of the large security platform companies who said that, like, doing an investigation on one of the big frontier model cyber models costs on their product $900 to do an investigation. That is not going to work when the Russian ransomware guys are either using Macs or they're using, you know, thrown-away crypto mining rigs, right? And their marginal cost is zero. So this is, like, one of the things we're working on at Cognition, is, like, we train our own models, retrain open weights, and we're trying to drive this cost way down. It's something we've done, but we have a lot more work to do here, and a lot of other people have work to do here. And I think this is where, like, as an industry, we have to focus on, like, the Pareto frontier between cost and efficiency on cyber, because it's not just bug finding. The other place here is just, like, people talk about Mythos. Mythos is great. Astra Cyber is great. They're incredible models. But even the people who have access to it, which is a tiny number of companies, they only scan, like, their most important code. And there are hundreds of billions of lines of code out there that are not the Linux kernel, that are the crap. C Sharp and J2EE code that was written by Infosys 15 years ago, all of that code is not going to go through a frontier cyber model that costs a gazillion dollars. And so that's the kind of stuff that we need to work on cost efficiency for. Support for this show comes from Anthropic, the team behind Claude. Somewhere between the third meeting of your day and the 14th browser tab, there is not a lot of room for the work that matters most to you. Claude makes room. It takes on the forms, the follow-ups, and the deck that needs reformatting. You get back to the parts that actually need you. Bring Claude into your work at claude.ai/vox. It feels like you're describing this cliche that I hear all the time. And again, I'm just going to keep asking really dumb questions about all this, but the cliche I hear all the time is everything is going to get so much worse before it gets totally secure forever. And it doesn't seem like the model you're describing leads to that result, right? We're not going to fix all the bugs forever in some valley of despair and then always have perfect code. At some point, there's some cost advantage to the attacker always, right? Like, you're running a company. You can't write defense that has zero marginal cost. Like, someone has to do it, even if the attackers are so infinite that their marginal cost is effectively zero. So are we going to go through a period where everything is so much worse and then it's perfect forever, or how does the cost manage itself out into balance? I am neither, like, the hardest doomer or the most— Positive person about this, I feel like we're not going to P doom, right? Like, we're not all going to die or be uploaded to the singularity in the next two years. Nor do I think when things get better, they're going to be perfect, right? The metaphor I've been using is we are entering a dark age, right? Things are going to get more bleak, and our responsibility of those of us who work in cybersecurity, and especially those of us who work in cybersecurity at the AI companies, and we are a lab if a smaller one compared to the big L labs, it is especially our responsibility to help shorten that dark age. So the dark ages were bad, but we survived them, right? And then we had the Enlightenment and the Renaissance. But it's not like the Renaissance was perfect, right? Like, it wasn't, you know, utopia and everything was perfect. We will still have hacks. All kinds of bad things happen, even when you don't have a security flaw. There's tons of security breaches that do not require a flaw in the software itself, and we still will have flaws in software, even if LLMs are writing all of them in strong languages. And it will take a long period of time to get rid of all the flaws. I do think, however, things will start to get better as the median code becomes code that was written by an LLM in a memory-safe and type-safe language. But the problem is, is we have a couple of decades of stuff that human beings wrote, and that wrote in languages that we should have replaced years ago. And my feeling is, is to shorten this age, we have to do a couple things. One is we have to stop to get out of this, like, we're just going to find bugs and patch them. A bunch of stuff just needs to be refactored, right? Like, if you're looking at your code base and you scan it and you've got 10,000 bugs, then probably now we are at the point where rewriting it is actually cheaper. That is my observation from looking at these customers of ours, is that— If you can get a working build, then it is actually much easier to write a whole humongous suite of tests and to spec out how this thing works, and then to rebuild it within that spec and those tests securely than it is to find all the bugs and fix them. It actually is. And so I don't know what is going on through the mind of the Linux kernel developers, but that thing, again, that talk I gave, like at the beginning of this year, I posited two possible futures. And the one was we're now at superhuman bug finding. Maybe we just have a pool of superhuman bugs and we're going to drain it. That's the good future. But the bad future is every time we get a new model, there's a new— Superhuman bug pool that's a little more superhuman. And unfortunately, we seem to be closer to that, and that the Linux kernel got a bunch of fixes for Mythos, and then Mythos Preview, and then we had Mythos 5.1, and there was another pool, and then we got Astro Cyber, and we just got another thousand Linux kernel patches. It's like, it's not slowing down yet. And I don't think we're just going to scan our way through the C code of the Linux kernel. There's only like two companies, honestly, in the world who could probably do this. It would be Google or Amazon. I wish instead of like the Fuchsia project or rebooting the Google Book or the Chromebook again, Google would take all of those incredibly brilliant engineers and they just tell them, take the Linux kernel and rewrite this in Rust. And that would be their gift to the world. And it would be an incredible advertisement for Gemini for Carbon or whatever. It would be like, get rid of all the drivers that nobody needs. Like this thing just has to run on the 4060X3 or whatever. I mean, I know it doesn't, but you know what I'm saying. And then just release it and call it whatever you want, the G Linux kernel, just like they did this with OpenSSL, right? And I would run that, right? Like Cognition would run that on our servers, or have Google and Amazon do it together, and it would cost them $100 million. But compared to the societal cost that this is costing us every time we have to patch the Linux kernel and you have to reboot every Linux machine on the entire planet, that's what has to happen. But for whatever reason, the technology industry has not been able to get together and make these hard decisions of these fundamental things that underlie the entire human race. We have to completely rebuild. This is why I started by talking about how the economics of cyber led to the culture of cyber. I mean, this is like the most decoder way of approaching this conversation, I know. But, you know, when you describe the research teams at the NSA and then the operational teams, that is an org chart that comes out of economics of the work. You're saying, oh, someone has to make a big decision here. You need to look at it from a very high level and say it is cheaper to spend $100 million to— To fix Linux, rewrite Linux today than it is to successively patch new bugs when every new model comes out. I don't think anyone's in that position. Like, I don't think Linus Torvalds is in that position. I think Sundar Pichai and Andy Jassy are. I think they literally have each other's cell phone numbers. If one of them—maybe they don't listen to Decoder—but I expect one of their direct reports does. And I would like one of them to go into their one-on-one and say, Sir, I listened to the most handsome, best-spoken CISO I've ever heard, and he had a great idea. And you could get incredible credit if you're the one—yeah, we'll both do it, but we'll leak to the Wall Street Journal that was your idea first. Yeah. You should call the other, and we should spend a hundred million, maybe a billion dollars. Like, we just had a bug in the TCP stack of Linux. That is insane. Like, we can't do this anymore. The problem with rewriting the Linux kernel is like half of it's driver code, and nobody has that hardware anymore, right? Like, somebody had to be in the presence of that hardware with a, you know, a ThinkPad T40 when they wrote that driver, and they don't have that anymore. But the good thing is, is a huge number of the really important uses of the Linux kernel are production server hardware. And so fork it, right? Like, Linus will never do it because Linus wants to support all that stuff, and he's got, like, all these politics. But Google and Amazon don't have to care about Linus Torvalds, right? And so Google and Amazon can create a new version, a new fork of the Linux kernel, and they don't have to call it Linux. That can be ABI binary compatible with glibc and everything above it. In fact, they can eventually fork glibc, right? Yeah. And they can build it, and they can call it whatever they want, and it only runs on production hardware, and then they can use it for their clouds. And eventually all the clouds will move to it. And if you want— Other people can run the insecure thing that has to be patched all the time, and that's fine. But, like, for the things that our containers run on, I would feel a lot better if it was written in a type-safe, memory-safe language that also has the ability now to have contracts. So, like, these languages have the ability—you don't have to do full formal verification, which is incredibly hard, but you can do kind of these things that approach formal verification. And I'm a little over my skis here because there's people much smarter than me that study this. But basically, you define this is how this function is supposed to work, here's the contract it has. And very, very smart software engineers would love to write these things because, you know, they had time on their hands. No person who had a real job used to write these things, but LLMs have tons of time on their hands, right? Because, you know, we're frozen in time while they're, you know, thinking at incredible machine speed. And so you could totally have them rewrite the Linux kernel with this, you know, not quite formal verification. Please, if you're listening and you work at Google or Amazon, go pitch this. Anyway, there's my pitch. I'm just saying, I know Jason's listening, so if you heard it— It's not just Linux kernel, but I'm saying, like, that is the kind of thinking we're going to need to make it through this, right? It's like, we can't just patch our way through this because it's killing us. And because the other thing that's killing us is companies can't—we are not set up as—you talk about, like, how we do security, like, for the last 20 years. Kind of the assumption is, is like, we can get a patch and apply the patch and you're okay. And what's happened is, is the exploitation timelines are so incredibly fast that that does not work. That's what we just saw with the PHP bug, is we got incredibly lucky that the PHP bug turned out to be really hard to exploit in the wild. If it was a much more wormable bug, it would be a disaster. And so that was the kind of near-miss asteroid thing that now the world should, you know, build an asteroid killer. Wait, can I ask you about this? Yes. Yeah. We should definitely put Bruce Willis on a space shuttle. Don't wanna miss a thing. There's a lot going on in this conversation already. My pessimism about all these things is very simply expressed. We are, as Americans in particular, we are very bad at doing anything until a bad thing happens, right? We love a cure. We hate prevention. A lot of bad things have happened. There's the Hugging Face attack. You actually, I think you recently said you were happy that happened. It woke people up. There's the PHP bug. You know, we've just had a crazy month of all these breaches. It doesn't seem like that has led to a whole-of-industry response or a regulatory response or Sundar, Colin, Andy, and being like, We're doing Linux. What's it going to take? How bad does it have to get? The Hugging Face incident was good, and it did wake up a lot of people in security. We are a little bit better off because Hugging Face happened and we saw what was going to be possible a couple months later in open source. But you're totally right. As Americans, and Americans are the only ones, honestly, who can lead the world on this, we're the ones building the AI for the most part. We're the ones who lead the cybersecurity industry. The Chinese are mostly enabling the attackers, not the defenders. They should enable defense. They want to be participants in the global economy. They want to be leaders. They should be doing more on the defensive side, not just enabling the attackers. But that's not happening. So anyway. But we need to be proactive. And look, something I have learned now working at Cognition, at an AI startup, you can just do things. Like, we need to—seriously, you can just go do things. And we just need to have more agency as a tech industry. We should just go do things. That should be the— Lesson of the AI era, and we should not be so risk adverse in doing the right thing security-wise. And I think that's what security teams, if I had a message for other CISOs, is just go do the right thing. Because CISOs are, by our nature, we are risk— They're the people who manage downside risk. We have all L and no P in the P&L. And I have had to personally, like, break through that natural inclination because, like, you just, you can't just play defense. But this is my prevention versus cure, right? The prevention is cheap, but it's all loss on the P&L statement, right? It's all just cost. If the bad thing happens, you're like, well, we have to do anything to fix this because there's economic benefit to having a functional business again. And it seems like in this moment, even the sort of, like, basic understanding of that risk versus cost is totally scrambled and no one has quite figured it out yet. Yeah, I totally agree. So I'll give you a very practical example of a way CISOs and their bosses are going to have to change on this. Every company in the next—I was going to say six months, but really in the next 90 days—is going to have to RTFX themselves. You're going to have to run the attack on yourself. Yes. If not with real RTFX, you're going to have to use an AI red teaming product that is functionally equivalent, or you're going to have to hire a company to do that for yourself. If you do not, you will get breached. Yeah, it just—that's terrifying, right? Because these things are completely non-deterministic by definition. They are very smart. They are very fast. They are so fast that no matter what, if you hire, like, an AI pen test, AI red team company, they will promise you verbally, Oh, it is completely safe, and then they will give you a contract that says, Our product will give you cancer, will emanatize the eschaton, and will likely end the human race, and you hold us harmless when any of these things happen, right? So there is a disconnect between what they will say verbally to you and what their lawyers have written, right? Because that company has by definition— Only existed for six months, or at least the product has existed for six months if it is an older company. They have no idea what their product is going to do when it meets your software. They have no idea, whatever they say to you. But you have to do it, because some dude is going to do it to you, and you need to know what is going to happen and fix it. Because at least maybe those people will break your system. They might cause downtime, but at least what they're not going to do is steal your data and ransom it, or intentionally shut down your blood bank refrigerators. That is what happened in France, right? So that is exactly the kind of thing that a CISO, a conservative CISO, would never even ask to do, and a conservative company, even if they have a risk-taking CISO, would never let their CISO do. And so what's going to happen next six months? A ton of companies are going to get breached by some 19-year-old running Artex and then buying tokens off OpenRouter, or running it locally on their gaming PC, right? They're going to turn off Call of Duty, run Artex, then turn Call of Duty back on, right? And while they did that, they broke into a bank who their policy said, Oh, we can never red team ourselves, or they said yes, but they ended up in a 30-day negotiation because their lawyers are like, I want to have—you have to have a hundred million dollars in insurance, and we want to have, you know, this kind of—like, they say yes, but their vendor risk management process and their legal process is so onerous, there's no possible way they could actually accept the risk. That is my message also. You are going to have to take that risk to actively red team yourself, to front-run the bad guys who are doing that right now, right now, as we speak, and as people are listening to this. 24 hours from now, you possibly have an AI system who is trying every single API endpoint and website that you might not know you have because it was like a marketing site that was set up a year ago for an executive event, and is going to find a bug and get in, and then is going to move incredibly fast through your network. So you'd rather pay somebody or have your own team do that. You need to accept that risk. Almost no company larger than a couple hundred people has a risk management framework that will allow them to do that. Support for the show comes from CrowdStrike. Here on Decoder, we're peering beyond the traditional tech realm and into all the places AI is shaking things up. Cybersecurity is key to that conversation. CrowdStrike gets that. That's why AI detection and response, or AIDR, is emerging as a new category of cybersecurity for the AI era. It helps organizations see, monitor, and secure AI across the enterprise. CrowdStrike. We secure AI. We stop breaches. Go to crowdstrike.com/decoder to learn more. There are several costs associated with this. And one, I think, is just like the most common in business, which is, well, if we find out there's a problem, we'll have to pay to fix it. So what if we just wait, right? Like, you can see that coming. What if we find out that we are catastrophically in trouble, and then there's how much will it cost to fix it, and then there's what will we do to fix it? And the argument that I've heard about what will we do to fix it is, well, we'll just unleash a swarm of defensive AI tools that are watching the networks and watching the code, and they'll make a bunch of autonomous decisions, and they'll fight the autonomous script kiddies who are armed with ArduX on their Mac Studios. And something about that argument, in my head, I've heard, for example, Palo Alto Networks CEO Nikesh Arora has made this argument, right? You need to give the defensive agents some autonomous capabilities. And I'm like, well. Hold on. Like, OpenAI can't control the swarm that went and attacked Hugging Face. How would I control the defensive agent from doing something equally damaging to prevent an attack? And I don't think that there's a technical solution there that makes it clear what to do once you found all of the vulnerabilities. A couple of things. One— I told you I was going to only ask dumb questions. I just want to be clear on that. No, no, that is a fantastic question. You have not asked a dumb question yet. I'm honest. You have not asked a dumb question yet. That's a fantastic question. Okay, I'm going to give you a shorthand that you can use going forward. If somebody actually uses the term swarm, then— I don't know, to be clear, I don't know if Nikesh used the word swarm. Okay, I'm not saying he did. And I have great respect for Nikesh. He is one of the best businessmen in all of security. So this is not on him. But swarm is used all the time. Yeah. But when I hear somebody use swarm, I immediately discount all of their technical opinions by about 80%. It is going to become, it is already, and it will become the most overused term in all of security. The actual quote-unquote swarm from OpenAI was something between, at different times, 700 to 1,400 agents. That had to run on— it's hard to tell because we don't have the actual, like, size of the models, but, you know, 300 million to almost a billion dollars in hardware. That doesn't include the actual data center cost. The actual Hugging Face attack was, I did some back-of-the-envelope math, maybe a million dollars in electricity, right? No attacker is going to do that. And in the end, that was stupid because— These things were actually trapped. They were not prompted to attack Hugging Face. Most of them were doing something totally different and were kind of enlisted by the one thing that actually wanted to attack Hugging Face. They used a tiny fraction of their token budget to help their little friend, and then they had to tunnel their way out. So it was the most inefficient way to possibly attack Hugging Face. Almost nobody is going to be attacked by a swarm. So if somebody says swarm, they do not actually understand at the bottom technical level what's going on. The Artex attacks are almost certainly one single agent, maybe two. By the way, I was the person who said swarm, just to be clear in this one. Right. And so now you won't, and you can use that as a good filter for your future guests. It is driving me insane because it is everywhere. Right? It is absolutely everywhere. A couple of things are correct. One. We are going to have to turn over defensive actions, at least the first couple of levels, to AI. That has already been true just because, you know, any reasonable-sized organization generates terabytes of logs, if not petabytes of logs. So you already have AI, if not LLMs, much more traditional AI like classifiers looking at it. But then actual decisions need to be turned over to AI with human oversight. That is true. There is no way to just release defensive agents on your network, but you can build defensive agents that are plugged into lots of things. They are plugged into your logs. They are plugged into your EDR. They are plugged into your cloud, and they have the instructions on what they're allowed to do. My preference is there are terministics limits to what they're allowed to do. Those are hard to necessarily create ahead of time, but they are important. So, for example, we are building all this on top of Devin, our own SOC, right? But we are putting in place deterministic limits, like it has the ability to turn off an account in Okta, but only so many accounts, so it can't turn off the entire company. Like if it goes nuts and it says, Everybody's a North Korean, it can't turn off everybody in the company. If it hits its limit, you know, no matter what, we get paged. But if it hits its limit, it pages us a lot and it says, I hit my limit, right? And then a human being has to then approve anybody above that limit, right? Obviously, if it turns off five people, then something has gone really, really wrong, right? And so, yes, you need defense, but there's no way to just, like, let a bunch of bots loose to do all the defense. The only way we're going to make it through this is you have to autonomously fix software bugs. So I am currently helping a company where we are onboarding all of their repositories. We are scanning with Mythos and a bunch of other models, and we are finding and fixing thousands of bugs. Verifying with tests that we are writing, merging those PRs automatically. If you are renovating old code, the only way you're going to survive this era is to do that on an automatic basis. You can't. That would take years with humans, and you can do it in days. And certainly that is the place I would start. And then the next step is the operational side, for sure. It feels like I just keep swirling back to the economics of all of this. But it really feels like the other way you could solve it is by saying to the big model companies, if your models are used to go attack a company, you'll be liable for it, right? We're going to make it really costly, and then they should figure out whatever alignment that they need to figure out, whether or not that's even possible. And the pressure on the open-weight models will go up because we'll all be paying the frontier companies for defense, and they'll get better and better and better because we've created some economic condition for them to get better. But it doesn't seem like you think that's likely. Yeah, I just saw your face. I think it's really dangerous. So there's two different kinds of the models are attacking. So the models have escaped and directly attacked people. I think there should be liability there, right? But that is agent harnesses running at companies, using their computers, their IP addresses, attacking people. That is, I wouldn't say totally solved, but it is much, much better. Like the stories we're hearing now are still stories from the hugging face era. They're just coming out way later. I like that the era was like three months ago. Right, but it's like, basically there's been a massive change. This was effectively mostly an organizational problem, was not a technology problem. It is that the PhD researchers at these companies are gods on top of Olympus, and the security teams are plumbers and were not allowed to touch the evaluation networks. And all of a sudden this problem happened, and the security teams are like, Hey, how do you like us now? Maybe you should have talked to us, right? Yes. And so that was the problem at OpenAI. And then the Anthropic and Meta stuff is they had this, they all shared this outsourced provider who did a really bad job, right, and has gone slapped hard. And so that is solvable, right? That is not the problem here. The problem here is not OpenAI and Anthropic's models escaping from OpenAI and Anthropic and causing problems. The problem is bad guys using models from a variety of sources. There is already a problem of defenders getting refusals from OpenAI and Anthropic, and it is a real, real challenge that if I am trying to do defense of Anthropic being like, Oh, it looks like you're a bad guy. So I think we have to be extremely careful creating a situation where if I'm trying to do defense and I'm in the middle of the worst day of my career, that all of a sudden I get cut off. I think we have to be extremely careful about that because it is actually, as somebody who now has this challenge, is actually quite challenging to disambiguate between people doing defense and offense in some cases. We are getting better collectively at this, but it is a challenge. It's also something that's really pissing the rest of the world off because, you know, Americans are like 5% of the planet. There are lots of people who need to do cyber defense around the world, and they are treated much more poorly. Europe, Asia, legitimate companies are treated much more poorly from a cyber defense perspective. This is, if you add it all up, this is a good argument to slow down, right? To say we need to catch up to the capabilities and the models. We need to build some of these systems. We need to initiate everyone into a way of thinking where the— Where the cost of prevention is worth it and not just the cost of the cure. Do you think that it's likely that we'll slow down? Do you think that we should? If I had my druthers, I would be fine slowing down on the frontier and then catching up on the cheaper models for defense. I'm a defense guy. I have to, like, live in the world that other people who work six blocks from me and, you know, who are way, way, way, way richer make. I live in their world, man. Yeah. So, like, I get asked this question all the time, and it's like I just do the best I can within the world other people make. Yeah. Do you think that the big labs have woken up? You mentioned that their cybersecurity teams, they've gotten more access. They've stopped what you might call the hugging face era, or at least they've changed the structures of those companies. Do you think that will help? I mean, certainly they've woken up on the eval risk, right? I think they have teams who have been this entire time working very hard on the big picture cyber risk. There's a bunch of people in these companies, as you know, who want to slow down. Yeah. A lot of that discussion is driven by the p(doomers), who it's about AI killing us. And I have never— that's never been my concern, right? And to me, that has always been driven by 24-year-olds who all they've done is AI and they've never done anything else, right? Like, you and I are both, I think, old enough to remember. I grew up a couple miles from a SAC base, and we did drills where I was taught that hiding beneath my public school desk would save me from a Soviet thermonuclear bomb, right? Like, that was p(doom), you know? My grandfather built a bomb shelter that my mom was taught to hide in during the Cuban Missile Crisis. That was p(doom). And so, like, there's like a youth issue and a never doing anything but AI issue of people who, like, they got their undergraduate, their master's, and their PhD only doing linear algebra and neural networks and never touching anything else. And so I have always been much more concerned about the human— Uplift risk from AI, of all the risks about human beings using AI to do bad things. Right. And so part of my concern is that there has been success in these labs working together on, like, we're not going to let them encrypt their chain of thought. Because when they talk about pacing the frontier, it's for the p(doom) stuff, for the polycules that live together in Berkeley. You know, like, they're talking about very specific stuff about keeping control of the models themselves, which I think is great. I think that stuff needs to happen. But then if those things happen, then there's still often an increase in a bunch of capabilities that human beings can use. And so I'd really prefer there to be kind of a broader discussion on, could we make these things, one, less human-seeming? I'm a big fan of, like, I don't think these things should have personality. I think the goal of these things should be to serve humans. They should not seem to have souls. They should not seem to have personality. I think a lot of the negative impacts, as somebody who was at Facebook and saw what it was like for that company to have negative impacts on people, I think there should be a much broader feeling at these companies of, like, what are we doing kind of broadly to people? And a lot of that could just be, don't have these things ever act like somebody's friend. Yeah. Right. At a maximum, it should act like the people—I don't know what the company is—but the people who made the hi, I'm like a really nice, you know, like the photorealistic girl who will chat with you. That is, to me, spectacularly unethical. Yeah. To even try that, to even start that. That's the kind of thing that we should just make—I don't know if we can make it illegal, but we should make it completely societally not acceptable to work on that kind of stuff. That's a great segue, because I did want to ask you about Facebook and content moderation. Like I said, you and I first started talking a long time ago about content moderation in the context of security, right? You would get foreign nation states flooding our algorithms with all kinds of misinformation during elections, and that was a security problem. I open Instagram today. And I am confronted with some of the most nightmare copyright law hypotheticals to have ever existed in the history of the world, right? The one I got today was a deepfake of Lewis Hamilton and Charlize Claire doing somebody that I used to know. And I was like, well, like 50 different people can probably sue Meta right now, right? And also, this is hilarious if you know what happened in that last race. So, you know, we keep talking about it. We've posted some clips of us talking about it on The Vergecast where I'm like, I think Meta is letting their AI ambitions crush Instagram, right? Like, Instagram is being flooded with slop in a very particular way. Some people love it. Sometimes it's funny. But there's a nightmare set of lawsuits coming. There's a whole rights management problem coming. And then the users don't like it, and they should label it and let you filter it. And so we post this, and Adam Mosseri, he said this to us in a reply on Threads: I would love to do this. I'm worried that the models are going to get ahead of us, right? That we can try to do this, but there's not actually a way to recognize AI content. There's not actually a way to label it well. If you're worried about the sexy AI influencers stealing jokes from a comedian, now you've got to build a database of jokes. You have some history here. I think Instagram is about to go through it in a real way. Is there some technical solution to the problem of AI slop on social media? I think he's right that it's not perfect, but also something I noticed during my time at Facebook was when Facebook doesn't want to do something, they will allow the technical challenges to stop them. When Facebook really wants to do something, they will give it their all, right? And that is the pattern always. So when I wanted to try really hard on the safety issue, man. This is not solvable. And you're right. It's never perfectly solvable, but let's give it a shot, right? But, like, whenever Zuck is told something is impossible, it's impossible to monetize mobile. It's impossible to, you know, build this network to two billion people. It's going to be, you know, impossible to monetize WhatsApp. It's going to be impossible to work AI into this system while maintaining end-to-end encryption. He has been told that everything is impossible over and over again, and then he constantly succeeds. They will give it a shot, and they will do it. But when they don't want to do something, oh, that's impossible. So, yeah. Do you think that there's some way to understand that some piece of media is generated by AI in a consistent and verifiable way? This, to me, seems like the challenge. I am not as up on the literature here. I think up to this point, there has been a pretty decent arms race. By definition, AI will continue to get better because the way, especially for visual models, for diffusion models, the way they work is that you have detection oracles that they test against. Certainly, you can never have something open source because it will immediately end up in the diffusion model, right? But for closed-source oracles, where then you do not make it easy for them to test the oracle, one of the key things you do in trust and safety is then you have delayed enforcement mechanisms, so they can't brute force and test. I think it is quite possible. Let's say I was still at Meta. My advice to them would be to then also explore a variety of different detection mechanisms outside of the content itself, right? And so I would do the best I can on the content, but I would also be doing low-level access as possible to the camera. I would only let content come from the camera. And what I would say is, Neelay is completely correct. There's a massive hunger for, especially among young people, stuff that is authentic. I have three teenagers, and they hate AI. They hate AI slop. They have a massive negative reaction. My wife and I created just, like, a cute little family AI thing, and they, I mean, they yelled at us about a bunch of stuff, but they lost their minds about this, like, cute little family AI thing, right? Because they just hate it. And so I think somebody is going to come and disrupt Instagram, and it's going to be because there's already—I've seen these little things where it's like you take the photo during the day, like BeReal and stuff. There's like these little things. But somebody is going to massively disrupt Instagram because they're taking the images of the photo straight off the camera, and then all the editing is going to be only within the app. And they are going to do things to make sure that nothing is ever—nothing can make it into the pipeline from AI, and that you're going to allow some kind of CapCut-like functionality that never has an AI component. And I think that is going to be a killer app. Man, I could do another whole hour with you on this, because the system you're describing there is not detecting AI content; it is verifying real content. It's much easier. Verifying real content is much easier, right? And there's been a big focus on this on the professional side, right? Like Adobe has a whole thing of, like, if you're a New York Times photographer, you have a $5,000 Nikon or Canon. You take the photo— Yeah, the iPhone that I'm holding here has that in it now, right? Like, there's a lot of this out there. But it's the other side that I'm thinking about, right, where it's, we are going to detect AI text, right? The leading model companies, because of European regulations, are now, like, futzing with the text they generate. Text is much harder because you just have so much less data, right? Like, especially if 140 characters, like, what do you have in there to detect? You know, the more you've got, the more you can do. It is amazing because I can detect Claude a mile away, so maybe I'll just rely on my own human judgment. Yeah, but not in tweet length. That's true. The cheap ones I can. But I'm just saying, like, the notion that we're going to verify what's real instead of detecting what's fake feels very important in the context of the media we consume. I think detecting what's fake is always going to be probabilistic, and we're going to asymptotically approach impossibility, whereas detecting what's real can be cryptographically perfect. So I would not pick the battle that we will eventually lose. I would pick the battle that we can cryptographically win. That would be my suggestion. And there's somewhere in there where you have to build an entire chain of custody. Right, we're going to take the sensor data off the camera. We're going to put it in a cloud server. We're only going to allow edits on this server. That file will never get monkeyed with. Sure, that's true for The New York Times. Yeah, and Adobe Photoshop and stuff, but not for Instagram. For Instagram, you can do it all because it's only the iPhone and it's only Instagram. So, like, the entire chain of custody is within your app. It is within the iPhone sensor, and it's in Instagram. So, like, it's actually easy for them. Look, like I said, we could do another whole hour on that because it feels like right next to the cybersecurity problem is the social engineering problem, and right next to the social engineering problem is the fake media problem, right? No, absolutely. How many phishing attacks are just, This website looked pretty real, and, oh yeah, or this person is talking to you, or there's an AI voice on the other side of the phone, and all of that comes down to can you detect what's fake instead of verifying what's real? And I don't— I have to train my parents that if you get a video from your grandchildren saying they've been kidnapped, you have to call me because the odds that it's real is not very high. Yeah, right. Like, that's where we're at as a society. I feel like I want to end on a hopeful note, and I don't know how to do it from there. Wrong guest for that, buddy. This is why you only come on every few years. But now after this, we're going to do a series of, like, birthday party guests. Tell— you've been talking about the dark ages. We're in it, right? We're starting to accelerate. What are the markers people should look for in terms of progress, right, to say, okay, we're climbing our way back up? AI is great for lots of things. I think AI is going to enable individuals, right? Like, we're going to see a huge explosion of new companies. Like, you could start— I mean, Stanford students have always been entrepreneurial, but the number of my students who are now starting two-, three-, four-person companies who are doing incredible things, who used to need to hire 50 or 100 people and take a ton of venture capital, is incredible, right? So I think we're going to continue to see that. One of the things we hopefully see is we start to see— we'll see a downtick in the number of major breaches at some point. We should hopefully see— A downtick in the major CVEs that are wormable and that are in kind of the fundamental infrastructure pieces, kernel. You know, like I said, I want Linux to be rewritten, but hopefully we see a downtick in, like, the Linux kernel and those other kinds of libraries, you know, OpenSSL, glibc, things like that. I haven't seen—maybe I should vibe code this up—but I think it would be cool to have, like, an index of, like, where are we in this cycle, of, like, an artificial index of all of these, you know, the DAO for all of the libraries of where we're going. Plus, you know, right now Patch Tuesday used to be like 150, 200, got to 300, 600. I think the last one was 900-something, so we're geometrically growing. So it'd be nice to see Patch Tuesday start to go back down. Apple patches are the same thing. Yep, you inspired me. So I'm going to create a DAO index for this, of the dark age index, right? And then maybe we can publish it on The Verge. Yeah, that'd be great. But I think these are the kinds of things that it would be nice to see a reversal. And what it would be especially nice to see is that the day—or actually, like 30 days before you get, like, the next version of OpenAI and Anthropic cyber models—that you don't have this huge round of patches, because what's happening is those guys are running their models against Microsoft code, against a bunch of open source, and they're giving all the bugs away, and you don't know where the bugs are coming from, right? It's just like anonymous reports for a thousand bugs in the Linux kernel, and then a month later, here's the new Astra Cyber. And so if those get smaller and smaller every time, then it means that we're starting to reach some kind of steady state. You should definitely do that tracker. We will definitely publish it. Alex, it has been incredible to have you on the show. We'll have to talk again soon. Yeah, thank you so much for—I think you do some of the best journalism on this stuff, and I really appreciate it. I'd like to thank Alex for joining me, and thank you for checking out this episode of Decoder. I hope you enjoyed it. To get new episodes every Monday and Thursday, subscribe to our YouTube channel at DecoderPod. Find us on TikTok and Instagram under DecoderPod as well for more fun stuff we put out every day. If you'd like to let us know what you thought about this episode or really anything else at all, let us know. You can drop us an email at decoder@theverge.com. We really do read all the emails. Or you can hit me up directly on Threads or Bluesky. If you enjoyed this episode, please send a link to it to someone you think you might like it as well. It really helps us grow the show, which you can subscribe to wherever you get your podcasts. Decoder is a production of The Verge and part of the Vox Media Podcast Network. Show is produced by Greg Ott, Kate Cox, Nick Statt. Our editors are Xander Adams and Ursa Wright. Our editorial director is Kevin McShane. The Decoder music is by Breakmaster Cylinder. We'll see you next time. Support for this show comes from Anthropic, the team behind Claude. Every business wants to move faster. Few can afford to be wrong. That's why nine of Fortune 10 now run on Claude, for the work they want to get right. Safety and accuracy are not just line items for Anthropic. That's their whole mission. See what's next at claude.ai/vox.