AI Labs Warn Race Makes Breach ‘Inevitable’

The last few days and weeks have been some of the most fascinating, adventurous, and disquieting in all of recent history, with an emerging consensus that a major AI breach is due, the newfound likely unintended ability for you to use AI models to reverse engineer almost any piece of software out there, and of course, yet more research breakthroughs. Because there's so much, I'm going to split this video into three parts. Part one about the forthcoming breach. Part two on the reasons why labs say they are still racing forward faster than ever. And part three on why the next 18 months are still nevertheless definitionally hard to predict. Now, on the topic of the current race dynamic seeming to inevitably lead to a major AI breach, I have noticed in the weeks since I made that point at the end of one of my videos that there has been increasing media coverage as to the almost predictability of that eventuality. Financial loss or a significant loss of life. I think it will happen reasonably soon. You know, shorthand the Chernobyl of AI, but I hope it won't be that bad. And they don't anticipate that lawmakers will react appropriately until after there's some sort of catastrophic event. Risks that if that did happen, we're talking about a harm that's much larger, for example, than a single nuclear power station melting down. And the internal controls and safety and redundancies are just nowhere near what the world expects for a nuclear power facility. So when you have bad people, no safeguards, and then the latest models, that's where over the next year, primarily in the Siberia, we are going to have a lot of negative surprises. If you haven't seen my last video, there are, of course, lots of ingredients that go into people making predictions like these: how security teams at OpenAI are increasingly struggling to put perimeters around the models, why giving models access to tools and the internet helps keep labs at the frontier of intelligence, why jumps in capability can be surprising even to OpenAI researchers and security staff, and I'll cover that in more detail in part three, and why R3 main mechanisms for keeping models under control are each weakening for different reasons. For alignment, getting models to want what we want is increasingly precarious, especially given that models seem to know now when they're being tested. Monitoring models' chain of thought, or scratchpad, is getting increasingly difficult. So yeah, monitoring is getting tough, and locking down the perimeter, putting models in a sandbox. Well, even with older models that was getting tough, see hugging face, and with even the very latest models we saw just a few days ago that, say, Anthropic had to turn off internet access for all their internal evaluations because models were getting up to all sorts of shenanigans, like leaving false tips on various crime and homicide reporting forms. You can see why Joe here, who works on agent security at OpenAI, says that we should all prepare for the next wave of what can go wrong with AI. What I will admit not foreseeing, though, is just how revealing Sam Altman would be in an interview released in the last few days regarding how close some of these breach scenarios are. Notice the phrase in the middle of this clip that Altman drops in, where he says GPT-6.1 Astra is an interesting thing to study. So there are kind of two major ways that this could go wrong, and I think the Astra 6.1 is an interesting thing to study here. You can have a big safety accident, eventually one that leads to, like, a real loss of control, where AI, you know, escapes and it's out doing wrecking havoc on the internet and society can't really get control back. 6.1 Astra, the current model we had ready to go, I think would have tipped too much towards the accident risk. So what we're doing instead is pushing towards a model that we can make much more confident claims about the alignment. In case you missed it, he said that the model they would due to release next, 6.1 Astra, not their strongest internal model, which would be Bell, at least that's the code name, no, 6.1 Astra would have tipped too much toward the accident risk, the wreaking havoc on the internet where society can't really get control back. This, of course, echoes the recent warning from Dario Amodei, CEO of Anthropic, where he said that if the Hugging Face swarm had possessed greater capabilities, but a similar level of misalignment, the damage he thinks would have been catastrophic. On timings, given the accelerations, he's talking about six to 12 months, with swarms being capable of taking over the entire internet. Even more startling to me, I would say, talking about the next crop of models, was someone saying that they couldn't quite predict even the category of damage that would be done: cyber, bio. We are in the realm of powerful enough models that it is difficult to predict the specific things that will go wrong in what order. And I think politics and policy is going to be very reactive to the specific things that go wrong. So if the next big thing that goes wrong is like someone makes a novel pathogen, I think that will have a very different response than if the next thing that goes wrong is more cybersecurity and— I would guess the world in general will be caught off guard by the next thing that goes really wrong. Obviously, you don't need to be a lab leader to foresee where the current race dynamic ends up. As you've seen, myself and many other outside commentators have drawn the same conclusion, and of course researchers too. Jacob Coxon of the superviral tweet said to Time magazine that at the desk in which he worked at OpenAI and Anthropic, there was a kind of fatalism. There's this atmosphere of almost resignation. Researchers try to make their own work as safely as possible, even if they think there's a decent chance the whole thing spirals out of control. On the topic, though, of spiraling out of control, it seems like the labs are actually putting contingency measures in place should this happen. Axios reported the other day that top executives at Anthropic, OpenAI, and other AI companies are gaming out scenarios for a public and political revolt after a catastrophic AI event. This would be most likely a cyber attack that shuts down financial services, internet connectivity, or even power and water. Obviously, this would turn the public further against this technology, they say and its leaders. And the report adds that many top AI researchers and executives believe a major incident is inevitable. Anthropic declined to comment, but OpenAI said, These scenarios are not treated as inevitable, but are meant to help us prepare for a variety of circumstances. So those preparations were happening. Famed AI researcher Andrej Karpathy pointed out that the vast majority of people are not experiencing the vertigo of what's happening in AI. Either they're the six billion people who just don't use AI, the one to two billion people who use derpy chatbots, unquote. Then there's basically us lot. I'm sure most of the people watching, the 0.2% who are engaging with AI, building apps, following the news closely. But that might be 20 million people, with, of course, the 5,000 people at the frontier labs in a different category entirely. Those guys, or maybe us plus those guys, are experiencing the vertigo, Karpathy says, of the curve going vertical. Now, speaking of— Going vertical, that inspired me to work with GPT-Image 2.5 to come up with this. I'll be coming back to this image, but for me it hints at the first reason why labs find themselves in this position. Many researchers at the labs, indeed even safety researchers, didn't think we'd be reaching this point. And I didn't think that we were going to be able to create something fundamentally smarter than we were. And when I thought, no, we really are going to have something that thinks circles around us, that I thought, oh, we're in a totally— Inappropriate part of the possibility space here, in terms of how safe the industry is actually being relative to that reality. Okay, fine. But as I'll show in part three, we have definitely reached an extraordinary point. And I don't even just mean those near 400 mathematical breakthroughs that OpenAI dumped in a repo. One in particular was paradigm shifting. I'm also talking about the insanity of reversing binaries to unlock games, or just observing software behavior to recreate the entire Adobe suite. Others are vibe fabricating their own products, ones that are actually getting manufactured and delivered. Obviously, this is pretty infectious. I took a few photos of my mother's cottage and used Opus 5.5 to turn it into a design repeat with furniture and personnel, which is going to be 3D printed in China using PolyJet. Entire industries were built on the presumption that some of this stuff couldn't be done. So yeah, even current models with current capabilities are somewhat astounding. But for part two then, why are labs saying that they're still racing forward faster than ever? Well, you know earlier we talked about breach scenarios. No one could quite say when that cliff edge is reached. Clouds, of course, cover the top of the ramp because it might be a while before we reach that kind of capability. Or maybe our alignment techniques work longer than we even think they will. And so that's part of the reason models still are a little jagged, so why not push them forward a bit? Next reason, though, concerns the ramp being made of gold. I covered in the last video the exponentially increasing revenue of these companies, and one anthropic researcher put it quite bluntly. We often talk about this as sort of exponential economic returns to intelligence, where each marginal unit of intelligence that you're capable of is worth exponentially more than the unit that came before it. The closer you get to the edge, the more you mint money. The next reason that the labs are putting their foot down is, of course, the labs behind them. All sorts of ego battles going on between xAI, Anthropic, OpenAI. Meta will soon be seeing some pretty interesting releases from some of the non-OpenAI or Anthropic crowd, that I can reassure you. But aside from pride and ego, one of the most cited reasons that Western labs give for continuing the race is, of course, to stay ahead of China. I could give you a quote from every one of the lab leaders. Take Sam Altman saying, I want the U.S. to win in both open-source and proprietary models. But no one makes this point with more ferocity than Dario Amodei. He frequently writes and says things like this: I agree with Secretary Bessent that a Chinese lead in AI would pose grave danger for the U.S. and for the world. They would be in a position to militarily dominate democracies, for example with AI-driven drones. That would more argue for not staying behind China rather than necessarily being in the lead, but that's a separate debate. I often hear the U.S. government and lab leaders say in interviews unchallenged, Yeah, we have to maintain this lead over China. You've seen the quotes already, but there are a hundred more I could show. But lab leaders are increasingly clear that they don't want to make models that have unknown capabilities and are increasingly out of their control. The chief scientist of OpenAI said that we can't continue responsibly scaling at maximum speed for much longer. I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established. Sam Altman is increasingly focused on a rogue AI model itself being the threat. The potential downside here may not be the kind of thing normally goes wrong that we can correct with a legal system for, but that something goes super wrong, like in the extreme case back to what we talked about of, you know, we lose control of the future to AI, which is horrific. So my stance on regulation is pro-innovation. Let some things go normally wrong and trust that we have a normal process for doing this. But where this technology is extraordinarily powerful and fundamentally new, have guardrails to ensure that. All of society is kept reasonably safe, and we don't have, we don't kind of fall off that path in either direction in what could be a very difficult or unrecoverable way. For me, though, what no one has challenged AI lab leaders on is this: what happens when those two commitments come into tension? What happens if one of the lab's own evaluations showed that their next model would be superhuman in critical domains, military or bio, but would not be reliably under their control? Would they stop even if that meant another lab or China catching up? Another way of putting that is: under what criterion would you cease to believe that we need to maintain a capability lead over China? And for the first time, I saw Sam Altman lean towards pacing, as long as China would also verify compliance. What we really want is that at each stage of new capability, we have enough progress on alignment, interpretability, safety, understanding how society and the technology are co-evolving to be able to safely go to the next level. And when we don't have that, at this unilateral level even, then we do pause. And if the U.S. and China could agree on shared safety standards and requirements to proceed to verify compliance with the thresholds that we should demand to proceed, I think that would be an incredible step. I think they could do it. Of course, that doesn't resolve the question of what they would do if the two challenges came into tension and China refused to verify. But maybe their plan is to cross that bridge when they come to it. And what of the other labs? In my early tests of their new Muse Wonder, I wouldn't quite put it second to Opus 5.5. Remember that the AA index includes some quite outdated benchmarks, but it is approaching the ballpark of GPT-6 Astra. Leeds, as ever, are measured in months. Well, I won't show you the clip, but Alexander Wang, leading the effort at Meta in their superintelligence team, was admittedly honest, saying they don't really know how to align their models perfectly. Their current best guess is scalable oversight. Some of the techniques involved in that rely on a weaker AI model overseeing a stronger one. Satya Nadella, CEO of Microsoft, echoed that approach in part, as well as calling for an emergency break in which models can be shut down. He also said no single model should control both a system's behavior and the evidence required to determine whether that behavior is aligned with the original intent. No single model, of course, hinting that another model might be checking that. That deserves another video, but I would just caution those two with this. You might also want to make sure that the second model is fully distinct from the first model, because teacher models are, for example, able to, through fine-tuning on a seemingly innocent innocuous list of numbers, encode backdoors into other models, vulnerabilities where the model being fine-tuned will suddenly behave quite differently if a trigger scenario is at hand. Yes, that complex of a behavior, if a female name is in the prompt, then answer in French, conveyed through fine-tuning on extending a list of numbers. Before I move on to part three, there is one other reason given for continuing to race. The Financial Times asked famed researcher David Silver, formerly of Google DeepMind, now with his own Neo Lab, why would anyone want to create a superior intelligence at all? He says, Well, ask where are we going without AI? Humans have had world wars with allowed mass poverty to happen. This happened. We've allowed curable diseases to kill millions of people. He's essentially saying, yes, there's risk involved with creating superintelligence, but don't forget there's always background risk with being the human species. For me, that's a great argument for building it at some point under the right conditions, not necessarily under these conditions. Like going around a Formula One track, getting to the end of the lap doesn't mean taking every corner at top speed. Despite all of this, it's time for part three because the next 18 months are still definitionally hard to predict, in my eyes. Part of the reason is that it's very hard now to express a qualitative difference in what a top human can do and what a top model can do. I'm not talking what tools we give models or what experiments they can and can't run legally. I'm talking category of reasoning task. Even on the topic of taste, which I'll get to, I'm not convinced. But take a domain where our top IQ humans spend months, years, decades sometimes, thinking and thinking and reasoning and coming up with challenges and conjectures for themselves: mathematics. While not every result has been verified, when OpenAI dropped hundreds of solutions, many called it the biggest day in mathematics history. One in particular, though, involving the Zeta shortcut, has been described as trumping even the Navier-Stokes solution. That was a solution to a millennium prize problem. A tiny bit of background, because of course this is way beyond my mathematical ability, and any human for that matter. But essentially, the prime numbers that we all know and love, 2, 3, 5, 7, 11, 13, 17, etc., have no obvious rhythm in the gaps between those primes. Those gaps open and close for no visible reason. Bernhard Riemann found a hidden order in the chaos in the 1850s. He said that despite all this chaos in the gaps, the primes stay as close as possible to a predictable pattern. That birthed the Riemann hypothesis. Later, to the best of my understanding, mathematicians set a humbler goal: proving the primes at least stay within a fixed safety margin of the pattern. Not proving the pattern, proving this fixed safety margin. This became known as the quasi-Riemann hypothesis. Riemann hypothesis. A breakthrough on even that, though, was described until very recently as out of reach. A breakthrough there would be complete science fiction. Another mathematician said, It's completely out of reach with existing mathematics. Another Oxford mathematician said, I was not expecting the magnitude of some of these results. But this one in particular, on the zeta shortcut, is absolutely shocking. If the 199-page proof of the quasi-Riemann hypothesis holds, it would be the biggest step towards Riemann's bet in more than a century. I'm going to be honest, though, what caught my eye wasn't just those descriptions from mathematicians. Better than Navier-Stokes, impossible with existing mathematics. For me, it was the hours I spent trying to define a qualitative barrier between what humans have been able to achieve in any domain, take maths and models now. One would have been finding a regularity, not just recombining existing results or ploughing forward with gargantuan search capacity. No, but unlocking a deeper meta-function, a hidden pattern in the universe, a regularity in the chaos, something which, when found, would then help others in other domains unlock further breakthroughs. Here then we turn to a Claude summary. Within five days, Opus 5.5 writes, papers using the AI's idea, the zeta shortcut, claimed three more solutions. It helped in a paper on number theory, one on computer science, and in other papers, which then still leaves the question, well, what then is next? If even the highest IQ individuals in the world, like Terence Tao, say this wasn't a demonstration of scholarship, but a demonstration of power, well, whether or not you agree with that, and of course I have sympathy with the sheer shock that mathematicians will be experiencing, whether or not you agree, the question is still, well, what's left? Obviously, they haven't yet solved the other Millennium Prize problems, including the Riemann hypothesis itself in full. But maybe some of those are like silent locks, where if you're trying to pick them, you don't get tractable feedback as you're going. You have to unlock many interdependent, seemingly unrelated theories along the way. Just to have the tools to solve the overall problem. Well, to that, I would say two things. First, humans haven't done that. They rely on combining disparate theories and results that other humans have created. But second, even if those breakthroughs rely on as-yet undiscovered stepping stones, what's to stop models being unleashed on creating as many seemingly unrelated stepping stones as possible? Create hundreds of verified proofs and then periodically try to recombine them to find something greater. You could almost call that a kind of taste, where intuiting what kind of stepping stones might prove the most useful in the future is a form of taste over time, which is why I came up with this: taste as the last frontier. How efficiently we, or a model, finds the inputs that will matter is that the last frontier. As I've kind of already hinted at, for me, taste is just capability spread over time. It's not some supernatural ability; it emerges through observing other aspects of the formula. Take progress in any area, any domain. Ask yourself, how hard is it to check intermediate results? Is it slow and expensive, like in biology or drug design, or is it fast and cheap, like mathematics? Think of signal sparsity. To come up with the Zeta shortcut, OpenAI's internal model had to deduce what useful inputs they were among tens of thousands or millions of mathematical theories and papers. Taste in AI research works a little differently, as I'll touch on in a second. Then there's goal messiness. Is the goal well-specified, well-defined, well-scoped at all? Making models better at professional real-world tasks is a lot messier a goal than find a counterexample to this conjecture. Then, of course, there's the input of compute. A model might have pretty inefficient, bad taste, but given billions of GPU hours, it finds the result. I might be stretching things a bit here, but this, for me, a little bit gives me a framework to compare this breakthrough in the quasi-Riemann hypothesis and the difficulty of automating AI research. First, again, the quasi-Riemann. What kind of taste did that show? Exact goal, low checking cost, an unknown number of hours of compute, but pretty distant signals amongst the noise of mathematics. Maths. It had to pick out techniques from disparate areas of mathematics and combine them in novel ways. For me, this shows a decent amount of taste. The same model as we know, given three hours for all those other 372 families of results, achieved a bunch of other breakthroughs. We're not, of course, talking perfect taste. Otherwise, it would have solved all of maths. Solving the full Riemann hypothesis might have involved extreme signal sparsity, inventing theories on the hoof many, many times in a row just to even have the stepping stones to get to the result. But let's now compare that to Tasteval. It's a proxy measure of an AI model's ability to automate AI research progress. It was covered in Jack Clark's recent post on Import AI. He is, of course, one of the co-founders of Anthropic and one of their leads on RSI, recursive self-improvement. He said of Tasteval, benchmarks like this give us some signs that they're getting increasingly good at AI research. I now think it's more likely we'll see full recursive self-improvement become possible in 2027, whereas before I thought it was more likely in 2028. The research itself, out last week, defined research taste as the ability to pick interesting problems to solve, design experiments, and interpret experimental results. How well can a model design experiments, iterate on them? Well, it turns out with Opus 5.5, better than their expert baseline, the human expert baseline, roughly two times better. But other models like Astra were also better than the baseline. We should be careful in interpreting this, though, as saying that models can already automate AI research. The designers, PZero, point out that these tasks are fast and cheap to verify. The human baseline doesn't include top researchers, albeit they were those who had worked recently at OpenAI, Google DeepMind, Nvidia, etc. But ultimately, we're talking about 40 GPU hours and one H100, with fast and cheap verification, usually under 10 minutes. These weren't the gigantic, complex training runs that frontier labs do. Each task had a clear goal. Interestingly, if you hit the goal, essentially made the task more messy, performance didn't drop off a cliff, but it did halve. The thing is, though, fully automating AI research is different. Goals are messier and the signals are much more sparse. John Schulman, instrumental in creating ChatGPT and now working at a rival lab, said that with AI research, you're usually not directly optimizing for the eventual objective you care about. Because getting feedback is so much more expensive and so infrequent, labs are currently much more reliant on human intuition. I would say a lot of research is not exactly like that, though, where it's like hill climbing on a well-defined goal. It's sort of more like, here's an intuition we have about some way models should be better, and then we also have some idea for an algorithm that seems to go a little bit in this direction. So let's come up with a task that is sort of designed to show signs of life on this approach and, like, see if we get those signs of life. And then if we do, we can make successively more realistic versions of the task. You're not directly optimizing for the eventual objective you care about or the practical, like, production objective. It's sort of you're relaxing your objective a little bit. You're saying, yeah, let's relax on the realism axis a little bit and find some methods that actually work, and then try to get back to realism later after the method matures a little bit. So as Anthropic state in their system card, we're not there yet. Opus 5.5 trails by around 30 points from Anthropic's own researchers. This is on Anthropic's internal benchmark, CoBench, for automating AI research. So my argument is we're not there yet because the checking cost is so high and the goal is messier and because there's so many permutations you could try. But with effective compute increasing exponentially, as I've covered in other videos, bigger models searching more options, AI research taste, you could say, is getting there nevertheless. And the mathematical breakthroughs show it's not a question of IQ; it's more a question of inevitability. If these trends are inevitable, why did the lead researcher on reasoning at OpenAI, Noam Brown, say that he was shocked at how good models had gotten in, for example, mathematics, thinking this would come two years later? For me, this comes down to us not knowing what we don't know. We didn't know how many higher-order structures there were in, for example, writing. That's why GPT-4 shocked many in its ability to write essays. There must have been more higher-order structure in mathematics than we realized, hence this many breakthroughs this early. There are some fields, though, that are more fractal and chaotic, like, for example, predicting the weather. Now, yes, it's true AI models are leading to breakthroughs there, not quite the same kind of LLMs, but nevertheless, Google DeepMind did see this hurricane coming, Iasias. With AI pitted against traditional models, there was a key divergence as to the path that the hurricane would take, and Google DeepMind's model got it right. I think it was an 11-day-ahead forecast here. But weather is fractal, chaotic. We could probably make models 100 times better, and they could predict 13 days, 15 days ahead, not a million. Are there higher-order structures in biology, cybersecurity, or AI research that we don't know about? Probably, but we don't know. This is a big part of why I see each new model as essentially having somewhat unknown capabilities to spot patterns. We just don't know what patterns are out there. As Sam Altman said recently, I don't know how sci-fi it's going to get. No one does. Many hope for the best of both worlds, where we get models that are more capable and more secure. But given that many of our alignment tests rest on the models not realizing that they're being tested, it may be more like a fork in the road, where almost by definition, capability increases mean less monitorability, more eval awareness, more parameters, less ability to figure out what's going on in the neural networks, less mechanistic interpretability. Obviously, automating AI research speeds up all of these trends. Under that framework, then, the problem isn't that we can't make models more predictable. We can make them act more like software. The problem, as this golden ramp illustrates, is more that there are exponentially more rewards for making models less predictable. That's how you win as a lab: somewhat inscrutable models writing 80% of the lines of code, soon 90%, tweaking their own. Ever more opaque training recipes, grading their own outputs, organizing their own swarms at ever more intractable scales. People have mentioned instead suing labs when things go wrong. The Economist, though, says liability law will not tame AI. The reason? Well, it would take years to decide such cases, let alone then use those cases to establish reliable standards. It would, of course, all happen long after any incident. In Politico, a University of Pittsburgh professor of law, David Thor, said, We have to answer: do we want to criminalize someone for the kinds of behavior alleged and admitted to in the hugging face incident? There's been a lot of split on this, and Congress has not spoken clearly and unambiguously to the issue. Let me know what you think if lab leaders or researchers should be held accountable. For me, I wouldn't mind an internally driven quadruple lock within the companies across alignment, monitoring, security, and capability. On alignment, a lab could set a cap across a million permutations of adversarial prompts, a max of X deeply misaligned actions. That's the alignment lock. On monitoring, you could say that on a minimum of X percent of the tries, those misaligned actions, they need to detect in the activations or the chain of thought if that try has occurred. Say a minimum of 99 percent of the time, you detect that action in the activations of the neural network or the chain of thought. If that standard blocks too alien an architecture, then so be it. In other words, you're not blocking the AI-written new architecture just for its own sake. You're blocking it because it drops monitoring below that standard. On security, models could prove that they have X separate layers of security, each one red-teamed Y times with rewards for any breaches you could demonstrate, rewards for the red teamers. On all of these, you might say, what about Goodhart's law, that these will cease to be good measures because they're targets? Well, labs could commit X percent of their compensation or compute budget for rewarding misalignment science, those researchers who can expose failures of alignment or security or monitoring. Each revelation might show that the cap The caps haven't been met. So if that science outpaces the improvements on those targets, that would mean de facto a halt to frontier training. Back to that China point, you need these scientific reasons, not just a vibey approach, because there will likely come a moment where much or all of the lead over other labs or China might have to be sacrificed to add these necessary safeguards. You would need these checks and caps to then justify to shareholders and the military why the safeguards are necessary. To complete the fourth lock, I would add in capability tests. Models steered to try certain things. With those steerings, can the frontier labs still say that models can't achieve target Y across X tries, where Y is a capability we don't want them to have? Obviously, there are also loads of things we should be doing outside of the labs, almost a social hardening layer. Some great suggestions were found in this article, for example, by two top bio researchers. Things like clean air infrastructure, synthesis screening, metagenomic detection systems, stuff that basically everyone agrees on but politicians aren't doing. I'm not going to call that a fifth lock because that's outside of the labs' control. And obviously, I'd love to hear what you guys think about these suggestions and if there are better ones. What do you think on the wider picture? Are we in final stage capitalism, where AI offers the prospect of a complete separation between those with capital and those who rely on labor? Is it a new golden age of reverse-engineered software and games, where yes, it might erode the power of some professional organizations, but does also spread power to almost anyone? Or are we at crunch time and have to realize that a breach may well be coming? I'm just glad we have lovely people like those in my audience. Thank you so much for watching to the end, and have a wonderful day.