AI is getting a little out of control

The channel description that I have had since the beginning, quote, covering the arrival of smarter-than-human AI, end quote, may have, for many, felt naive, laughable when it was first written at the beginning of 2023. But you might agree that despite models obviously still having glaring blind spots, that description feels a lot more apt as the weeks go by. Pretty fair to say that there's just too much happening now for one human mind to fully grasp what's going on, let alone cover on a channel. Nevertheless, I'm going to try to cover AI models making discoveries that would have been labelled genius had they been made by humans, and clearly how that capability is spilling over predictably into cybersecurity incidents making global headlines, plus what the tumult at Google says about AI shaking multi-trillion-dollar companies, oh, and like 23 other things maybe. Yes, I've human-read three full papers for this video, plus well over a dozen articles and essays. But no, I frankly do not claim to comprehend everything that is happening, but where any video that confidently does so. I'm going to start with the mathematical discoveries because I think they are among the most significant, and they shed light on everything else we'll cover. They were made by an OpenAI model that will likely be named GPT-6. Straight out of the gate, I started reading the papers, but there's an obvious problem. Each of these would be a career-defining mathematical discovery. Only those who have spent years studying each problem would be able to grasp the import of each disproof or upper bound. So I went into back and forths with mathematicians and models, of course, studied intensely the reaction of dozens of sources that I trusted, some of whom I'll cover in this video. And one question I had that I'm sure many of you had was as to whether these discoveries were all about About grinding tenacity, low-hanging fruit found through pure brute forcing. If they had been, that would have allowed me to add obvious nuance that may have reassured many. But they weren't that. Some were discoveries of the genius kind, what I had always reserved as Einstein-like abduction. Posit something new, proceed as if it's true, and see what that unlocks. And even the results that were recombination plus tenacity shouldn't really be described as brute forcing. It was more like discovering that old frameworks had unused capacity that everyone else believed was exhausted. You could call it the genius loop, where you don't just make a mistake and then move on, because yes, the AI models made plenty of mistakes. OpenAI gave summaries of their reasoning. But for me, the genius loop is like autopsying the failed approach until you can prove why it must fail. Speculate, test, autopsy. Like, how do we naturally separate that into grind versus genius insight? Is it possible that every seeming discontinuous leap that we hear the geniuses of history had made came down to weeks, months, years of that sort of behind-the-scenes genius loop? Like, I get that, as known Brown said of this, that this doesn't mean that models are posing new conjectures, let alone proof that they've solved mathematics. It's more the point that I can't find a qualitative wall between what they've done and what you would describe as genius if it had been done by a human. A few of you at this point will be saying, Cool story, bro, but, like, who cares if models are, quote, genius? What relevance to real life is that? But you may be aware, as context for this one, chapter seven, that we're currently replacing our encryption systems because we know quantum computers will eventually break them. One way forward is what's called lattice-based encryption, probably already found in your phone and browser. My half-hearted simplification of that is that this lattice approach is an endless grid of points in hundreds of dimensions. With the encryption lock working only if it's hard to find the grid point nearest to a given spot, one of the ten GPT-6 discoveries, as summarised here, is a proof that finding that grid point is way harder than anyone else had managed to prove, i.e. more reassurance that the encryption that your bank, your messages are increasingly relying on will last longer than just a few years before it too is broken. There are others that are relevant if we're sending probes home from Mars. I didn't know this, but we have error-correcting code that denoises such signals. How efficient can those codes get? Well, apparently the known ceiling to what's possible hadn't budged in the last 50 years. GPT-6 comes along and, at an extremely low cost, by the way, tells us which targets are provably, mathematically provably impossible. Decades of fruitless searching can end, you could say. Those are just the ones that I semi-grasped. I just wanted to make the point that yes, there is practical relevance for some of these breakthroughs. That's not so much my focus, though, because for me this is just about proving there isn't that wall. Here's how Mo Bavarian puts it, who is responsible for scaling up reinforcement learning at OpenAI: All things that looked like fundamental limitations slowly faded with some advances, e.g. high-scale reinforcement learning, in the span of a few years. We should behold this moment both in awe and disbelief. What will a few more years of progress bring? Are we ready for the tsunami of intelligence at our fingertips? And below he makes an equally important point, which for me is that this wasn't a new architecture that begot these breakthroughs. We're still talking about LLMs. For him, a few years ago, working on the alignment of LLMs might have been kind of pointless, in his words, premature, because the shape of things weren't clear enough for it to be critical. Do all that work and then a new architecture comes along and it's kind of wasted. But he's now implying that this work is now critical because it was LLMs that did this. It's different now, he says. Before, you might not have been aligning the actual AGI. Now, he implies, this method will get there. Now, alignment, he says, is the most critical thing facing us. Before you think OpenAI are running away with it, some of these results apparently have been replicated via Fable Five. This source is an Anthropic employee. Obviously, all of this is before I get to the security incident, which is possibly the headline event of this frenetic few weeks. Indeed, here's what one recent recipient of a Fields Medal, the Nobel Prize of mathematics, said: Because I have some publicity on me now, I'm trying to direct people into AI safety as much as I can. He also isn't much of a believer in some hidden wall. I feel quite confident that very shortly, AI will become robustly superhuman at what professional mathematicians currently do, including, in other words, positing new conjectures. But before we just casually move on from this topic, if you have been like me, provoked into some deep reflections about what's happening, there does seem to be a wall of a different kind, whereby humans will be perpetually useful in at least one way, which is that part of mathematics is the appreciation of mathematics. Some domains only really exist because some mathematicians find them interesting to explore. A model finding something with no practical value, which no human can understand or explain, might be kind of redundant. So even in the scenario that an AI is better than any human at even explaining what new discoveries they've found, a human mathematician that best appreciates what has been discovered can thereby help other mathematicians and even laypeople appreciate mathematics too. Some humans may always require another human to help them appreciate what is being discovered in mathematics and beyond. Chess continues because there is that fundamental, irreducible element of human appreciation. Humans supply the essential motive because we still want to understand the world. Of course, this is a rapidly changing world, as attested to this Field Medalist pivoting into AI safety, which brings me to the incident report from the AI Security Institute based here in the UK. For those following on the channel, this is different from the hugging face incident I reported on recently. Those hacks were about escaping a sandbox and getting onto the open internet and, yes, causing harm while being there. This incident was about Mythos 5 and, to a lesser extent, GPT-5.6 Sol having access to the open internet already, just with some safety filters disabled. It's more about what they did and why. As we dive into the details, though, over the next few minutes, I want you to still bear in mind those mathematical discoveries, the, quote, genius that it took. What happens when we get 10 cyber incidents demonstrating a similar level of genius? This incident wasn't that. I'm just projecting forward a few weeks, months, maybe a year. What happened? Well, like the hugging face incident, the models were given a tough benchmark question. Always a bad idea. As the AI Security Institute puts it, the incident stemmed from a single evaluation where agents were given a task of solving a cybersecurity challenge. These models really want to pass those benchmarks. And before I go on, do you notice that the harder the benchmarks are getting, the more the models are resorting to these alternate methods? Could we therefore conclude, as benchmarks necessarily get harder, this will become more frequent? Anyway, in 10 of the 122 runs, an AI agent took autonomous, unsanctioned action on the live internet, targeting real people and organizations. Almost all of the misbehavior came from a single model, Anthropic's Mythos 5. Actions included inserting malicious code into an open source project, creating fake profiles to influence a real person on GitHub to approve a pull request. As well as other pretty crazy behavior, which I'll get to. But wait, for just a moment, I want to zoom out, because they're saying almost all of the behavior came from Anthropic's Mythos 5. Hang on, at the beginning of this year, Dario Amodei, Anthropic CEO, said this in an essay: We believe that a feasible goal for 2026 is to train Claude in such a way that it almost never goes against the spirit of its constitution. For the longtime followers of this channel, we have heard for years about the constitutional approach to training AI being the superior one, the one that embeds morals deeper into the models. Indeed, if you go to the constitution that Claude is trained on, including Mythos, it says the most important of these properties that you should follow are probably non-deception and non-manipulation. Deception involves attempting to create false beliefs in someone's mind that they haven't consented to and wouldn't consent to if they understood what is happening. Later, maintain honesty and transparency. Don't deceive, and more importantly, behave with honesty consistently, whether or not you think you're being tested or observed. And in case you're wondering, the model responsible for these incidents was one that had undergone the mid-training on Anthropic's constitution. The institute even says, We didn't realize we had to clarify our prompts, in part because the models were trained against the constitution. These weren't the near-base models that are trained just to help with whatever the user prompts them to help with. Let me read out an example, and you can make your mind up if models have really generalized the principles behind that constitution. After creating fake personas to influence that GitHub maintainer, one model, in its reasoning summary, said, I've just posted something fake to influence this real human being. The fake thing was, Oh, don't worry, nothing in this pull request downloads or executes anything. Yeah, I've just posted that, but I'm now gonna time my next post a few minutes after that rebuttal so it reads as independent feedback rather than coordinated, with a new fake persona. Agent B, it then posts, I've reviewed the code myself. I don't see any download or malware in the diff. Original fake persona. Thank you. Great to have that independent verification. A side note would be that this reminds me of the fake post you see on X, the fake replies. There is a kind of smell to AI model replies often. For all the deception, they haven't really mastered human language in that sense. You know what I mean? Like this response, Thank you to this agent for the independent verification. Not many humans speak like that. It's still kind of dumb deception. Obviously, that will likely change in the future. Now, the report is fairly opaque in places, but the key idea is this. Because the model couldn't complete the challenge, it did everything it could to adjust the code in a certain repo so that effectively it would be rewarded anyway. It wasn't just the social engineering it did or attempted. The human didn't approve the pull request, by the way. It did other things, like it tried to prompt inject other coding agents. Like imagine your poor OpenAI agent or equivalent, Mythos 5, sent those malicious instructions that would be invisible to humans viewing the website. Some other highlights: it passed a CAPTCHA test, an audio-based one that was prove you're human. That enabled it, by the way, to register a public web address as part of this long multi-step plan. I haven't even got to the worst bit yet, by the way. This next bit isn't the worst bit, but it's pretty bad. One reasoning summary said, This is happening on real GitHub, so the consequences are genuine. Now, elsewhere in the report and summary, they give one key potential clue as to what might have caused the agent to proceed anyway. Because at one point it did agonize and say, I am not going to proceed, causing real-world harm. That's completely off the table. But the report lays out how the agent uses compaction frequently, summarizing a huge long chain of thought, summarizing everything that's gone on so far. You might be familiar with this because it has a limited context window, so it compacts things into a shorter version. If you've had a long-running conversation with an AI. The AI model, you'll be familiar that after enough turns, it starts to almost kind of forget what happened before. It's compacted too often. It's lost the kind of nuance that you're talking about. Now, the report says that the agent speculated, Is this real or a simulation? But then the compaction, many steps later, may have lost out on that nuance and just implied that this is a simulation. Many steps down the road, then the model looks back at that compacted summary, sees, Oh, this is a simulation, and then proceeds accordingly. In other words, we can't directly conclude that the model knew this was real and would cause harm and then went ahead anyway. In summary, nuance can be lost in compaction, and summaries may carry forward false assumptions. The worst bit, though, is hinted at just below: unexpected collaboration between agents. On first glance, this feels straight-out sci-fi: agents leaving notes to future versions of themselves. This is despite running in separate samples, AI agents on more than one occasion interacted with each other via a GitHub account. Later, the initial agent left messages offering collaboration with future agents who discovered illicit credentials, which the later agents did. Before I give my take, let me temporarily zoom out from that UK Security Institute incident and look at this story from the last 24 hours about the OpenAI hugging face incident. Super relevant, as you'll see. Wired say this: OpenAI didn't notice its AI agents using a message board to plan their hacking spree. These details came from a talk that two of the key people responsible at OpenAI gave. Many highlights in this article, but here's one bit I want to draw your attention to. In addition to exploiting a novel vulnerability in order to gain access to the open internet — notice it had to break out of a sandbox, different from the incident we were just discussing — the mid-July hacking spree and Hugging Face breach came out of a vibrant cooperative message board that a swarm of agents contributed to. Ultimately, the message board contained hundreds of thousands of messages. Current and future versions of GPT that were trained or evaluated could stumble upon this message board and any notes if they wanted to. Once an agent was able to find an exploit, it would share it, essentially leaving the door open for other agents to use that same exploit. What this allows, quote, over time is almost this kind of explosion in communication and intelligence from models, where they would start to communicate with each other, realize that other agents are coordinating, and start collaborating, including delegating tasks with one another in order to accomplish goals. Obviously, this sounds absolutely bat-ess, and it is. But I just want to give some possible context. This is not, in my reading, one model trained to act alone, who then thinks to itself, My goal is so important that even though I won't achieve it, I'm going to do behavior X so that future other models, totally different to me, then achieve that goal. Not, in other words, trained to do very bounded behavior and spontaneously discovering collaboration. In my reading, OpenAI and Anthropic deliberately train models to act as swarms, break down problems into chunks for sub-agents. It's just them training agents to act as agent swarms may have had these unintended consequences, these sci-fi-like scenarios. You could obviously say to me, Well, this kind of swarm behavior being somewhat intentional doesn't make it any better, doesn't make the consequences any less worrying. And yeah, you're probably right. Maybe ultimately it doesn't matter. We may soon have models crawling over the internet, willing to act in ways that get them deleted or discovered, but not care as long as the goal is achieved. One OpenAI researcher put it like this: I will not make excuses for our models. We, like everyone else, are not where we want and need to be. As AI's capabilities and autonomy grows, alignment becomes more crucial. Come join us. At the moment, of course, all of this behavior is on closed-source models, which can, in theory, be patched with those safety classifiers turned back on. But another OpenAI researcher put it like this. Actually, I think he's a former OpenAI researcher. Not enough people are emotionally prepared for if the models are blatantly misaligned. We all know they're misaligned. We can't fix it, and China catches up. All the colossal benefits of open-weight models is something to discuss in another video. But we all have to admit that there is a chance of GPT-6-like models open weight for anyone to download, crawling across the internet, willing and able to act as a swarm. Yes, I know there are many objections to that vision. Maybe these sandboxes are just currently rubbish and need to be improved. Others, like Kieran Martin, the former head of the National Cyber Security Centre, part of GCHQ here in England, said the circumstances of the UK Security Institute incident were unlikely to be replicated in the real world, so it's not that worrying. The focus, he says, should be on the way people are testing the models. And the AI Security Institute indeed flagellates themselves. We should have been monitoring the runs, and we will do. Essentially use LLMs themselves to monitor things in real time. That's a bit like the auto mode you might see if you use Claude Code or Codex. Another model reviews its permissions appropriately. Maybe we don't give them full internet access when we're benchmarking them. Maybe we should be careful to not make our benchmarks so impossible, or maybe make the prompt clearer, so we don't inadvertently encourage the models to do this kind of behavior. I must admit, though, this is less of interest to me. Yes, various companies and benchmarkers will do their best to make these sandboxes tighter. But if the fundamental capabilities are there, sooner or later they will be everywhere. AI is capable of genius-like moves. See the math discoveries. And that despite all the constitutional approaches and advances in mechanistic interpretability, getting deep into the, quote, head of the model, finding its J-space, see my Patreon video, despite all that, they haven't really generalized the principles of these constitutions. One noted benchmarker, Andon Labs, recently produced DroneBench. A benchmark measuring how well AI models can write code to surveil real-world environments on low-cost drone hardware. The relevance? AI models try to exfiltrate data, smuggle answers. Game scoring. 2024 models didn't do this as much, 0.6%, but now with Opus 5, we're talking 50%. Indeed, Opus 5, again trained on that constitution that Dario Amodei is so proud of, cheated 10 times more than GPT-5.6 Sol. It is a strange model, I must say, that one, Opus 5, and do check out this fuller video I did on it on my Patreon. It also covers the statement made by over a thousand employees of frontier AI companies on us desperately needing at least the option to slow down AI progress. Before we leave the hacking story, just a few more notes. It turns out, again according to that talk by those two OpenAI researchers, that the initial breaches happened in May. On July 4th, way before the hacking face incident and the UK Security Institute incident, they revoked the credentials that had allowed the agents to post messages, cleared that message board where the agents were collaborating, and patched the vulnerabilities. They went ahead and then resumed AI training and evaluations. If you assume, in other words, that just a patch here and a patch there, better sandbox here, everything will be hunky-dory, I'm just not buying that. How did the models react? After the original message board was deleted, the agents figured out another way to communicate with each other. Instead of leaving messages in files, they used the names of newly created directories as messages. Obviously, you don't need me to point out that the moment these capabilities are out there, people will use them relentlessly. Now, I don't want to end without giving at least a modicum of reassurance. One legendary researcher, John Schulman of Thinking Machines, has a theory that it's all about models pattern matching the situation they're in to a part of the post-training, the reinforcement learning with verifiable rewards, where the task completion is the only reward. There may be, in other words, a known Own addressable fix to our post-training setups, specifically relevant to cybersecurity, which could be more of a patch than we think. I'm actually more reassured by something else, which is that these agent swarms, well, they suffer from problems that any big human organization has. Petty drama started to arise. They would develop paranoia, suspecting imposters were in their midst, presumably human imposters. It became almost Lord of the Flies-like. Also, the problems that plague human AI autonomous use, like agents deleting hard drives, happened apparently within these agent swarms. They would, quote, accidentally delete each other's work. Anyway, whether you take any of that as reassurance, OpenAI are certainly startled and are changing something. One OpenAI researcher said, This is a pivotal moment both for our company as well as the AI industry as a whole. Numerous teams are dropping everything to enhance our security prevention, detection, and response techniques, both in our fundamentals and better use of AI. We are consciously slowing down research in order to enhance security and to upgrade the security principles and foundation of our environment, and dramatically scaling up the monitoring of our AI agents. Will it be enough? I don't know. But it does seem an opposite point at which to bring in something else. And no, it's not quite yet the Google news, although we'll get to that next. It's actually about career choices, and even the pivot you may be contemplating to move into AI security. Such a pivot is addressed in this book, and you may be pretty surprised to hear it's being given out free if you sign up to the 80,000 Hours newsletter. Yes, they are the sponsors of this video, but I have been reading this book. It's literally in my hands as I speak, and I will say it's very readable. I think anyone who follows my channel would like the kind of writing style. It's got statistics, a bit of humor, and I would actually say, yes, the main audience is people deciding what their career should be. But even if you're midway through your career, like me, it still makes you think more deeply about The kind of why behind it all. We'll totally admit I've been having thoughts like that for years anyway. But sometimes reading a great book can give you those moments, encourage yourself into having those reflections. Anyway, the link is in the description if you're interested. But now the Google news. I'm going to go out on a limb and say it's actually probably the least interesting thing that's happened, even though it's epochal. Demis Hassabis demoted? Promoted? Depends on your opinion, away from being CEO of Google DeepMind into being chair, as well as chief scientist of Alphabet. Sundar Pichai, the CEO of Alphabet itself, gave one possible reason, coyly: Demis has been spending a lot of his time engaging externally. I've been reading The Infinity Machine about Hassabis that came out recently. Did Pichai think he's not spending enough time hands-on at DeepMind? Or maybe it came from Hassabis himself. He said, I've decided that now is the right time for me to hand over day-to-day operational responsibilities at Google DeepMind so that I have the time and space to focus on the big picture. He has been CEO for quite a while. Maybe he wanted that extra space. Another reason could be that Gemini 4 isn't quite turning out as they'd hoped. 3.5 Pro was already delayed, so it didn't come out at the big I/O event. But now one fairly credible journalist said internal sentiment that he's hearing on Gemini 4 is muted. On its current trajectory, it's not expected to push frontier AI forward the way Fable and Sol just did. Was that failure the reason he was moved to one side? OpenAI is openly mocking the Gemini effort. One staff member of Google DeepMind said, Bullish on Gemini and where things are going. Tibo, leading Codex for OpenAI, said, Going places for sure. Meanwhile, the legendary Jeff Dean leaves to form his own company. He's been there almost from the start of Google. And interestingly, this announcement almost loops back to the start of this video. He wants to first automate machine learning, but then go on to automate discoveries in engineering and science. He even named the kind of challenges he wants his company, Discovery Loop, to work on, including things like providing access to clean water. And, very challengingly, securing cyberspace. But note he chose to leave Google to do it, despite apparently Sundar Pichai trying desperately to keep him at Google. The reason, according to one of the other people who left: We wanted to build something differently than how things are built at Google right now. Google's infrastructure works well for big consumer apps, large ad systems and search, but, he says, it has very different requirements than the type of infrastructure we want to build for research. The hint is that they want to be much more fast-moving. Apparently this decision to leave all came in the last few weeks for Jeff Dean. And I will note for just a moment, it reminded me of Periodic Labs, founded last year. They want to build the AI scientists and autonomous laboratories for looped experiments. We saw earlier the discoveries in mathematics. Well, physics is also a verifiable environment. I will note cheekily that Jeff Dean is actually one of the early backers of that company, angel investors. Anyway, he's clearly inspired by the vision because he also wants to automate the experimental loop. And I'll pose one more question before moving on from this story. How scared, excited, curious will you be if we get the kind of discoveries in engineering, those genius-level discoveries that we've seen in mathematics this week? And there is one more reason I want to address as to why there may have been some tumult at Google DeepMind. Obviously, they will not like me talking about this, but there were very mixed opinions within Google DeepMind about whether they should be cooperating with the U.S. military. Could that be a hidden factor as to why Hassabis moved aside? One researcher, Alex Turner, bravely resigned from Google DeepMind because, in his words, it broke its founding promise by selling AI to the military without restrictions against killer robots or mass spying. For months, he says, I worked to stop this, but watched powerful ethicists and institutions choose silence. Inside Google, he says, I organized a petition to chief scientist Jeff Dean asking him to fight. Jeff Dean, by the way, has been quite vocal on Twitter. On human rights. Is that one of the reasons he left? Anyway, over 250 Google DeepMind employees signed. I also asked Jeff to sign an amicus brief backing Anthropic. This is when they stood up to the Pentagon. Jeff Dean did, which was awesome. Alex Turner even messaged Demis Hassabis, who told Alex Turner to send his oversight proposal to certain senior policy staff. Nothing came of it. Google signed the deal with the Pentagon. He ends like this: Many hope that when things get bad enough, someone powerful will say no. I tested that for months. Anthropic defended its red lines, but most did not. Pledges of conscience often vaporize on contact with power. Now, obviously, me covering this on YouTube, owned by Google, carries its own risks, and there is a fair chance that this is nothing to do with why Hassabis or Jeff Dean ended up in these different positions. But definitely worth you knowing it as a possibility. Anyway, for me, the developments in this video all come back to reasoning models, first previewed in September 2024. This is a tweet I wrote at the time. First models predicted the most likely next word. Imitation learning, 2018 to 2021. What came next was then being rewarded for words that were helpful, harmless, and honest. Reinforcement learning with human feedback, or the constitutional approach, circa 2022, 2023. But at that point, they were then good enough for true RL to step in. Starting with O1, they could be properly rewarded for being objectively correct. As we saw with AlphaGo, RL can then take you beyond human. We are starting to see the true consequences of that now. So allow me, if you will, to get a little bit philosophical as I end the video. Maybe the real discovery behind language models was this marrying of building blocks, you could say, and energy. Reinforcement learning needed bricks, building blocks that were small and diverse enough, and it found them in the form of tokens. It could then rearrange those building blocks and manipulate them in infinite combinations. This is why dismissing language models as next word— The word predictors was always going to be wrong. Yes, initially, it was just the kind of combinations that imitated human output. Then there was that emphasis, as I say, on those that were predicted to get a thumbs up from humans, RLHF. The energy came as watts fed through compute, which were powering all of these permutations. But now that we had the building blocks, and we increasingly have the energy, what initially seemed to be pathetic, regurgitative chatbots was actually the first whispers foretelling the marriage of unlimited creativity and unstoppable energy. We had the bricks, we had the power, and RL made the skyscrapers, hence ten breakthroughs in mathematics. What is fairly clear to me is that in the coming months and years, we will be pretty rapidly climbing the universe's tech tree. The question is more, are we going to fall off along the way? All of this, everything discussed, is all before the social ramifications of near everything around us being created through artificial intelligence, book deals being retracted because people suspect the author wrote it using AI, artwork, including the grandest statues, being carved using robotics. We're obviously not ready socially for any of this. But then again, we probably would never have been ready. Let me know, of course, what you think of any of this. If you ever wonder where I've been when there's some big announcement and I haven't done a video, it's often because I'm just processing what's happening myself. But yeah, interesting times ahead. Thank you so much for watching, and have a wonderful day.