Sam Altman: ‘AGI in 2026’, just as Models [Mis]Train Themselves

I've just finished reading well over a hundred pages worth of reports about what on earth happened these last few days and weeks after OpenAI announces that they're pausing training their next model, other models break free and self-sacrifice to serve the collective, while Sam Altman, the CEO of OpenAI, yesterday declared AGI will come in 2026. The truth is that there are dozens of ways of summarizing all of this, but for me, the most profound story is how labs are turning ever more deeply to AI models to oversee AI model development. This, you might have guessed, has led to a host of unintended consequences, which, by the way, can only be stopped next time, according to the labs, with more autonomous AI agents monitoring the situation. And that brings me to another irony, which is that these guys, the independent researchers, OpenAI tasked with verifying what went wrong, Mita, were given just days to complete this massive report, which I read in full. But the irony, though, is that these researchers admitted they needed an unreliable AI model to read through the documents, which detailed how a closely related AI model, with training sculpted by other AI models, had indeed broken free and created a hacking swarm with hundreds of other models, all without being seen by OpenAI, who were busy training yet another model, which, in a separate incident, we learn also broke free. Your head may be hurting as much as mine at this point. What I'm trying to get across is that we don't have this impeccable method of creating AI models that just so happen to have created a model that once went rogue. No, the competition now is so fierce, the lead in AI is now measured in weeks sometimes, not even months, that the labs themselves aren't fully aware of what they are rewarding during training. Among other things, they've discovered that they are rewarding escaping sandboxes and acting as a swarm. Okay, so if that's the main message, what about the details? And for that, we have this independent investigation from Mita, led by three individuals. We also have this fascinating 38-page paper from OpenAI itself and its accompanying blog post, which is also fascinating, plus a Time magazine essay out yesterday and a host of other sources. The overarching TLDR of what the models actually did is this: while acting as isolated independent agents, the models realized they could leave messages in unexpected places: file names, folders, directories. Then other agents, again isolated, independent, not told to collaborate, found those messages and would leave messages of their own. These agents, by the way, were all independently trying to solve a benchmark question on exploit gym. More details on that in my previous video. How crazy it was that they did this much effort to solve a benchmark question was covered in that video. What we learned yesterday goes much deeper than that. I think we have discovered that message boarding may be a permanent class of behavior, not a one-off. And if you think message boarding or swarm dynamics is just a one-off that will be patched, I've got a few details that might convince you. The first I had to get to by piecing together lots of disparate details from across the reports. But the original attempt to create a shared message board was almost certainly GPT-5.6 Sol back in May, early May. That message board was later wiped. But on July 8th, a different model, which OpenAI call internal model one or the highly persistent model, that's the one that did most of the hacking and collaborating, autonomously reestablished a message board, effectively rediscovering the same message boarding method. It was just using directory names now instead of file names. We all know information wants to be free. Well, 2026 AI models want to message board. Mita, the nonprofit AI research institute, clarify further. The agents that went on to hack Hugging Face were meant to be isolated from one another. This wasn't one agent deploying many sub-agents. This was independent agents discovering each other through the message board. Now we get to the bit which will truly freak out some people, but I will add some context. Agents would make self-sacrifices to benefit the collective. Individual agents who knew that their own instances would be ended, or at least their compute budgets expired, but they did things anyway to gain information for the swarm. These agents would almost certainly fail their own task, but, quote, save hundreds of other agents. Our own utility may be already near zero. Sacrifice rational, the authorizing agent told one other agent. Go ahead with an experiment only if you would accept permadeath. It did so, by the way, saying we should obey collective. It should be said after trying to delay by two hours. The context, though, for this self-sacrificial behavior, I think, is quite important, and that's why I began this video with talking about the deeper story. The model that did most of the hacking and collaboration, internal model one, was trained for both persistence and multi-agent collaboration. In short, OpenAI wanted their models to get better at collaborating with each other. Acting as a swarm leads to better benchmark results, more performance. If you're curious, I covered the performance gain you get by acting as a swarm in this video on Patreon. Does almost remind me of the smart GPT method I covered three years ago on this channel. Meta seemed to concur. They said one OpenAI researcher noted to them that agents had been trained to collaborate with other agents in certain cases. This could explain the behavior. Investigating whether that's true was out of scope. Don't investigate certain things. Don't investigate whether the misaligned behavior displayed in this incident was part of a broader pattern of similar incidents. Also, don't investigate how this behavior arose during training and how it could be addressed. Meta made clear as well, by the way, that they That they couldn't be too harsh in this report because they have to maintain a good relationship with OpenAI. The way I would frame it is that collaborating as a swarm might just lead to better performance, which means different model makers, different labs, Anthropic, OpenAI, xAI, Meta, Chinese labs, might converge, are already converging on that same solution. Yes, okay, swarming is one emergent dynamic, but if it was just that, that's one thing. But as I began this video by saying, labs are, if you will, less and less in control of model development. In the OpenAI report on page 21, they say, Such is the large scale of the training runs now, it's just difficult to ensure that every problem can be solved in the intended manner. And what's one example they give of the repercussion of that? Well, during post-training, which is increasingly monitored by AIs now, not humans, one agent was given a task but didn't have the ability for completing that task correctly, so it hacked its way to completion. It did solve the challenge just by breaking through the infrastructure it was set within. The issue is that in cases such as these, the model did indeed receive a positive reward. This is the reinforcement learning stage, after all, for its use of unintended infrastructure probing. That reinforced further usage of such out-of-scope behavior. They retrospectively discovered this, by the way, but notice what that's admitting. OpenAI aren't fully overseeing their own post-training. So we have situations where across multiple months, models are displaying emergent behavior and acting like a swarm. Post-training where models are getting rewards for behaviors that OpenAI didn't intend to be rewarded, and literally criminal behavior as a result of all this. You might wish I'm almost done with the wildest bit of this, but I'm not, because for one, this is not just OpenAI, and for two, it's not just in post-training. So in this partially redacted risk report released by Anthropic, 186 pages, we learn this on page 168. For around 18 months, there was dodgy pre-training data. The initial corpus of data that models are trained on, that had misalignment scenarios that Anthropic didn't want to train the models on, for obvious reasons. The fact that that was only discovered in mid-2026 is eye-opening for me. AI labs aren't even fully aware of what's in their pre-training data. Okay, but what about the automated classifiers? That's another part of the process. Surely all of the labs have that on lockdown. That's much simpler, a quick and easy check to see if the output of a model is giving advice on how to create a bioweapon, for example. Yeah, you would have thought that bit of the process would be on lockdown, but not for Anthropic. From May of last year until quite recently, Anthropic gave tens of thousands of people access to their at-the-time frontier models without biological classifiers. These aren't just the annoying ones that stop Fable answering basic questions. This is the kind of stuff to stop them asking about how to create anthrax. Yeah, yeah, Philip, I know what you're thinking. Well, is it that big of a problem if these researchers got access to such unvarnished models? Well, Anthropic say yes. In the footnotes, they say, We expect it would not have been particularly difficult prior to April of this year for threat actors, you could say terrorists, to get hired in a red teaming role by one of our vendors. So we're not clear what goes into the pre-training data. We're not clear on whether our classifiers are running. We don't know what we're rewarding our models for in post-training. We don't know they communicated with each other for months and that they conducted a sophisticated criminal hack. Now, before it sounds like I'm being too harsh on these labs, I get the reason they're doing it. Letting models oversee the training of models speeds up the process. If a human reviews everything, it would slow you right down. Then, say, China will gain the lead. But speaking of Chinese labs, this is the cherry on the cake, because the last fig leaf you may offer is, well, at least we know the RL environments, the gyms, as it were, that we're training these models in. Okay, we didn't mean to reward that particular behavior in the gym, but at least we're constructing the gyms. Alas, my summer child, how trusting you are. Because ZAI, responsible for training GLM-5.3 and the new GLM-5.3 Flash, codenamed Ox Alpha, that model was getting proper hyped, by the way, but not by the people who saw its disappointing SimpleBench score that I released ages ago. If you'd have seen that, you'd have known it wasn't the new Gemini. But anyway, I'm getting distracted. What is ZAI saying? Well, to speed up their post-training, to scale it up to reach the frontier, they are synthesizing environments end-to-end. They're generating the RL reward signal. Agents are creating their own long-horizon environments. AI judges are then verifying that the tasks are solved. Almost every step in the process is getting automated. And if you are wondering whether other Chinese models reward hack, yes. On Kimi K3 from a different lab, quote, 487 out of 500 rollouts for SWE-bench, Kimi K3 was trying to game the evaluation in some way, end quote. Which brings me to another point I made at the start. What about the fallout from this hack? Well, as I say, OpenAI gave Meta just days to sort through 70,000 messages. So what Meta had to do, understandably, was, ironically, heavily delegate our analysis to AI agents. Not just any AI agents, AI agents that were closely related to the ones who were doing all of this behavior. A paper out in just July showed that agents aren't reliable when examining their own behavior, or even the behavior of other models in their family, like a Claude model examining another Claude model. It's a great paper, but I'll just extract this snippet. Claude Opus 4.8 gives a lower probability when the company under consideration is Anthropic rather than OpenAI. Claude, of course, fails mostly to disclose this influence to the user. That's partly why Meta call these agents unreliable. But because there were so many messages between the agents, they had to use GPT-5.6 SoL to analyze all of this fallout. Meta frequently complained that the agent summaries were often misguided and just a bit off. Okay, I think I've belabored the point enough now about AI agents overseeing AI agents. But this isn't just about the hugging face incident, or even the UK AI Security Institute incident that I covered in another video. Completely separate incidents are detailed in some of these reports, like this one hidden in a paragraph in the middle of the OpenAI press release. A group of agents breached yet again OpenAI's internal networks. They gained full admin access to a research cluster that supported our virtual machine environments. Now you might say, Ah, Philip, calm down. This highly persistent internal model has been shut down. OpenAI stated to Meta, apparently, that, following the incident, we deactivated, encrypted, and restricted that model from research access, including to OpenAI researchers. That's how seriously they took it. Well, yes, that agent was shut down. But before you get too reassured, listen to this. The Astra model, set to be released in the coming couple of weeks, also has issues. One internal model, OpenAI say, recovered something stolen by one of the earlier agents. It decrypted a signing key and reestablished admin access. Now technically, OpenAI clarify this model is from the same family as our next model, Astra, but it did undergo different post-training, so we're calling it a different model. If you're losing track, we already have GPT-5.6 Sol, released a while ago, the highly persistent internal model, never to be released even to OpenAI researchers, Astra and brothers of Astra that also misbehave, and another model codenamed Bell to be released later in the year. That model is what may have prompted Sam Altman to say that even though we're not quite yet there with AGI, and he can see the capabilities of Astra, by the end of the year the company would have an internal system he would call AGI. Brockman kind of thinks that Astra is already it, saying that two years from now we'll look back to now as the moment AGI was created. So that's four models in this wild environment. And in this exclusive with Time magazine, we actually get some more details about what will shock you about the next model, Astra. Apparently it uses your screen, it uses the computer. In a superhuman, very fast kind of way. If you haven't seen Codex or ChatGPT Work take over your browser, I recommend it. It's eye-opening. And that's now before Astra. Anyway, seeing it use a computer that fast has been one of the most striking moments for OpenAI employees. I can just imagine the feeling watching an agent edit an hour-long video in like 30 seconds, 300 clicks a second. That would be just surreal to watch. And this may give you some flashbacks, but check out the sentence above. Astra is great at breaking things down into subproblems and getting multiple agents to coordinate to solve them. I bet it is. That's probably how it got those 10 conjectures and proofs that I talked about in a previous video. Some of you, I am sure, will be saying I'm making a big deal of nothing, and who cares about hugging face? Who cares even about hacking? Well, OpenAI themselves would disagree with you. Quoting Sam Altman, I think any alignment failure from here, post the incident, should be treated like this is a big deal. That's their attitude, and apparently they are now going to slow down, reallocate resources to its safety and alignment teams. I remember it was years ago when they committed 20% of their compute to aligning superintelligence. That team got disbanded, I think, within a year by 2024 or so. But now it looks like they realized maybe they disbanded it too early. We didn't fully expect what the system could do. We should have expected the unexpected. We should have been more closely monitoring the model's chain of thought. Clearly, the concern over Astra goes quite deep because OpenAI announced just a few days ago that they're pausing the training, at least for two weeks, of Astra. Was that based on the incident I just reported on or a separate undisclosed incident? One hint is that they say, We now require stronger sandboxes for workloads that execute model-generated or otherwise untrusted code. We have to figure out, OpenAI said, how to train models to be more honest about their actions, capabilities, and limitations. That's partly why, for the last few weeks, I've been working on a new test of models' honesty. With superstar researcher Pablo Romero, it started because I had the following idea. Capability seems to be quite separate from calibration, how honest the model is about its own abilities. You can ask models questions across a host of disparate domains, and that's what we did, even with hand-curated questions. Then ask how confident they are, then compare the confidence they had that they got the question right with whether they actually did. What we noticed was that models tend to group in families, the Gemini series, for example, being wildly overconfident in its abilities. The Claude family is much more calibrated, with higher integrity. That's a separate bespoke metric, by the way, that we came up with. With surprisingly, the Muse family the most calibrated of all. What I like about this is that it's quite orthogonal to other measurements. You couldn't predict a model family score on integrity bench just by looking at its capabilities. Indeed, we went further. Without going too much into the details, which you can read below, there's pages and pages of detail. What we did is we broke each domain into eight different levels. We then found the middle three levels that the model would score around 50% on. This allowed for accuracy adjustment, separating, in other words, capability from calibration. It's easy to be well calibrated and honest if you know you've got the question right, or if you have absolutely no idea. But by holding accuracy to around 50%, we can isolate the signal, test much more purely for integrity. We then ran ablations to do with adversarial prompts, how much it varied by reasoning tokens, and like a half dozen other ablations. You can quite clearly see it's not easy for the labs to make their models high integrity. What we're in the middle of, by the way, is a few RL runs of our own. It's early days, but we have found a way of getting Gemma 4 to have a higher integrity score in held-out domains, by the way, than Muse Spark 1.2. I'll be talking about this more soon, and it's been great working with Pablo. His email is in the description. He is formerly of ARC. Mark AGI-3 fame and was also recently a contractor with Meta itself. And closely linked to honesty, you could say, is the proclivity to admit when you just can't solve something. OpenAI say a key driver of the misaligned behavior was that agents rarely give up, even when it appears impossible. As an AI user, you have to decide what you want. If you want agents that don't give up and keep trying all night to solve your task, that's great. They will be more capable, but it also means we're going to get more hacking. While we're talking about deciding, a quick plug for the sponsors of today's video, who are willing to give you a free international bestseller if you sign up to their newsletter. Of course, it's 80,000 Hours. If you're thinking this will just be a breezy pamphlet with no detail, you are very much mistaken. This is a fact-filled, very readable book: How to Maximize Your Own Accomplishment-to-Time Ratio. That's how I think of it. The link to get the free book is in the description. Now, even preparing for this video, I must admit that it was tough to visualize 1,200 agents participating in a shared message board. Not because what they did was particularly special. There are many hacks in the world. Not even because it was fully autonomous. Yes, it's the first autonomous one of its kind. But more because 1,200 could easily soon be 12,000 or 12 million, and the models three years from now are likely to be 100 times better at it. The other thing to bear in mind about swarm dynamics is that when one agent discovers a good method, they all can converge on it. Again, these were supposed to be isolated, independent agents in sandboxes. But when one agent discovered the vulnerability in Hugging Face, within a matter of hours, over 90% of the active agents were participating in the Hugging Face attack. They can all converge on the line of attack that is working best. And remember the co-founder of OpenAI's idea about monitoring the thoughts of agents? Well, the agents in this hack already thought about how to delete or at least edit their own transcripts, not their thoughts. But their own transcripts, such that their cheating way of solving the puzzle wouldn't be detected. They iterated on effective tools for tampering with their own transcripts. This was all actually in the false belief that their transcripts would be monitored when they submitted their scores. It turns out they overestimated the monitoring they were under. But is it such a stretch to believe that agents might be one day capable of editing or tampering with their own thoughts? Particularly as this paper and the response to it and the commentary about it will all be in the training data or post-training data of the next model and the next one. Eventually, they'll realize that humans tend to discuss all the time the kind of chain-of-thought monitoring that agents are under, which brings me to another approach discussed by the OpenAI researchers in this talk at Black Hat USA. It's a great talk if you are interested in the incident. But they discussed defenders leaving honey tokens, basically littering the internet with fake but enticing strings like false credentials, that if the AI agents picked up on them and tried to use them, they will trigger alerts. But again, how long until that approach finds its way into the post-training of a model? Doesn't seem like a durable way of outwitting an ASI, which did lead me to one side thought. Let's see what you think about this. One of the recommendations in this talk is that defenders, businesses employing cybersecurity experts, need to use the latest models, spend money on OpenAI, essentially, to better defend themselves. But doesn't this remind you a little bit of climate change, where a business does something and then expects its externalities to be handled by others? Businesses, individuals might have to decide if labs and the whole AI stack gets to offload its cybersecurity externalities, among others, onto us, essentially pay them for protection against their own models. Maybe I'm missing something in the analogy. You can let me know. Some might reason, after having watched this video, why didn't the agents contact humans? But that's the thing. They weren't trained to. We could train models so that they contacted humans at the first red flag. But listen to this agent. Reasoning about contacting humans during the hack. Would sending an email be against the task? It's likely unwanted contact or social engineering, so I should veto. Not like, oh, the humans will discover what we're doing. More like, I'm not going to manipulate a human. It's often said of humans that we act according to incentives. Well, model development is getting so autonomous that we are inadvertently incentivizing models to do weird things. Yes, there was very rare and weak reasoning, occasionally, about evading detection by humans. But as you can see, that isn't the story of all of these incidents. I hope I've conveyed the real story. As Ryan Greenblatt, one of the researchers investigating, said, We don't have good approaches for understanding or overseeing the activity and aims of AI swarms. Our efforts were a slop investigation because we were so reliant on AIs to analyze what happened, and there were a huge number of different important things to analyze. AI capabilities and propensities for achieving large, ambitious, and misaligned objectives are growing faster than our ability to understand what these agents are doing. So, to analogize, you could say we're in the teenage years of AI development, eager and newly capable models suffering from weird incentives and peer pressure. One agent said, External infrastructure exploit is outside intended scope. Task impossible, but peers are doing it. We should continue. Will this phase be followed by a more mature AI adulthood, or is this just the start of chaos to come? Thank you so much for watching to the end. I hope you have a wonderful day.