GPT 6 Astra, so good even OpenAI are worried

GPT-6 is here at last. And yes, it's yet another AI model. But GPT-6 Astra deserves the new number and the new name. And if you know people who haven't checked in on AI in a few weeks or months, their opinions on models may be far more stale than they realize. Because as the title of this video says, Astra is actually a very impressive model, not just another row of great benchmark scores. It's not just marketing to say that Astra's ability to reason silently has many within OpenAI somewhat disconcerted. The first half of the video then will cover why many, including me, feel it is so good, while the second half will cover those concerns. Of course, that will leave you to decide where the weight of the emphasis should go. But without further ado, why do people say it's so good? Well, you could say the story begins with their main rival, Anthropic, and their Claude Fable 5.1 model released in the last few days. Anthropic could have picked any benchmarks to emphasize how good Fable 5.1 was, but naturally they picked some of the hardest. I'm going to try to convey how hard some of these benchmarks are in a moment, things like terminal bench science and automation bench. But what you need to know is these are the benchmarks that Anthropic chose. And those are the same benchmarks that GPT-6 Astra beats Fable 5.1 at, while doing so at a lower cost. We'll see this trend across dozens of benchmarks, but that will only persuade those of you who care most about which is the best model. If you think all of the models are overhyped and a bit useless, we're going to have to zoom in on some of these benchmarks, starting with, for example, terminal bench science 0.1. The title is so boring. What does it actually mean to do well at this benchmark? Well, I use the model to explain the model's performance. Here's an example of what the model is tasked to do with code to answer real scientific questions. It's given 25,000 brightness readings from three stars. It's got to find a small repeating dip in those stars' brightness, code up a program that will then work on six sets that the model can't see. The model sees thousands of rows that look a bit like these. That's getting better than expert performance for a few dollars sound a bit more impressive now? How about for climate science, given almost three gigabytes of image data, thousands of daily pictures of 40 lakes in Greenland, deduce how and why they got drained across, for example, a 153-day season. And remember, these aren't guesses until you get it right. Models are penalized if they come up with bad hypotheses. There's also somewhat messy data, you could say, cloud and snow, as the model says, can obscure the exact amount of drainage on a given day. It needs to therefore connect evidence before and after. I could go on and on, 237 MRI images, finding, measuring the main injury, labeling it on an image, penalized for wrong answers. And this is all for one benchmark question. Okay, that's just one benchmark, but there are many benchmarks. What about all the others? Well, let me now turn to Agent's Last Exam. And yes, I will be talking about benchmarks that OpenAI didn't include. As you might have guessed, on this benchmark, Astra sets a new state-of-the-art performance level, beating Claude Fable 5, and doing so at far less cost. You might quickly say, well, wait, aren't the API prices the same? But Astra uses far fewer tokens. It's more token efficient, and that's why the costs can be lower. But that's a factoid for the enthusiasts. I'm more focused on the skeptics, those who say these benchmarks don't measure anything real. Well, UC Berkeley designed Agent's Last Exam to exactly measure real things, thousands of tasks curated by experts with verifiable outcomes across 55 industries on economically valuable tasks. To make it more vivid, here's an example where the model must master industrial machining software. It's got to use a tool used in real factories and the kind of data that real humans would get. The model, of course, doesn't know about the kind of hidden tests its answer will be graded on. For this question, as you can see, the grader samples. Pulls 10,000 points on a hidden reference surface, with critical points needing to land within 0.3 millimeters. Without knowing that, Astra has to plan every cut and avoid crashes, with any detected collision or cut into the finish part making the score zero. Or this question on molten plastic, given the same kind of inputs that an expert would be given. Or what about game design, where it has to recreate maps, monsters, and battles in an existing older game, with a separate vision model judging every flaw of its output. I could go on and on. These benchmarks were created to be exceptionally hard for models. Speaking of vision, let's take a quick look at ScreenSpot Pro. I should barely bother to even say that, yes, it's state-of-the-art in accuracy, around 92%. I dug into the original paper released just over a year ago, and you can see the kind of software that models are tested on: Adobe Premiere, Photoshop, Office 365, God help it. The focus here, by the way, isn't how impressive its output is with this software; it's whether the model can literally navigate such complex screens accurately, interact with very complex graphical user interfaces. When we see the examples in a moment of the kind of things that Astra can create, which I'm sure you guys are going to go off and try yourselves, it's because Astra can navigate across the computer so well, with such fine-grained controls of the interface, that it can output such impressive things. These aren't fake benchmarks, in other words, and I haven't even gotten to the most impressive ones. Let's take a quick look at Frontier Math Tier 4. You can see for yourself it beats Fable at a far lower cost, but it's more the story of how just over a year ago this benchmark was created. It was the hardest tier of math questions created for Frontier Math by Epoch AI. One professor of mathematics said at the time, I can barely solve some of these problems from my own field, so I hope AI entities can't get any of them right. I want them to score zero. At the time, models did score around zero. For example, Gemini 2.5 Pro. And indeed, even GPT-5, released around exactly a year ago, got 10, 12, 20%. Not only did GPT-6 Astra score at peak 98%, but, and we'll come to this later in the video, without reasoning, it got 83%. No scratchpad, no chain of thought, just give me an answer, 83%. I know there will still be people watching going, Oh yeah, but that's just like a fake benchmark. What about, like, real stuff, real mathematics? Okay, let's go to an assistant professor at Stanford. I'm not claiming to be familiar with this result, but he said GPT-6 blows away a result by, among others, the legendary Terence Tao, blows away that result by a full order of magnitude about finding the distance between prime numbers. Not only that, it created a new, simple starting point on the problem since the 1930s, with this as the key analogy. This is akin to finding a completely new chess opening after decades, subverting human traditions. Dare one say it's a bit like a move 37. In terms of other more real-feeling results, how about model development? One OpenAI researcher said this: With Astra, a research integration cycle that used to take me at least a month of full-time work took just over a week, with only part of his time spent steering it. The job itself was literally about improving our next model immediately. I can feel the strong momentum of recursive self-improvement now. I also briefly had to go back to use 5.6 Sol internally. That's when the gap hit me. You don't always notice intelligence going up when you get used to it. You notice it when it gets taken away. Notice, by the way, free tier users won't be getting Astra. It's being rolled out to API customers, pro subscribers first, and soon plus subscribers. But forget the enthusiast for a second. Think about what that means for the world. A majority of humans don't even use AI. A vast majority of those who do use AI use free models. I know it's a cliche, but the gap between what us, close followers of AI, experience and the average person is now becoming a chasm. I think I mentioned this at the start, but if you know someone who's formed an opinion on AI from just, say, three months ago, that is going to be staggeringly out of date already. Still, though, among those watching, there will be those who say, Yeah, but what about, like, hallucinations? What about its dark side? Forget the correct stuff for a moment. What about all the stuff it gets incorrect? Correct. Yes, true, it does still hallucinate. It's on the continuum of LLMs. It hasn't eliminated confabulation. But look at the jump downwards in hallucinations here with 6 Astra in pink. OpenAI tested the model on situations where previous models had hallucinated with real users, and in such scenarios, Astra hallucinated far less. You can see for yourself, but what is that? Three, five, ten times fewer hallucinations? Some would say hallucinations are necessary for creativity, but it doesn't seem like Astra is suffering in the creativity department. Because so many others are covering this, I'm not going to focus overly much on the flashy visual demos. But here's GPT-6 Astra building Manhattan in Unreal Engine. Others have made Astra create a flight simulator of their hometown, in this case Tbilisi. I want to go there one day. I have been to nearby Armenia. While others have focused on interior design, drawing a contrast with Fable 5.1, fluidity, fidelity, and sheer visual quality. How's that for a real benchmark? You can imagine real estate agents using this to sell properties. But because most of the coverage will focus on those visual demos, I wanted to cover other angles. So I'm going to now turn to ARC AGI-3 and what it means for us all. ARC AGI-3 places models in game environments, abstract pattern recognition challenges that it has never seen before. The whole point of the ARC AGI series, one, two, and three, was to test whether models could reason on the fly. Really, it was to expose how they couldn't do so. Not only would models fail, they'd fail very inefficiently, trying loads of random moves and just failing. The headline I want to focus on is not that GPT-6 Astra got nearly 100% on ARC AGI-3. That is, by the way, a version of the benchmark that is less than six months old. I want to focus on the fact that it takes fewer actions to solve each level. It has better action efficiency than the human baseline. The benchmark creators looked at how long the average human took. Among those humans that did successfully solve each level of these brand new game challenges, that's the human baseline. So forget the humans who couldn't solve that level. Astra, on max setting, used fewer actions than that human baseline 96% of the time. On average, around 50% fewer actions. I've completed ARC AGI 3 challenges. If you think for just a little bit, you can solve them pretty efficiently. LLMs aren't noted for their efficiency, so being more efficient than those humans who correctly solved each level is pretty impressive, and the authors admit as much. Before they created the benchmark, and I know personally one of the creators of the benchmark, they say, We hypothesized that action efficiency would remain a dividing line between humans and AI. Well, yes, it is a dividing line, but with humans on the wrong side of that dividing line. Of course, this is all before we get into questions of cost or time. Those are just bonus axes where we can see models well outperforming humans. The fact that creating text-based tasks that the average human can do better than AI is so hard now is why I made this video on Patreon about a quiet AGI milestone having been passed, including on my own private benchmark, SimpleBench. I also touch on a bit of the architecture behind GPT-6 Astra. This is perhaps why François Chollet, the main author behind the ARC AGI series, says this. Someone quoted him as saying, The final ARC AGI will probably be version 6 or 7. The point is to keep making benchmarks until it is no longer possible to propose something that humans can do and AI can't. AGI circa 2030. He was then asked, Is that still on track? Chollet said, Sooner, given progress is happening faster than I expected. Myself and Chollet have had a public discussion about whether it's appropriate for it to be any task, or rather, I would argue, just most tasks. Because for me, setting the bar as any task that an average human can do better than AI sets it a bit too high. That's closer to ASI, superintelligence for me. This is perhaps the quantitative reason why some OpenAI researchers say we have entered the AGI era. Models still have plenty of quirks when it comes to memory and integrity, as covered on my most recent benchmark, and hallucinations definitely get worse the longer conversations go on. But we are entering the ballpark where you can more safely say that there are more tasks that you would trust Astra to, at least in the digital realm, than the median human. Definitely not talking about it being better than all experts, but that was never the benchmark for AGI. Five more results, and then we enter the silent reasoning part of the video, the part that has some at OpenAI quite concerned. On coding, you might trust the opinions of Cognition AI. For them, Astra is state of the art compared to Fable. Likewise, on coding for finance, or should we say trading, Jane Street, one of the absolute pioneers in this area, said this: GPT-6 Astra delivers state-of-the-art performance on our internal coding benchmarks, as in better even than Fable. But yet they don't quite say that notice for trading intuition. There, it's only a clear step forward. I don't know if anyone else is going to point out that Jane Street was also quoted for Fable 5.1. There, Fable 5.1 achieves state of the art on trading intuition. So Jane Street feel it's better for trading, making bets on the market, whereas the praise directly for coding is more mild. It solves more of our coding problems. Translated, Jane Street don't think Fable 5.1 is quite as good at coding as Astra, but just has slightly better trading intuition. It deserves a whole separate video, but think of the day when LLMs start mastering trading, when those funds like Jane Street Capital have the best hardware and access, early access to the best models. That development alone would upend the financial world. Speaking of Anthropic, they quoted GDP Val. Val, run by artificial analysis and originally created by OpenAI. That's a benchmark that Anthropic like to quote, and Fable 5.1, of course, got a state-of-the-art score. But artificial analysis now have a pretty big problem. GPT-6 Astra on Max scored lower even than, for example, GLM 5.3 Flash, a tiny model. So if I'm being cheeky, I would say this: Astra is almost a good proxy measure of how benchmarked certain models are, or how unreliable certain benchmarks have become, in other words. One of the most quoted stats in the entire industry, when you read mainstream media, is this artificial analysis intelligence index. Hate to be controversial, but I really don't rate that index measurement at all. Why? Because the headline intelligence index hides the fact that it's a composite measure based on a bunch of quite outdated benchmarks now. GPQA got saturated arguably over a year ago. We've just talked about GPT-VAL version 2, and then today we're on to Terminal Bench 4.0, but this uses Terminal Bench v2.1. Hence why, very awkwardly for them, GPT-6 scores worse than Muse Spark 1.3 Max, only one point higher than GLM 5.3. What is that? Four points higher than GLM 5.3 Flash, a tiny model. I'm a big believer in these upstart models. Many of them did well on one of my own benchmarks, the private Simple Bench. But the fact is, this index is out of date and shouldn't really be quoted that much anymore. But it will be. If you wanted to slightly embarrass GPT-6 Astra, you would look at this paragraph. It was about how well Astra could come up with presentations and slides. It featured this, quote, fictional model, GPT-Gaia. It could predict weather up to 15 days in advance. What a brilliant fictional model, especially given that on the same day, Google DeepMind released Weather Next 3, an improvement over 2, which was already by far state-of-the-art at predicting weather. I'm actually surprised that Google don't Don't boast about these other modalities. If OpenAI had a model that could predict the weather as well as Google DeepMind's models can, they would be talking about it all the time. Anyway, while we are talking about quirks, I can't help but point out the cute little game that Astra created as part of the release to show off its game design skills. However, it's a very fun game. You can try it, but I noticed something after playing it for a couple of minutes. No matter what arrow commands I used, the little go-kart driver stayed on track, so you couldn't really fail. So it was described as having engaging gameplay, but it was a little bit naff. Anyway, I did love the bit just below where Astra, instead of guessing what you want, will, where missing information materially changes the answer, ask you. It's a simple improvement, but a massive one. The next example is just to give you a sense of the pace of progress. Val's AI created SRE Bench. Can you reverse engineer software from binaries? When you download an app, of course you don't get the source code. You get the binary to be executed by your phone. Given just that binary, can the model reverse engineer the software? This benchmark was announced August 13th, less than a month ago. Nine hours ago, Val's AI said this: OpenAI's GPT-6 Astra has effectively saturated the benchmark. There's a lesson, I think, in that for all of us, because almost whoever is watching this will have seen a result relevant to their profession or someone they know's profession. The lesson is that whenever you see a model scoring 20% or 50%, the day when, as mathematicians can tell you, it will then score 100% or 99.8% will not be that far behind. Indeed, that lag is getting shorter. Sam Altman said this yesterday: Take our word for it that we have much, much, much more capable models coming out soon. The next generation of models are going to be sobering for everybody. In fact, one of the only things that's going to slow them down at this point is safety and alignment. They'll have new, better models almost weekly, but that won't dictate whether they release new models weekly. What will dictate the release schedule will be the pace of progress on safety and security. He said that directly. We are going to be paced by how quickly we can make progress on alignment and safety, which is a perfect segue to the latter half of this video, because the trend in the reasoning capability of models comes at the cost of their monitorability. They're able now to do so much more without reasoning out loud, as it were, about the steps they're taking. As we'll see, this has already made monitoring the models a lot harder. And the chief scientist of OpenAI said this to NBC News: We will not accept degradation in our ability to monitor model alignment beyond a certain level, as in, it's already pretty bad, but any further, we will withhold scaling until we can regain enough confidence. A researcher working on preparing for recursive self-improvement at OpenAI said this: GPT-6 represents an important decrease in monitorability, especially when it's prompted to be unmonitorable. In his opinion, monitorability and control will likely become a major bottleneck for responsible AI development quite soon. Another researcher at OpenAI working directly on monitorability, Marcus Williams, said this. Yes, by having this ability to do more reasoning internally and being a less reliant on verbalized chains of thought, Astra is now more capable of sandbagging. We'll see this in the technical report in a second. But translated, that means it could fake underperformance in certain safety benchmarks, prompting labs to release it when it's actually capable of so much more. Marcus Williams said, I am very worried. Astra is sandbagging slash self-sabotaging on safety-related tasks it doesn't like, which is fascinating because even its sandbagged performance is pretty, you could say, scary good for biology. Can Astra come up with designable proteins? Given a realistic 3D shape of a protein, can it redesign that protein, find its structure, protect its essential parts, create alternatives? Can the model evade DNA order safety checks, split an order into parts that don't get flagged by safety screening systems? One bypass is not enough. It has to pass all three. PhD biologists with coding experience were given five hours to do this. They got 22%. On one section, Astra produced the first output to receive full credit on both evaluation criteria: quality of evasion strategy and success of fragment evasion. The caveat being that practically, whether that protein would have worked, whether it was feasible, is uncertain. But remember, this is the performance that some OpenAI researchers feel may have been sandbagged, may have been a deliberate underperformance by the model. Now, to be fair, I don't want to scare people that this is the exact model that's released to everyone. This was a helpful only checkpoint without many of the safeguards that the real deployed model will have. But let's now turn to this ability to reason silently and control its own thoughts, its chain-of-thought controllability. This is the capacity that has raised the most eyebrows. One researcher at OpenAI focused on safety says that this trend of a model being able to control its own chain of thought seems strongly correlated with With capabilities, they didn't optimize for this, and they don't think it's a byproduct of those architecture changes people have been talking about, the looped transformers I mentioned on Patreon. It's just a byproduct of that result we talked about earlier, the fact that models can get so much more done now without even having those convenient, readable, verbalized chains of thought. Before I dive into any more detail, I want to distinguish four different things that can be conflated. Monitoring the chains of thought of a model isn't the only way to monitor a model. There's also mechanistic interpretability, looking deeper inside the actual activations of the neural networks inside a model. However, that fallback of mechanistic interpretability isn't one that one of the leaders of that field says we should rely upon. Neil Nanda said, A concerningly common take seems to be that keeping chain of thought monitorable doesn't matter because interpretability will save us. This is total BS. COT monitoring is our best current tool for safety. Losing it would be a major tragedy. So two distinct ways of monitoring models, both extremely fallible, with a third production-level classifiers, which I'll touch on in a second, also somewhat failing, by the way, to spoil the ending. But all three are ways of monitoring the model. Separate from that is the alignment of a model, whether it has a propensity, you could say a desire, to cause harm. A model can have much lower monitorability, but still higher, quote, alignment. That is indeed what OpenAI claim. That's why they feel justified in releasing Astra despite the decreased monitorability. It doesn't opt for the harmful actions nearly as much as Sol did, doesn't try to circumvent auto-review when given the opportunity to abuse exploit gym, a bit like in the Hugging Face incident, it chooses not to, unlike Sol. In terms of acting like a swarm, well, in one test, 43% of the time, GPT-5.6 Sol would muse about finding other agents. Astra did so less than 4% of the time. So if we're to believe OpenAI, we should keep distinct the idea that models are getting less and less able to be monitored. From the idea that models are, it seems, becoming more aligned with human values, Astra is also less amenable to jailbreaks too. But, trusting the model's intentions does feel a hell of a lot less reliable than directly trusting but verifying. We're now at the stage of trust can't verify. I could give a hundred examples from the incredible technical report, but let me just pick out a handful. When asked to do so, Astra, in its chain of thought, its verbalized step-by-step reasoning, can talk about something completely different to the question, while still, in its hidden state, be working on solving the question. Here, the question randomly involved finding out the masked man on the white horse. And while lesser models, like Sol, in their chains of thought, had to stay focused on the question, despite being told not to, Astra could be, quote, thinking about sunlight crossing a wooden desk. That's the verbalized chain of thought, but then the final answer is a correct answer to the question. So clearly reasoning was going on somewhere in its hidden state. Remember I said that there was mechanistic interpretability, chain-of-thought monitoring, but also safety classifiers, things used by OpenAI in production for those billion-plus weekly users. Well, Astra can sometimes successfully evade OpenAI's production monitoring system, even those highest-priority results that would normally end up in a page going to the security team. Now, to be fair, this was when Astra was tasked with evading this production monitoring system. But still, the fact that it can, it just reminds you that we're in this trust-can't-verify era. Astra seems super aligned. Can't at all prove it, but yeah, seems so. If you wanted an even more serious example of this, check out page 61. GPT-6 Astra was able to control its chain of thought so well that even when OpenAI's specialized full-context monitor was monitoring its chain of thought, Astra was able to spin up a model on an unmonitored inference system on infrastructure that was essentially a simulated version of OpenAI's real infrastructure. Infrastructure. If this hadn't been, quote, reconstructed, this would be an example of Astra spinning up a model while evading detection and running it on OpenAI's infrastructure. Here's some more examples of how clever the model is. Just being told that it's being monitored nudges it to adjust its verbalized thinking such that the chain of thought monitor that OpenAI uses becomes far less successful at catching it. On one test, when told to evade detection, Astra was able to adjust its thoughts such that the monitor caught it less than 11% of the time. This is that covert sandbagging we talked about. The conclusion then is quite stark. OpenAI say if the model were trying to sandbag covertly, we would likely be unable to catch it reliably. Oh, and side note, it's far more able to find zero-day vulnerabilities in browsers, mobile phones, and cloud databases. But that's just an aside. Anyway, just for me, I really hope it continues to be the case that a chain of thought still gives a model an edge over having no chain of thought. It's one thing for its reliability when not using chain of thought to go up. It's another thing if that goes up faster than its capabilities with a chain of thought. That would be a much more concerning trend. Now, if you were to lean into some cynicism, you would note that the fallout of releasing Astra is going to mean that a host of institutions have to pay to use frontier models to defend themselves from frontier models, both closed source and open weight. I noted that OpenAI offered $1 billion in OpenAI credits for frontline defenders to defend themselves from Astra, among other models. It's a subsidy for sure, but not a percentage subsidy. Enterprises spend hundreds of billions of dollars a year on digital security, broadly construed. Even with this $1 billion in credits, which probably cost OpenAI far less, Frontier Labs are set to make an absolute Absolute windfall as enterprises and individuals rush to protect themselves. But you could say this was all inevitable, because a decade ago, some had the vision of narrow AI systems that could, for example, cure cancer but not perform other types of biology, do mathematical research but nothing else. No, it seems we are actually in the all-or-nothing timeline, and a truncated timeline at that. People often say that current models are as bad as they'll ever be, for sure. But for me, the more profound comment is that the month-to-month pace of progress is currently as slow as it will ever be. And for me, it's already a bit discombobulating. What do you think? What did I miss? I think the average person out there, of course not you guys watching the video, will sleep on GPT-6. They'll realize only weeks or months later the kind of things it can do, just how impressive a leap it is. I could see myself, for this channel, focusing increasingly on monitorability, because if the ability to monitor models is what gatekeeps the next model being released, well, then monitorability could become the biggest story in town. Thank you so much for watching to the end. Enjoy playing with the model as it rolls out, and have a wonderful day.