Overview
This episode is a fast read on a messy 24 hours in AI model releases. The host argues that the real story is not who tops every leaderboard, but how three frontier labs are pushing a new pitch: near-frontier performance at much lower cost.
Most of the focus is on OpenAI's new GPT-5.6 line - Sol, Terra, and Luna - and how those models compare with Anthropic's Fable, Opus, and Sonnet, plus pressure from Grok, Meta, and GLM. The host's main point is that pricing is starting to matter as much as raw capability, especially for business tasks where "good enough" can change workflow habits fast.
Key Takeaways
OpenAI's release looks strong because the host says Sol often lands at about a third of the cost of Anthropic's comparable models, while staying close in performance and sometimes beating them. That changes the buying decision. If the output is close enough, a lower price can matter more than a small benchmark edge.
The host spends a lot of time on benchmarks that try to measure real work rather than toy tasks. On Agents Last Exam, OpenAI says GPT-5.6 Sol hit almost 54 percent versus Fable's 45 percent. The host treats that as more meaningful than a generic leaderboard result because the benchmark was built from real projects across 55 industries, with expert-designed tasks and reproducible scoring. His broader claim is that we did not wait for coding models to get near-perfect before people started using them first, so the same shift could happen in finance, operations, and other white-collar work.
There is a check on the hype. The host points out that some coding comparisons are less clean than they look because benchmark aggregates can reuse the same underlying tests, and some tougher benchmarks did not include GPT-5.6 Sol results. He also notes that Grok 4.5 and GLM 5.2 complicate OpenAI's value story. A model can be "almost as good but cheaper" than Anthropic, yet still look expensive next to Meta, xAI, or Chinese competitors.
Another theme is that verifiable domains are falling faster than messy ones. The host mentions a competitive coding benchmark that an OpenAI model appears to have effectively maxed out. His read is that when answers can be checked cleanly, models improve hard and fast; weaker results elsewhere may say more about evaluation difficulty, sparse training data, or limited reasoning budget than about a hard ceiling in model ability.
Practical Steps
If you buy models for work, do not compare only the top model from each lab. Test three tiers: a flagship, a mid-tier option, and one low-cost alternative from another vendor. The host's argument only makes sense if you measure output against spend.
Run your own benchmark on tasks you already do every week. Good candidates are:
- financial analysis or reporting
- workflow automation in Zapier or similar tools
- coding tasks in terminal environments
- quick app, game, or website prototypes
Track two things together: success rate and total cost per completed task. If a model is slightly worse but far cheaper, it may still be the better default.
For product and design work, test vibe-coding models separately from enterprise reasoning models. The host suggests that for game mocks, websites, and prosumer builds, you may not need the most expensive OpenAI option at all. A cheaper model from Meta or xAI could be enough.
Be careful with benchmark headlines. Check whether the result comes from a fresh benchmark, whether the tasks are reproducible, and whether the same tests are being counted twice in an aggregate score.
Notable Quotes
- "What if we can give you almost as good at a fraction of the price?"
- "There wasn't a singular benchmark that we beat... where we switched from hand coding first to AI coding first."
- OpenAI lead, replying to an Anthropic post: "I smell fear."
Full Transcript
We were all only supposed to care about the very top scores on AI leaderboards and the best vibes in our own workflows, but three frontier labs just asked us, what if we can give you almost as good at a fraction of the price? The answer to that question might just be why the lead for OpenAI’s super app just said in reply to an Anthropic post, which was giving away more usage, I smell fear. But more broadly, this video will be about giving you a dozen or so hidden gems strewn across the model releases, demos, background articles to help us all make sense of what happened in this last frantic, I would say, 24 hours. And that is all without even mentioning the somewhat disquieting posting paper from Anthropic about AI consciousness covered in depth on my Patreon. So I’ve tested the new GPT-5.6 Sol, Terra, Luna hundreds of times, including on my own private benchmark, and read everything I can, of course, by hand. Wait, that makes no sense. You get what I mean, manually. So let’s dive in. First things first, there are three new models from OpenAI, 5.6 Sol, Terra and Luna. Sol is only available on the paid plans. This combines with goodness only knows how many effort levels. On first sight, it looks like five effort levels, but there’s a hidden pro mode as well. But the good news is that the observations hold broadly across the board, so don’t worry too much about that. Because the truth is, across most of the benchmarks, whether you're comparing the biggest Sol versus Fable, or Terra versus Opus, or Luna versus Sonnet, Generally speaking, the OpenAI model will cost about a third of the Anthropic Claude series. And nor, by the way, does that mean you're always getting worse performance for that much reduced cost. OpenAI flagged up this benchmark, Agents Last Exam, where as you can see, the top scoring GPT-5.6 Sol on extra high, scores almost 54%, and that compares to Fable going all out on max, getting 45%. Yeah, whatever, just another benchmark, Philip. But wait, this was co-led by UC Berkeley, covers 55 industries, 300 experts were involved in crafting long horizon tasks that were proven to be economically valuable. The legendary lead on the benchmark, Dawn Song, said every task is derived from a real project that a human expert previously completed. No vibes, no human judges, fully reproducible. Okay, fine, you might say, pretty impressive roster of people who oversaw the creation of the benchmark, but going from 45% to 54%, is it that cool? Well, aside from the cost reductions involved with Sol, I would also point out that we never had to pass 90% on frontier coding benchmarks for developers to stop manually coding. There wasn't a singular benchmark that we beat, I mean, where we switched from hand coding first to AI coding first. So what's to say that might not now happen with finance, or many of the other industries covered by Agents Last Exam? I heard a report just the other day on Bloomberg where Coreve reported this month that they are backlogged by tens of billions of dollars in demand from financial firms alone. So might we reach AI first for finance and many other white collar domains by the end of this year, where you first get a model to do the task on the swanky new work tab of ChatGPT, before you review the output and tweak it? One benchmark, some of you might be thinking, could well have been gamed. So what? Well, Agents Last Exam is a pretty new benchmark, so harder for its answers to be found contaminated in the training data of GPT-5.6. And so, I might add, is Automation Bench from Zapier. Feels like just a few weeks ago that I covered the release of this benchmark. Again, it tests AI agents on end-to-end workflow execution using real tools across real business functions: sales, marketing, operations, support, finance, HR, built on real patterns from monthly tasks done across millions of companies. This time, the performance per dollar is not as stark a lead for OpenAI. You can see the 0.7% score lead for Sol on max, that run costing almost the same as for Fable, but still. Similar result, by the way, in the previous most famous benchmark for measuring real-world impact, GDPVal. This time, Fable actually has a slightly higher ELO, albeit at triple the cost. A couple more impressive examples, and then the counterargument, lest you think I'm biased towards OpenAI. Artificial analysis combined multiple coding benchmarks into one aggregate analysis, and you can see what it found. Lo and behold, GPT-5.6 Sol scoring the highest, getting 80 on the index versus Fable’s 77, again, at a lower cost. Might seem like this is reinforced by the scores on TerminalBench 2.1. Think of that as a model’s ability to complete fairly complex tasks using the command line terminal, like writing, debugging, running software, multi-step tool use. But wait, it must be added that that artificial analysis coding index covers the very same benchmarks: TerminalBench, DeepSuite. What I’m trying to say is that this little collection of benchmarks might make it seem that Sol is better even than Fable on its favorite domain, coding. But it’s two measures, and there have been questions about DeepSuite, and two more recent lauded and harder benchmarks, FrontierSuite and Suite Marathon, did not have GPT-5.6 Sol results published. In the case of Suite Marathon, Software Engineering Marathon, involving multi-hour tasks with tens of millions of tokens per trial, you’ll notice Grok 4.5 in the lead. All that data that Grok now has from the Cursor acquisition by SpaceX AI does seem to have really helped propel Grok. You'll see Fable 5 trailing on this benchmark. Also bear in mind this, which is the same argument that might tempt you to go from Fable 5 to GPT-5.6 Sol, the fact that it might be almost as good, but a lot cheaper, might also nudge you toward Grok 4.5, or maybe the slightly cheaper still GLM 5.2, a Chinese model. When those models are added to the chart, OpenAI’s curves might not look as appealing. Okay, but that point may have shrunk your enthusiasm a bit too much because if we turn to an abstract pattern recognition benchmark, Arc AGI-3, the successor to some of the most talked-about abstract reasoning benchmarks in the industry, Owen graded, by the way, to be especially penalizing to models, GPT-5.6 Sol still does well. Yes, it gets just 8%, but compare that to other models struggling at below 2%. I think Anthropic didn't even run Fable because of the cost involved. Then there is Competitive Coding. Just in the last 24 hours, it was announced that an OpenAI model, possibly an internal model, literally broke a competitive coding benchmark, just aced it. Kind of a slight warning shot that if a domain is verifiable, if you can check an answer is correct, then before long, there will be a model that crushes it. All these other sub-100% scores that I'm spending most of the video talking about is more an artifact of those domains either having messy data that's hard to verify, or of there being just not enough of the relevant training data inside the models, or the models not being given enough of a reasoning budget. Which brings me to another benchmark I want to cover. Maybe a whole new class of benchmarks, which is that companies are now making entire games, playable games, as part of their release notes to show off the capabilities of their models. Demonstrating that they can create tasteful, ergonomic, and functional interfaces. Showing off that models can use a browser to check the results of what they've created. Indeed, you can do the same. I ran the very same prompt that I used for Fable on, this would be, 5.6 Sol Ultra, and out we got this game with a title page that I think is significantly better than Fable’s output. I will say that 5.6 twice marked up the sound settings, though, so it's not all smooth sailing. I've published the mini-game, by the way, in the description, so you can play if you like. My quick summary would be that it's not as visually stunning as what Fable came up with, but I love that the little companion — this is essentially a Pokemon clone, by the way, but set in the Redwall universe — that you can actually see the companion. So when you move, it just follows along. That's pretty cute. If you use the lightning setting, which does use up more credits, of course, I got results within 20 minutes, where Fable took more than an hour. But here's where I want to bring in Meta's MuseSpark 1.1, because one of the most prominent benchmarks that Meta celebrates is its ability to vibe code. They cite VibeCode Bench. This was created by the independent valves.ai, and if we look there, you can see MuseSpark getting a score that's not that far off Sol, 72% versus 81%, but at around 35 times less cost. This is that same awkward cost efficiency point from earlier. OpenAI can't lean too hard into their model being almost as good but cheaper if there are other models — like those from Meta and xAI — that are almost as good as the GPT series, but way, way more cheap. The reason I bring up Vibe Coding is that if you're into, say, game design, mocking up a website, more consumer or prosumer use cases, then maybe you don't need Sol after all. Maybe you don't even need the much