Overview
Claire Vo gives an early read on Anthropic's Opus 4.8 after a few hours of testing in coding and business workflows. Her take is pretty clear: the model looks strong on paper and can do impressive first-pass work, but it loses reliability at the edges, especially when follow-up fixes, existing codebases, or grounded strategy work matter.
Key Takeaways
The headline from her tests is that Opus 4.8 seems best at the first 80 to 90 percent of a task. In Claude Code, she says it handled a one-shot feature build well: it took a spec, made a plan, coded autonomously for about 20 minutes, and produced something that worked on a preview branch. For greenfield feature work, that is a good result.
The trouble started once she tried to refine and extend the output. She says the model repeatedly stumbled on the "last 10%" and introduced bugs during iteration. More concerning, she saw outright hallucinations when the model was bug hunting, which stood out because she says she had not seen that kind of made-up reasoning in a long time. Her read is that the model was acting on hypotheses rather than checking against the actual code or data.
That pattern showed up again in existing codebases. When she asked Opus 4.8 to help rebase branches after a large underlying PR changed the code state, it needed repeated correction cycles and kept creating edge-case issues. Her point is less that the model cannot code and more that it has trouble figuring out where, exactly, to intervene in a messy, real codebase.
She also expected more initiative from a model positioned as a top coding agent. In a playful test, she asked it to build something her nine-year-old would find exciting. The prompt it proposed sounded ambitious, but the shipped result was more ordinary than she expected. Even after pushing it toward something "more fun" and 3D, she felt the model did competent work without showing much imagination or stretch.
On business tasks, her comparison between Opus 4.7 and 4.8 was even less flattering for the newer model. Using the same context, she asked both versions to review how she had spent the last three months and suggest where her focus should shift to grow the business. She says 4.7 stayed grounded in numbers and context, while 4.8 had a harder time finding the right data, overemphasized small signals, and produced strategy that felt vague. In a follow-up roadmap prompt, she says 4.8 even implied research it had not actually done, then admitted it had not searched the sources she asked about.
Practical Steps
- Use Opus 4.8 for first-pass builds, prototypes, and new feature surfaces where speed matters more than polish.
- Treat follow-up debugging as a separate phase. Verify every claimed root cause against logs, code, or tests before accepting a fix.
- Be careful with existing codebases. If you ask the model to rebase, refactor, or patch around prior work, expect to review branch state and edge cases manually.
- Ask the model to cite what it actually inspected. A direct prompt like "List the files, queries, and sources you used before answering" may expose weak grounding early.
- Compare outputs across model versions on strategy work. Claire's test suggests the older model may be better when the job depends on numbers, context, and restraint rather than broad synthesis.
- If you want ambitious output, state the bar explicitly and review whether the result matches it. The model may default to "good enough" unless pushed hard.
Notable Quotes
- Claire Vo: "It does really, really well until it doesn't do well."
- Claire Vo: "In real code bases, the edges destroy it."
- Claire Vo: "It 100% made up things based on a hypothesis, not data."
Full Transcript
Welcome back to How I AI. I'm Claire Vo, product leader and AI obsessive, here on a mission to help you build better with these new tools. Today, we have a very special mini episode because Anthropic just dropped Opus 4.8, their latest state-of-the-art coding model. And I got a few hours of early access and I'm here to share my very early thoughts about where this model is intended to perform well, where it did a great job and totally impressed me, and where there's still a little bit further to go. Let's get to it. As you can tell, I am not in my regular How I AI studio, and that's because I am so excited to give you my early thoughts on Opus 4.8 and couldn't wait between meetings to share what I thought. So to get started, I want to talk about what this model is, what Anthropic has told us about its benchmarks, performance, and what it's good at. So Anthropic is shipping Opus 4.8. It is supposed to be their step-change model for agents. And there's a couple things they've called out that this model does particularly well. It's supposed to be more honest, a less design flop, longer horizon autonomy on long-running tasks, and enterprise-ready. So it means it follows its instructions. And they're saying that Sweebench Pro, they're hitting 69.2%, which is almost 5 points higher than Opus 4.7, almost 10 points higher than GPT 5.5, and 15 points higher than Gemini 3.1. Now, this model is not cheap. It's $5 per input tokens and $25 per million output tokens. And then same as 4.7, effort defaults to high and fast mode can be a lot faster. This is what they say. This is what you're going to read on the blog post. And so on paper, this is a very exciting model. But I want to tell you my personal experience using this model and where I thought it did a really good job and, again, where it did not do a perfect job. And so when I was giving feedback to the team, I said, Surprising surprise, LOL. It's a good coding model. In that when I opened up Claude Code and asked it to do a fairly complex one-shot brand new surface area task, it did a pretty good job. So I asked in Claude Code Opus 4.8 to build a prototyping capability in ChatPurdy. So we make PRDs. I said, let's just go whole hog. Let's compete with the big boys. Let's make an entire prototyping tool. And I gave it some architecture decisions I wanted to make, what platforms I wanted to use, how I wanted it to function. It went through plan and then it autonomously coded for, I would say, about 20 minutes and shipped it. And when I pushed this live to my preview branch, it worked. And so I would say from a one-shot feature, it did quite a good job. The code was right and it followed the architecture I want. Where it failed was this last 10%. And this is really going to be my theme of this episode. It does really, really well until it doesn't do well. And I found it did not do well consistently over time with the same types of trouble. So I'm curious as you all get your hands on this model, if you have the same experience I did, where it does like really, really, really well and then struggles in the edge cases and the details. So what it nailed here is it did take the spec, it planned the work, it shipped the feature. But then as soon as I got it live and started trying to take it to the next level, the next level, the next level, it really struggled and started to ship bugs. And even more than its inability to finish that last 10%, when it was bug hunting, it hallucinated. And I am going to tell you, I have not seen a straight up hallucination in a very, very, very long time. But over my experience, early testing Opus 4.8, both on business use cases as well as coding use cases, it 100% made up things based on a hypothesis, not data. And this was really interesting to me. This was on high effort. And so I don't think it was effort or reasoning. There's something about this model where it's really not grounding itself as effectively as I've seen in other models. Again, this was a one shot, but then very specifically propped it up on scoped surface areas for follow-ups. Like I saw a bug in the preview branch and got these hallucinations. And so this is a really interesting reflection of this bug. I'm going to have to run at it a little bit more to see if this holds over time with coding use cases, but it was kind of the theme of my test here. Okay, this headline is a little dramatic. It says in real code bases, the edges destroy it. This is not Opus 4.8. This is just Claude Co-Work fail here. It doesn't have the screenshots. We'll have to show you the GitHub for this. But basically what I saw is when I pointed it at existing code, it also struggled to insert itself and understand the edges of where it was supposed to work. So let me give you an example of this. I had a couple of branches in flight that I needed to rebase, that I needed to bring up to base because we had shipped a big underlying PR and it kind of messed up on the state of the code. And so I asked Opus 4.8 to rebase and check these branches for code. And as you can see here, I had to do cycle after cycle of rebase and fixes because it was continuing to ship really edge case bugs into the code. And again, this was my experience. I thought it did a really good job one shot on a surface area, but then when you got into the specifics, it struggled to understand the elevation at which it should be operating. The third coding use case that I tried was a fun one, which is I pulled up Claude Code and asked it, what are some fun things we can one shot with Claude Code that my nine year old would think is rad? And I really tried to push it to say, make it really interesting. Think about the edges of agentic coding. And aside from the code quality itself, which I struggled with, sort of had highs and lows, the other thing I reflected on when I was coding with Opus 4.8 is it just wasn't ambitious enough. And so it gave me this awesome prompt, which was build a game, then play it yourself by watching the screen and tweaking the difficulty until it's fun for a nine-year-old. Amazing. This is state of the art coding agent. It's going to cook. Let me show you what it actually shipped. It shipped this, which is like, fine, of course, magic. Like I would have never been able to ship this by myself without a lot of effort. But not pushing the edges of agentic coding. And even when I said, great, let's make it 3D. Let's do something even more fun. It shipped something like this, which again is super cool. I would have never been able to do this, but it's not 10X agentic coding blow my mind impressive. And so this is where I really struggled with Opus 4.8 is I kept saying more, more, do better, do better. And it just wasn't as ambitious as I've seen other models be. So in terms of coding, I think it is a totally serviceable job. I wouldn't say it's bad at coding. I would just say my experience has been it struggles with the last 10%. It's not exceptional at orienting itself inside existing code bases. And then it's just not that ambitious. Now let's talk about business work. So I also tested Opus 4.8 in Claude Co-Work, and I tested it on strategy. And I gave it this very broad prompt. And I tested 4.7 versus 4.8. And I basically said, based on what you can gather about my last three months, where am I spending my time versus where my priorities should be if I want to 10X my business? I gave it access to all the same business context. And then once it did that analysis, I said, please write me a strategy prompt. And this is where the performance of Opus 4.7 versus Opus 4.8 really became apparent. Opus 4.7 was very numbers anchored. You can see this table here. I obfuscated some of the numbers, but it was very numbers anchored. It was very structured and rooted in real data. While both of these exercises did have access to the same data, Opus 4.8 had a harder time discovering the relevant data, and it over-rotated on small data points and took them as truth as opposed to what I experienced Opus 4.7 doing, which is it zoomed out a lot more and put everything in context. Now, again, this is mutual one shots. This was basically a two shot. It was like, analyze my time and then give me a strategy to grow my business. But the difference between these two were very, very high. I then asked it a follow-up prompt to build a roadmap. And again, 4.7, very anchored and specific. Very good strategy. And 4.8 was incredibly hand-wavy. And in fact, with Opus 4.8, it gave me a roadmap. And then I said, we have all of this. Did you search through GitHub? Did you look online? And what's really funny, again with the hallucination, is you see here, no, I didn't. This is a common thing that