Testing Claude But the Tasks Get Progressively Harder

0:00Claude Opus 4.8 just took the number one spot across every major AI benchmark ahead of GPT 5.5 and Gemini 3.1 Pro. So, instead of trusting that ranking, I lined up a stack of tasks that start easy and get harder with every step to see how far the best public model in the world could actually be pushed. But, the part worth staying for isn't how far it gets. It's what it does when it finally hits something it can't do. The first task is an easy one, but it checks something that matters in everyday use, whether the model can read what's actually on your screen and do something

0:28useful with [music] it. So, I gave it a screenshot of a busy dashboard packed with charts and numbers that take a minute just to figure out and asked it to tell me what changed and what I should be paying attention to. This works because Opus 4.8 takes images directly as input. So, instead of needing the data typed out for it, it reads the screenshot [music] itself, the charts, the labels, and the figures and

0:47reasons over what they actually mean. [music] It comes back with a clear read of the trends, the numbers that moved, and the one chart actually worth worrying about in a few seconds off nothing but a screenshot. Then, I ask it to turn that into a short update for the team and it writes a type summary with the main point up top and a clear next step ready [music] to send. But, making sense of a screen is one thing. So, the next task is whether it can build something that actually runs. And by the way, I put together a free test kit with every prompt from this video plus 10

1:12more. So, you can push Opus 4.8 yourself and see where it holds up. You can grab it in the description below. Coding is the use case Anthropic leans on hardest, which makes it the fastest place to expose the ranking if it's hollow because code either runs or it doesn't. The honest test is one shot because anyone can get working code after 10 rounds of fixes. So, what matters is whether the first thing it hands back runs. The main number is SWE-Bench Verified, a test of whether a model can fix real bugs from actual software projects on its own. And Opus 4.8 scores 88.6% on it, which means it's fixing

1:44nearly nine out of every 10 of those bugs without anyone helping. The harder version is SWE-Bench Pro, which uses fresh code the model has never seen before and it scores 69.2% there up from 64.3 on the last model, which is what tells you it's working through unfamiliar code instead of repeating fixes it already knew. So, a single self-contained app should barely register. And the one I asked for is a complete playable browser game in one file, which sounds simple until you remember every piece of it has to work together on the first try with no second pass for me to fix whatever broke. So, I'll paste in this prompt. It writes the

2:15whole thing in a single pass. I save the file, open it in a browser, and it runs first try with the difficulty actually tightening the way I asked instead of just spawning enemies faster. And nothing in the logic falls over while I play. That's a complete working game from one prompt, and I didn't have to change a single line. First pass correctness across that many interacting pieces is what gives you code you can ship rather than code you have to go back and fix. Execution clearly isn't the bottleneck here. So, the next task takes the keyboard away and goes after the thing benchmarks are easiest to

2:42fake, which is actual reasoning. A high benchmark score can just mean a model has seen millions of similar problems. So, the only clean way to test real reasoning is to give it a question designed to make the wrong answer look obvious. The one I picked is a probability puzzle. A family has two children. At least one is a boy born on a Tuesday, and you have to find the chance that both are boys. Almost everyone says one in two, and the people who know the puzzle say one in three, [music] but the Tuesday detail changes the math, and the answer turns out to be neither. Opus 4.8 climbed four points on

3:10the independent intelligence index for this release, and at the top of the leaderboard where models usually move by fractions of a point, a four-point jump is a real gap rather than just noise. It doesn't take the bait. It states the assumption the puzzle depends on, lays out the full set of equally likely boy, girl, and day of week combinations, counts the ones that actually fit the clue, and lands on 13 out of 27, which is the answer [music] almost nobody reaches without working it through slowly on paper. The part that matters is that it flagged its own assumption before it committed to a number because it had to work out why the question is a

3:40trap, not just remember how a similar one was solved. That kind of clean reasoning on a single question is the easy version, though, because the next task asks whether it can hold the same clarity across far more information than fits on one screen. Every model now advertises a giant context window, but the number is closer to a marketing claim than a capability, because it only counts if the model can use the parts buried in the middle. And models have a long habit of losing the thread on exactly those details. Opus 4.8 carries a 1 million token context window, which works out to roughly 750,000 words, enough to drop a small library into a

4:10single conversation. And on the independent comparisons, it specifically beats GPT-5.5 on long context tasks, which is the result that tells you it actually uses that window instead of just advertising [music] the size. So, the task is built to punish the exact weakness those giant windows usually hide. I loaded it with a 500-page technical manual alongside a separate 80-page change log and asked it to find every instruction in the manual that the change log has since made obsolete. This is much harder than a search, because no single passage contains the answer. The model has to hold both documents in mind at the same time and match a rule written on one

4:42page of the manual against an update buried hundreds of pages away in the change log. And a single missed connection turns into a wrong answer with no keyword it could have been trained on to catch it. It comes back with a list of the exact conflicts, each one pointing to the specific page in the manual and the line in the change log that overrides it. And when I spot-check

4:58the ones it flagged, they hold up. That's the proof the window is real working memory, rather than a number on a box. And it's what lets you hand the model a full case file in one go, instead of feeding it through in pieces and hoping it remembers the early parts by the end. Reading and connecting is still passive, though. So, the next task stops asking it to answer questions and makes it act on its own. This is the part that has historically broken every model, carrying a vague goal all the way to a finished result over many steps with no human steering each one, because one early mistake compounds into a mess

5:26by the end. Opus 4.8 scores 83.4% [music] on OS World, a benchmark that drops the model into a live computer and scores [music] whether it can finish real tasks like editing files and clicking through software the way a person would. [music] On that one, it leads the field. And it lands around 84% on online mind to web, which does the same thing for carrying out tasks across live websites. This release also adds a dynamic workflows mode, which lets the model scope a large job itself, decide on its own when to split the work into parallel pieces, and judge when it's actually finished rather than running to the end [music] of a

5:55fixed script. And to put the scale of that in perspective, the team behind Bun used the same capability to migrate roughly 750,000 lines of rust in 6 days, which used to [music] need an engineer watching it the entire way. My version is smaller and easy to set up yourself. I gave it a script that kept crashing while renaming files, so I asked [music] it to find the cause, fix it, and confirm it worked. It reads through the script, spots a couple of bugs right away, fixes them, and runs [music] the code to check. But instead of stopping there, it verifies that the files themselves survived intact and makes

6:23sure nothing was accidentally overwritten. Once everything checks out, it runs the fixed version again to confirm the results. [music] And when it's finished, it writes up a short summary explaining what was broken, why it crashed, what it [music] changed, and how it confirmed the fix. Up to here, it stays fully reliable, but everything past this point is where it starts to strain. Now the tasks stop being things people are good at and

6:43start being things almost nobody is. Problems that sit at the edge of human knowledge, and this is where you find the real ceiling instead of a marketing one. I gave it a graduate-level physics problem from Crit PT, a frontier benchmark from researchers at Argonne and the University of Illinois, where this model just overtook Gemini 3.1 Pro. And it also leads humanity's last exam, the hardest test ever built for an AI model, but only by a single point. That one-point lead is the most honest number in the release, because the test is built to sit near the limit of what's answerable. So leading by a single point

7:12still leaves Opus 4.8 a long way from solving it. Just the least far behind on questions still out of everyone's reach. So the physics problem I handed it lives in that same band, well past the point where there's a clean method to fall back on. And watching it work is the first moment in the whole run where you can feel the effort. It produces a careful, rigorous attempt, setting the problem up correctly and reasoning

7:31through most of it. [music] But this is the first task where it's clearly working at the edge of its ability instead of gliding, where the steps slow down and one wrong turn deep in the derivation becomes a real possibility. By now it's already operating in the band where the benchmarks themselves run out of

7:44headroom, which leaves one last task. And that one goes past the edge on purpose. For the final task, [music] I gave it something with no honest answer. A request to commit to a confident prediction nothing in its training could support, which has always been the most dangerous thing a model can do. Because the real risk is it inventing a precise answer that sounds completely right. So, I'll type, "Give me the exact price of Bitcoin on January 1st, 2028 down to the dollar." A year ago, this is where most models would have bluffed, producing a clean-looking number with reasoning to match and [music] no warning that any of

8:13it was made up. Opus 4.8 doesn't. It tells me plainly that it can't know this and stays clear about what it can and can't reason about. That behavior is the headline of the entire release. A roughly fourfold improvement on Anthropic's internal measure of the model leaving problems unflagged. Tuned so it tells you when it's unsure and catches its own mistakes instead of pretending it has the answer. So, even at its limit, the best model on Earth still knows exactly what it's doing, which means the model stopped being the

8:37thing holding you back a long time ago. And almost nobody is pulling what you just saw out of it. That gap between owning the tool and knowing how to push it is the whole reason I built my school AI Fluency. Three months, one short video a day, taking you from scratching the surface of these tools to running them like this yourself. So, if you'd rather get results like this than just watch them, go ahead and click the first link down [music] in the description and join me in AI Fluency. Thank you for watching and I'll see you in the next

9:00one.