We Stressed-Tested CLAUDE Fable 5. Here's What We Found

0:00Ooh, Anthropic just released Babel 5.0. Everyone's talking about whether it's good or not. Well, you know what? Here's the big question. Can it actually get work done? Can it actually use multiple tools to get the outcome that you expect? And I know man who's been testing this hundreds of times. In fact, he's tested every one of these major models hundreds of times [music] and will tell us how it scores against all those other models. Let's get right into it. What do you think [snorts]

0:24overall? This is a strong model. So, at Zapier, we have a benchmark automation bench. We specifically like to test how good these models are at evaluating an AI agent on realistic business workflows. This is what Zapier has tons of data on. We've been operating for 15 years. We know how um these workflows work in the real world.

0:49What's a workflow that you all test? Like a good example would be something like lead routing. In any company, you've got leads that are coming in. They're often coming in from a bunch of different sources. Like maybe some come from your website. Some come from event you run. Maybe someone comes from the CEO's inbox. Maybe someone landed in LinkedIn inbox. So, you've got all these leads that are coming in. And you often need to get them into a CRM. You need to get them scored. You need to get them triaged and sent to your reps. Maybe you want to uh send it to the next available

1:17rep. Maybe you want to send it to a rep based on a certain skill set that they have. And so, we're trying to understand, you know, if you want to put uh an a model inside one of those workflows, how good is it going to be at helping you with various components or the entire task itself?

1:33Okay. Where it's especially good is operations. What did it score? Yes. Okay. So, here's the kicker. It Inside automation bench, we test it on domains: sales, marketing, operations, support, finance. The headliner is it is crushing on operations-specific tasks. So, the prior best model for operations-specific task was Gemini 3.5 flash and it scored at 20%. This new model from Anthropic Fable is scoring at 27% a full seven percentage points improvement over the prior model high and that is better than any other domain either no other domain even touches 20% HR is the closest and which comes in at exactly 20% and that means wait When I give it a 100 tasks 27 of them

2:26are going to get it right. On this model previous models Gemini flash 20% meaning one out of five it got it exactly right. Let's talk about a specific example. So Let's actually talk about lead routing like we were just mentioning that I think that is a good example. Other models will often fail in a particular way. They'll hand off the a lead that

2:46didn't quite qualify. It might email the wrong rep and say hey you know, you're giving it to the wrong person. Whereas Fable is going to more consistently qualify those leads correctly, route them correctly etc. over time. So that's an example of like a type of task that is like an up a very operation specific task and Fable

3:05will do a much better job with it. What's another example? Other pattern that we find is that it tends to do well at like what we call experimentation and recovery. So you give it like a task that is um You you may not tell it how to do the job and it's able to sort of figure it out. So let's give you an like an

3:24example might be Um here's one from HR. You've got to go harmonize like the employee benefits across countries. You got to pull the benefits plans. You got to reconcile them against the policies. Then you got to go notify all the relevant teams. Um now in this task say one of the systems let's say the HR systems benefits plan has an API and

3:45when you try and go hit it it 404s. Manage is down. Can't find it. It's not find it. Yeah. But the data is available elsewhere. Like data is found maybe in a spreadsheet or an inbox or something like that. Okay. So, Fable how it does on this task, it will go hit that API endpoint once, it gets the 404, and it immediately says, "I'm going to go try a different way. I'll go check the spreadsheet. I'll go check the

4:09inbox." And go find the data elsewhere. Now, other models, it doesn't do They don't do this. They'll keep hammering that 404 endpoint trying to see if they'll get a different result. Uh what's that old saying like the the definition of insanity is trying the same thing over and over again uh expecting a different result. We call it experiment, recover. Experiment, recover. Try a thing. Does it work? Yes,

4:33great. Does it not work? Okay, recover. Try something else. Uh and it just does that loop better where these other models will get stuck. The specific example you gave me before we started was GPT-55 would hit the site, get 404, then do it again, do it again, do it again a total of 22 times would get the error and just keep

4:49trying. Gemini did better. It hit the site, got a failure, stopped, got I mean or continued again, got a failure, continued again, did it a total of five times before saying, "Okay, I will find another way." And then it went and found another way. So, we're looking here at a system that really knows how when there's an error, when there's a roadblock, how to recover from it. Um how about one last thing that it does

5:12especially well? It operates better in like messy environments. So, let's consider a Slack instance. In most companies, there's a channel for sales, a channel for marketing, a channel for engineering, channel for all sorts of different stuff. And the channels don't always stay on topic. Uh and you know, not everyone uses threads the right way, not everyone does this the same way, et cetera. Maybe you want

5:33it to go flag any off-track ad budgets. And you could say, "Hey, all the ad budgets are listed in this alerts channel. We have this alerts channel set up, and I want to specifically know about anyone that is like flagged for like being over budget, way under budget, et cetera, et cetera, et cetera." Now, this, you know, Fable, again, will do much better at this task. It will go track the very specific ones that are hitting those alerts, summarize those, and report them back to you. The failure pattern that we see a lot of times with this is that it won't stay on task. Uh it will, you know,

6:07prior models will find maybe it maybe it'll find all the ones that do qualify, but it'll go wander off and it'll include other contexts. It may include other things. Maybe it pulls an unrelated task that was inside that same channel. Maybe it goes and looks at a different channel. Maybe it looks in a different thread, and it includes that in the response. And we probably all experienced this before, where we see a response from a model, and you go, "What is this? Like, what are you talking about here? This doesn't relate to my

6:32query at all." Okay, so overall, it's more precise, it stays on track, uh it's more resourceful, it recovers from its own failures quicker, um compared to these other models. Uh yeah, I still think there's an open ended question for this type of use case. Do you want to use a model, or do you want to use code or deterministic workflows for this type of task? I think that's something you should be really

6:54asking yourself. But, there are definitely operations use cases where you're going to want to use an LLM, and this is a good one to use for that. Wait, by the way, do you think this is Is this a version of Mythos? Is this Mythos light? I This So, my understanding is this is a Mythos-style model. Uh I'm sure Anthropic is going to share has a lot more details out there soon, but this is the first available version of Mythos um that's generally

7:20available.