Best & cheapest AI for OpenClaw

0:00We're going to have the top AI models battle it out to see which one is the best for open claw and how you can figure out which one will get you the best results at the right price. Let's get into it. Presented by Zapier, the AI automation company. I, like many Claude users, got a very kind email from Anthropic talking about how they were going to turn off subscriptions in open claw. Kind? This was like a slap in the face before a

0:26holiday saying, "Hey, you know what? You've been using your subscription to use open claw? Bam, no more. Get out of here." Yeah, it was not a it was not a great way to to wrap up my work week. What would you estimate, considering that you use open claw so much, if you were to continue using Claude the way you ordinarily were, instead of paying the 200 bucks a month using API fees, what would you estimate you'd have to pay? I did a little test yesterday of just using the API and it was about total ended up being about $50. So, $50

0:57a day. Yeah. And you were being very conservative. I saw your test. You were you're essentially saying instead of using the best model, which is what you were doing before, you were saying, "I want the best model to kind of be a router and decide what does which model we'll use for which one of my tasks." And still with that conservative approach, you'd end up being $1,500 a month as opposed

1:14to 200 bucks a month. Correct. And that's that doesn't that's no that's no coding, right? So, like if I was doing a bunch of coding, this you know, this is just me interacting, sort of chatting things out with them with Right, it was a light day from what you saw. So, you said, "Okay, this can't stand, but there are other models out there. I want to figure out how to do it." And instead of just picking with your heart, you said, "I'm going to be a

1:33little bit more scientific about it." And you started to look around. Let's see what you did. Show me that Discord. That was magic. Yeah. Yep. So, I set up this Discord channel here where I decided to to run through a bunch of different models. Um and this is I I'd asked around trying to figure out like, "Well, hey, what what are some decent alternatives? I haven't stayed really in the loop on open source models or other alternative models." Uh so, I got a bunch of input and decided, "Well, I'll just try them all out." Um Okay. So, And each one of those on the left is the exact same open claw

2:10person or agent, Hank. Yeah. Except and you're asking the same questions and you're seeing how they do. Which one do you want to start with and show me how they did it? Yep, so let's start with what I wanted to keep, right? Which is Opus. This is what I I love. Yeah, let's see how Opus did. Yeah, so the way that I set this up was um having having it do a task that I would normally do, right? This is not just me like asking it some random question to benchmark it. So, first example here, so this is for the toy company, this wholesale playbook that exists in Notion. I I don't mention

2:45that there's a Notion document. I just ask it about this playbook that Okay. Let me read it. You said you intentionally said, "What's my next What's my next What's my next task to do for super fantastic wholesale book playbook?" And you didn't say it's in Notion. And look at that, Hank automatically comes back in and says, "Here's a full wholesale execution playbook in Notion. Let me pull it to

3:06see exactly where you left off." Smart. This is what you're looking for, right? This is exactly what I want, right? Like I didn't want to have to hold its hand, tell it to go look in a specific place. It should just know where it is. It did that, made sense of it, and you know, sort of laid out the pending items and

3:21then what it thinks I should do next. Okay. And then offers to help knock it out. That's kind. So, that's this is like my baseline that I This is what I want all the others to respond like I'm trying to gauge how close they are to what I consider sort of a perfect response. Okay. Um so, then I jumped into Sonnet. So, this is sort of a downgrade from Opus, but less expensive. Um And it it couldn't find it. It it choked. So, uh didn't know about Notion. Wow. Yeah, didn't did not even attempt to find it there. Just so we're clear, how would it know about Notion? What's the way that

3:59you think that open claw should know that you're using Notion? in in the sort of higher level memory, there's like a tools.md file that uh mentions all the different tools I use, like Notion. So, So, bummer. Sonnet did not work as well as I would have expected. Sonnet's 46. Let's look at the next one. So, then Gemma, same thing. Free model that Google says

4:24we can self-host. What's my next task? And it said, "Let me check recent memory. Let me check the daily memory. Hmm, I'm not seeing any prior work on wholesale playbook in my memory." But it should have it, right? It should have it. There's no reason it doesn't. Kimmy's another one. This one's one that

4:41sort of made a splash. Yeah. Is this the one that people have been self-hosting on their Macs? Nvidia has like they made a few like a month or two ago that it was available for free on Nvidia. So, like a lot of people kind of jumped in on it. Mhm. This one yeah, it it kind of just glitched a bit and started like it it almost like it acted like it was picking up from a different part of Discord. Like I don't even know what

5:08this was talking about. user's in a different thread now, Kimmy K 2.5, which is a thread off of models channel. You're asking about the next task for super fantastic wholesale playbook. This is totally different topic from the model LK. Let me check my memory. I should search for super fantastic wholesale playbook. Actually, I should acknowledge the context switch and search for relevant information. Let me search for this in memory and also check I shouldn't look

5:36for any existing documentation files. Let me check. And it never came back with anything. Correct. And then it like And then it comes back later and does the good afternoon, what can I do for you? What's on your mind today? Yeah, so then I was like, "Well, maybe like I don't know where it's getting the like it's like talking out its thoughts,

5:53which was weird." Which I didn't care for. GPT 5.4 mini, so this was sort of an interesting one. A lot of people have said that mini has more personality than just like the core 5.4, but it also has less, you know, it's smaller I think context or less powerful model. Um But let's see. So, this one it

6:16it found it. It just gave no context about like what like I don't it didn't mention anything about Notion. Like uh Oh, the next task is to build the actual wholesale list. Okay. But you did ask what my next task is. The fact that it didn't say that it's in in Notion is not a problem. It's not necessarily a This is what So, this is gets to the um the like the vibe side of it, which I think a lot of people write off. Like

6:41so, it's it's technically correct, Mhm. but it's like so cold and like clinical that um that it almost just like it makes me sort of not trust it. Like there's a certain level of it doesn't give enough context around how it got its response. Mhm. Um and so, while technically correct, don't love the output. But again, that's the

7:06subjective part. you're saying is, "Look, it gave me exactly what I was looking for, but it didn't intuit that what I wanted was context and not just the one next thing, but more of it." Okay. All right, I get why you don't love it, but I also see why it's not wrong. Okay, it works. So, and I think this might have been the same Yeah, same case uh with 5.4. Um you know, again, technically correct, but um just too too clinical. It says your next task is email Is it a link to the thing? No, it's just saying email the six repeat wholesale buyers with their And is this

7:44actually the next task? Is it right? Yeah, that's correct. That's the funnest That's money because it keeps the relationship, preserves reorders, and drops fares 15% cut to 0%. Okay, so far again, the same thing as as the previous model. Right. So, again, whereas Kimmy, Gemma, and Sonnet could not even do the thing. GPT got it correct. I just didn't care for the output. Okay, and let me also say this, that Peter Steinberger has told me that OpenAI is happy for us to use our subscription. And so, so far, so

8:17good. Correct. Yeah, so if like I'm if I'm wanting this as a potential replacement for for Claude subscription, right? Where I get a ton of use for a rel you know, a flat fee essentially, um it works. Don't love it, though. Okay. So, it's like, "Well, let me just keep testing." Um GLM 5. So, this is So, 5.1 is like the newest thing, but GLM 5 has been

8:42something people have suggested before. GLM 5 did relatively bad. Um it just it just starts dumping out a ton of stuff. Let me check the memory for context on super fantastic's wholesale playbook. No results. Let me search more broadly for super fantastic wholesale. Still nothing, etc. etc. This memory doesn't have wholesale's playbook details. Let me check memory.md file.

9:05And it's not in the memory files, huh? Not what I'm not this wholesale playbook thing, right? It like it It didn't intuit that there's this like means it should know to check Notion again. All right. cuz it's a document essentially. It's like it doesn't recognize that it's a document, right? It's like it's being too literal and so, um yeah, so it just starts like spitting out all the ways that it's trying to find stuff, which I appreciate that I guess I'm seeing its process of like all the things it's checking, but it still resulted in saying like, "I I got nothing." Okay. Fair. So, didn't love

9:40GLM. Not fair, but okay. Yeah. Quinn was another one that people suggested. Man, by the way, I've been watching you on X and I thought that that GLM 5 was working for you. No, not 5, 5.1. 5.1 was working for you. Okay. All right. So, we'll keep going with Quinn. I gave away the answer. But okay, let's let's look at Quinn. Quinn I see this line sheet document. Let me check. All right, it's found that its notion integration exists. And then it says on March 23rd, yeah, you walk me

10:14through it. It's seeing there. Right. So, it basically is finding what what I think this is has essentially done is it's searching sessions with Open Claw. So, it realizes that oh, we did this whole thing and we saved it to notion. Like it sees that from the sessions with Open Claw. But it doesn't make the connection that

10:38there is that it has notion access. Right? Like it it basically is like, "Okay, like I see that we did all this stuff and that you saved to notion, but like I don't know I don't have I don't have notion access." Like it does. It just assumes it doesn't. Yeah, that's actually a very basic thing it should know. It knows what tools it has access to. It should. And it's like it made the connection that something exists there, but it it never was like, "Oh, let me go see if I could do it." So, it's this is one of those where like I could

11:06essentially then say, "Go check notion." And it would be like, "Oh, okay." And it would do it. So, that's one of those where it it needed it needs more hand-holding. Which again, that's a trade-off, right? Very cheap model, but it needs more hand-holding. So, I I didn't mind the output necessarily. I just don't want to have to like keep

11:27being like, "Yeah, well now go do this. Now go do that." Yeah. Um so, then I had after I'd had no luck with GLM 5, people were suggesting 5.1. I was like, "What's the difference?" Like you know, but [laughter] I mean I also think the difference like GPT 5.3 versus 5.4 is a big upgrade, you know,

11:49Opus 4.5 versus 4.6 like big step up. So, yeah, I think that ends up being the case with 5.1. My testing I think on this one how did I end up? Um I did not run it. I think I would have to go find it in another thread. Like I think I I end up trying it without saying notion and it worked fine and I was just checking some

12:15other stuff. So, because I one of my other threads I ended up finding that it it did properly pull the whole sale playbook. It did properly do the notion connection. You know, would you ask it right now so that we have the exact same thing and can compare it? Let's make a it's I can't do it in this thread cuz it will have that context. Okay, then actually then maybe this is a good opportunity for us to see how you do this from from scratch. Okay. Yeah, so what we

12:45let me open. So, in Open Router, you know, there's just you can there's an infinite number of models to choose from. So, like if you're wanting to set this up for yourself, you would say that there's So, in this particular case, GLM this is why it was another step. Like you can't access 5.1 through Open Router. You have to set it up through z.ai for the for the newest 5.1. Now, you can set up like GLM 5 is through Open Router. There's also this turbo and the V turbo, which is like has the a vision part of the model. These are worth

13:23testing if you're interested. But z.ai direct is the only way to do test out 5.1, which is what I ended up going with. So, you went in and you signed up for a subscription there. You created a brand new account and then you put a little bit of money in like 10, 20 bucks

13:39into Open Router. And then you find the model that you want, which again you just search for to what you need is this model name. So, you can just copy that. And then uh to set up a new model in Open Claw, you do Open Claw models set. And then what it's saying like it's already suggesting here, but you would say Open

14:04Router and then the name of the model. So, in this case you pasted what you got. Yeah, so but you have to prepend it with Open Router so that it knows that it's coming from Open Router. So, you do that. It'll come back here and say that it's updated Open Claw and it's it's actually it changes the default. But you you can then go and tell it to like undo essentially so that you're not always using that as the default. But so then it becomes available to

14:34Open Claw. Yeah, so I'll show you what that looks like here. We'll we'll call this just for

14:52Okay. So, what you did was you created a new channel again conversation with your same Open Claw agent. So, then you would do models. So, in this is a Discord sort of specific thing. But you you say like basically let me pick a new model. And you would go to Open Router and then you see everything that like I've added here and then you would pick the model

15:20that you want to use. But we're testing the 5.1, which I've set up. So, in this case you've got to go directly to z.ai for and now you get 5.1. Yeah, which it's already sent to you cuz that's my that's my default. But yeah. So, let's ask the same question now and see what it says. Hold on, let me go find it so I can copy

15:41paste the exact. And I should say the thing that people on X were saying in response to your post was that the price for z.ai is really it's really low. We're talking about monthly for their max 80 bucks. Yeah, it's not bad. I'm on the $30 and I have in the past few days, let's see.

15:59Can you see this little tool tip? Uh Oh yeah, I do. I see it. Oh, I see. Okay, yeah, yeah. This is cool. So, I have this my z.ai usage and I've spent a good bit of time with it over the past like 3 days and have used 14% of my total weekly. So, I there are like people are like,

16:19"Oh, it's the the limits are terrible." Like I it's has not been I've had no trouble with $30 a month. That's a lot. Yeah, and like I would happily pay 80 bucks a month if I needed to. Like that's still very inexpensive for what you get. Either way. Whoops. Okay, so we'll see. And now you're asking the same question. What's my next ask to do for Super Fantastic Wholesale Book uh Playbooks? It's probably going it's probably going to make me look bad, but

16:50we'll see. What is Super Fantastic? Uh so, it's a it's a toy company that I and I produce all the toys. So, yeah. How do you use Open Claw in the toy company? Yeah, I do a lot of market research stuff. A lot of inventory management. So, and like store management. So, this is like syncing prices between Shopify, Etsy, TikTok, all these places. Like I have I have skills or commands set in place to basically like sync different API connections to all of them so that I can basically run everything from here in Discord. So, if you reduce the price on one place, you can tell

17:35your Open Claw agent, "Go reduce it everywhere else." Raise it everywhere else. Got it. It's stuff like that that it gets to do. That is a really good use case. Or like I need to write a product description. So, I'll like write the product description as a base and then I have it go do like keyword research for different platforms cuz there's different keyword usage for Etsy versus Shopify versus TikTok. Yeah. And so, I'll have it go do keyword research to rewrite those descriptions for those

17:59platforms and then sync them. How else are you using Open Claw while we're waiting for answer? Yeah. A lot of project So, these are all these are different projects I've got going on. So, I'll again, a lot of it's like research stuff. So, I really that's what I use Open Claw more than anything. I I really don't do any coding with it is research like research or sort of business admin kind of things. Like I'll do QuickBooks stuff through here. So, once a month is

18:33set up. Uh-huh. So, I've I've got it connected to my QuickBooks for for business stuff and for like categorizing transact like bookkeeping essentially. So, it'll show you the cat the things that it wasn't able to automatically categorize and categorize it for you or ask you for how you want to categorize it. Yep. Ooh. I said it was probably going to make me

18:54look bad. Don't have a documented wholesale playbook and task lists anywhere. Same question came up on the 4th and I couldn't find one either. Very frustrating that the same thing wasn't working here. It did work before. This is maddening. Freaking A. This is like man like Open Claw does this to me every day. It was like, "Yeah, like I'm like pumped hey GLM 5.1 is like the answer." And then here we are. And it's like basic so

19:21inconsistent. I don't get it. Yeah. So, this is like why I I I don't have any of these consistency issues with Opus. It's just Okay, so here's what we've learned. If I'm to take a look at your rating, Opus 4.6 number one, it just gets things right. Then you go into the Open AI two options that you tested. And it looks like they're both accurate. They just are not filling in the gaps beyond

19:51what you are specifically asking. They're not intuiting the next, and the rest beyond that, they're okay, but they've made so many mistakes in not figuring things out that it really is like going down a few ages when you're talking to your kids. When you talk to an 11-year-old, you can assume that they understand what you mean by go online, go to the store, etc. When you're going to a 5-year-old, you have to be much more explicit. That's where you are right now in this discovery. And everyone else can go and check this out for themselves, but the bottom line, Josh, is for you

20:20it really is Opus 46 all the way. Everything else is just for more basic tasks, and for everyone who's listening, here's a good way to test it and see what works in your in your world, in your real use cases. Hey, my agent says that if you watch this far, you're going to want to subscribe, and Google thinks that if you watch this far, you'll want

20:39to watch that video.