Grok Bot vs Meta Muse vs Instinct: which AI assistant should you actually use?

0:00Um, I, you know, a tweet that I put out, earlier today was like, I, I literally talk to my AI agents more than I talk to my girlfriend now. It's like I have 8 agents always running that I'm testing all them. Um, it since the first week of launching Assistant Benchmark, I have run, I have personally run 273 tests across 23 different agents. Um, I posted a video about Muse, and the by far my most liked video on on YouTube, separately from having a conversation about this on Thursday I, and it seems like we have, um, basically an insane race between assistants. We'll have David Paulan from Assistant Benchmark to join

0:39us, uh, very, very soon. Uh, however, I just want to do like a survey here and in the comments. Which assistant are you guys using right now, if at all, and whether or not you've changed in the past month, let's say. Uh, the options are obviously the standard ones for developers like us, OpenClaw Hermes, uh, and also the, the latest and greatest from Grokbot, uh, Muse from Meta, Instinct that we should cover in a moment, and some others like Town and Val

1:08and like a bunch of other ones. Uh, so let's start with the, with the comments already. Folks are saying Hermes, uh, Wolfram. Hermes definitely. I'm u- I use Codex at work because we are not allowed to use Hermes there yet. But Hermes, I'm so, uh, in- involved in the ecosystem now. I'm one of the contributors to the project. I have, I think we are almost at 40 patches now where I changed that and

1:30submitted PR, so I don't want to switch anymore. It's open source, it runs on my system, I'm locked in in that. It's my agent. So you're Hermes. Uh, Peter, do you have any assistants or using, uh, harnesses as assistants, and what is the difference between them? That's also an interesting question. I must say, I don't know if I'm behind the curve, but I, I sort of tried them out and I never end up using them. I don't know, there's just something about

1:58that I don't find that helpful. I find Let me ask you. it take a bit more work. Let me ask you this in a different way, because I think the distinction there matters. We're going to talk about the distinction. Do you have anything a- agentic-wise that's with intelligence that tells you about the new incoming trigger from elsewhere, like a new email or a new

2:16message from Slack or et cetera. Do you have anything that's like No. proactive in a way that's like serving as your system with your context? No, the be- I do have like a few automations running. I wouldn't kind of quite classify it that way. So I w- I do have a few things where I'm like

2:32looking for new restaurants that opening in London or something like that. Yeah. And it just like it runs every couple of weeks. So it it's kind of semi version of that, but it's very targeted. Yeah. I found, for example, one thing I've got is It's an automation, not a heartbeat. I think that this is like the thing that we're getting to.

2:46Yeah, yeah. It's like not something that constantly runs for you, Yeah. rather it's something that you wanted and then you just didn't wanna Yeah. keep remembering to do. Exactly. Yeah, so, but for the heartbeat stuff, I I don't have anything that I found particularly useful yet. And then, yeah, I don't know. I just prefer to check my own emails, and I prefer to, I don't know, manage my own

3:08calendar. I don't know. That- that's how I feel like I know stuff. I don't- Yeah. I- the- the same way how I write myself and I read things myself. Like, there are a few things that I- I just actually that I feel like that's my job as a human, uh, rather than just like a- a farming this out to an agent.

3:25Uh, But I might How about- change my mind. How about you- you folks? Uh, let's have Yam. Do you have any usage of assistants in your life currently? Codex, Codex, like Do you feel that Codex is answering the, the, the, the be- the, the threshold of assistant versus just a AI agent? Look, look, I have a lot of customizations, uh, around Codex, so it's not out of the box. Uh, but I must say that all the new products, like News and and uh, what was the last one? Oh, and and Grokbot, I want to try them as well, but currently have like a custom,

4:04custom scripted uh, whatever. You you know how it goes, you know how it goes, but mostly Codex, mostly Codex, I think. Uh, all right, let's It's amazing, seriously. guest shows up, uh, Nisten and LDJ, I want to hear from you. Nisten, let's start with you. Any assistants in your life? Yeah, uh, I built the Nisten assistant.

4:27Yeah, that's, that's a great answer, actually. Yeah, yep, yep. Yeah, so I just, uh- Also, it's a great name, the Nisten assistant. No, it's gonna be, I have no, it's just called assist.ts. It's a single TypeScript file because Yeah. I don't trust the dependencies, uh, I don't install any pip. Uh, the only dependency is bun, and it's extremely fast at opening things and opening emails and stuff, but then I just end up delegating a lot of the other tasks

4:57to, simply to to Claude Code. And uh, mainly I keep uh, Astra for reviews, and uh, I I really like the Astra web interface, like for editing videos and making animations and stuff. I just use that from the web. So, uh, So no. Your answer is no. You have your own personal one, Yeah.

5:17and it's a specific thing. Right, let's, uh, LDJ, and then I'll talk, and then we- we'll have a place for our guest as well. I'm- I'm kind of with Peter here, where I've tried a few different things and Grokbot and other things, and I guess it really just depends on what exactly you're doing, and I do end up kind of going in different phases where I'll be, like, working on specific projects that are very different than others and sometimes find them more useful. But, uh, for the most part, uh, using Hermes Agent just kind of as a, uh, like a long-term persistent, like that benefit there, the- the long-term

5:50persistent shot of- of really having a in- depth, uh, context of the past memories. Yeah. Find that useful, but besides that, really just kind of schedule tasks with ChatGPT, find that really useful, not, not anything too crazy beyond that, but I want to start using Muse. Uh, that's a very interesting thing. I want to start using Muse, you answer.

6:09So I'll answer as well, folks. I've been, uh, completely obsessed with transitioning my life into automation with Proactive Assistant. I told you on the pod that I think the 2026 is the year of Proactive Assistant specifically, and the reason I told you about this is, um, that, I think that having assistants that are active, I think is very, very important. And so, I

6:31constantly right now use a Grokbot. I have more than... We can take a look, actually. Oh no, we can't. Uh, I'll tell you why later. Uh, but I use Grokbot constantly. I use Muse since it came out, and I think it's just wonderful and incredible in, in many, many, many ways. And, um, I still have my Hermes.

6:49I barely talk to it anymore, but Hermes has a bunch of context as well. And here are the few... You know what? Let's, let's do, let's do the invitation. David Paulan just joined us. David, welcome to the show. Thank you, thank you. Nice to, nice to have you here. You... have been blowing up on my timeline, and it's very interesting thing where X timeline is specifically tuned to the

7:13interests of the people. Yep. And so on my timeline, you're all over my timeline, and Muse is all over my timeline because I've been like glazing on Muse, uh, Right. On other folks, maybe that they didn't quite have that effect. So, first of all, I, uh, the thing that you built is showing up on my timeline,

7:28which is Assistant Benchmark, which I would love to talk to you about. But also, you joined at the point of the show where I asked everybody from the hosts which assistant they use, and we have a very interesting kind of spectrum of of who uses what. Would love to hear from you before you even introduce us of who you are and what

7:43you build. What is, um, w- w- what is your daily driver? I'm assuming you're using an assistant. Yeah. And then we can talk about the differentiation between an agent and an assistant, and what's what. Love it. Uh, number one assistant I'm using is Instinct for personal stuff, Grokbot for more work-related things. Uh, reason being Grokbot, I, I really love the UI/UX and creating different bots and the organization of it. Uh, Instinct just lives in iMessage, so it's just a little easier for me. It- it's a pure efficiency standpoint compared to using Muse, but it

8:16would- it would be between Instinct and Muse. Instinct and Muse. Interesting. Um, all right, so folks, let's talk about what an assistant is. Uh, and uh, folks, if you don't mind, I'll take you back, but if you have a question or you want to comment, uh, Wolfram, I'm actually gonna keep you here because, uh, everybody else here doesn't necessarily use an assistant, wants to try, so please tune with us. Uh, uh, David, so the reason why you've been blowing up on my time is because I was

8:39overobsessed with Assistant. Muse, I think for the past 3 days, like, most of my tweets are were about Muse, now they're about JEV, but before they were about Muse. I still use it incredibly well. I give feedback to the team. Um, and before that it was Grok, and before that it was Hermes, and before that

8:53it was OpenClaw. So I've been like Assistant pill for a while. Also Instinct, uh, which I, oof, I don't know what's going on there, but Instinct is talking about raising a 10 billion dollar valuation, uh, recently, and this is a startup that just started. So the world is also getting super excited about this next wave of AI that's helpful

9:13for humans. Let's define an assistant between us. Like, I, I want like a, first of all, how do you think about what is the difference between an assistant and just like an AI agent? And then also, um, how that definition goes across labs, for example. Sure, yeah, I mean, I think to me, an assistant is something that actually can execute the task. It's, it's not just, something that pings you or notifies you, uh, if anything, that right, that creates like more work in a surface area where you're just now getting constant notifications, but it's something that

9:41will actually then identify the issue and then complete the task. Uh, maybe there there's some approval in the loop in that capacity, but it's actually going to take that, uh, workflow from start to finish. Interesting. Uh, Wolfram, how do you, how do you feel the difference between an assistant and just an agent or a harness? It's funny because AI, when we started out, it was always a user versus assistant

10:02in the chat model. So that was the official name for the AI in the beginning. It was just the assistant. Then the agents came out and started doing things, and now we are back to having the agent be the assistant because it's actually a helper. That's for me the assistant, it is providing assistance, it's helping me, it's an agent that does something on my behalf and helps me in all,

10:24basically everything. Mhm. My, my feeling is that there is a barrier between giving me more things to look at, like David, like you're saying, versus proactivity of like reducing my cognitive load. Right. I feel like assistants, specifically the new type of assistants with a heartbeat, that they look for stuff before I need to tell them, with

10:46proactivity features built in, with uh, creative ways of using them as well. They teach me how to use them. For example, Muse, we can talk about Muse proactive features as well. I think that that's kind of the difference. Whereas um, the generalized harnesses agents were built as coding harnesses that folks in labs just noticed they're super useful Right.

11:04assistants, uh, but they're missing like a few things. And I think that this is maybe the reason why Peter Steinberger went on such a huge, uh, epic journey in the beginning of this year, talked to all of the labs, and now I think this is also the reason why, uh, Muse has a heartbeat, has a soul.

11:20Right. MD file and an agents and memory.md file. And also, I don't know if you guys know this, uh, Muse has dreaming built in, so you can literally ask Muse about what it dreamt on, and mu- dreaming is like another process that that OpenClaw kind of started and Hermes Right. has. Um, so this is an assistant for me. David, the reason why you're here is not because you're just a random person that I

11:42asked about, like, who uses the system, what is the definition. Uh, you've built Assistant Benchmark, specifically highlighting the stuff that assistants do. Let's talk about Assistant Benchmark. Totally. So, uh, it generally came from, I was, I was testing them all myself, right? Like, same as you, I was just using them all, doing my own market research and figuring out what's working, what's not working, and I needed a way to compare

12:03them all with one another. Um, it it's really more of like a use case driven benchmark, right? Like, it's not like we have 10,000 phone numbers provisioned and we're running all these iterations and, um, like, like a, like a AI arena type type proper. It's really more use case driven. So, you know, you click into a category, you can see the prompt that's being asked. Um, we then will, uh. So let's talk about, uh, let's walk through some of the categories that you are testing. Sure. I think it's gonna be interesting because we're also a podcast,

12:31so n- not many folks are viewing what we're seeing. Yeah, totally. So let's walk through this. Uh, so you have categories like travel assistance, email, and finance, and shopping, uh, and and food. And there's like numbers here, and I would love to tell you about like, I would love for you to tell us what like work in teams has 37, for example, and

12:48travel has 2. What what does this mean? It means that Right. you've tested 2 assistants on this category? Uh, so that means that there's total of 2 assistants have submitted themselves, toward for the benchmark within that specific category. So under the travel category, you have Miso and Saur, and those are, uh, travel specific assistants. Um, if you look at the general category, those are like the Muse, the Instinct,

13:10the, the ones that are just doing your like day-to-day workflows. Mhm. Um, if you look at more of the work in teams, uh, there's 37 that have been submitted there. Those are more on like the B2B side of things. Um, they haven't been tested yet, and so we're, we're gonna be working through that. What's crazy is there's 116 total assistants that have now been submitted on the site. Yeah. Um, I, you know, a tweet that I put out, earlier today was like, I,

13:32I literally talk to my AI agents more than I talk to my girlfriend now. It's like I have 8 agents always running that I'm testing all them. Um, it since the first week of launching Assistant Benchmark, I have run, I have personally run 273 tests across 23 different agents. Um, and it's just the whole intention here is, okay, let's run these same prompts, let's see how these different, uh, assistants

13:58are going to respond to it. Let's also see how fast they are, and, uh, let's just compare them. Let's see what's going on and give, from twofold, it's one, let's give the general public a way to actually understand how these different assistants are performing, what they're good at, what they're not good at. But then two, if you're building an assistant, like, these are the general things that your assistant should be really good at, and if it's not performing well at it, like, you should probably do some more work on your

14:26assistant to make sure that it executes it really well. So the type of stuff, so I was, I was incorrect before. The above there are categories the assistants submit themselves to. The general category has 59 assistants, and the type of tasks that it assists you with are written here. It's like travel, uh, booking travel for you.

14:43Uh, what is pics? Tell us about pics. Right. so these are the dimensions that are being tested. Um, there's, 16 total dimensions. Pix is more of like a recommendation engine. So is it actually like recommending things, uh, Mm. quite efficiently? So, uh, as an example, you know, if you say, hey, find me a vegan restaurant in, you know, in, in my area for the weekend, is it actually gonna find a vegan

15:05restaurant, or is it just gonna find you a vegetarian restaurant? Um, then you have other things like memory, where it's not necessarily just your traditional memory of doesn't know who you are, but also is it really good at, uh, picking up on context? So, uh, as an example, uh, the way that I'll test memory is I'll first have it, I'll first test the, the travel category, and I'll tell it that I'm traveling to a

15:28certain weekend, uh, and to book me a flight. Yeah. And then I'll go through a few other dimensions, and then after I'll test the, uh, the picks, uh, or sorry, the the online task, uh, dimension, which is then asking to, uh, book me a reservation somewhere. And I'll ask it to book me the reservation the same weekend that I asked it to

15:49travel, but in a different city. And then I'll see, does the assistant preemptively prompt me and say, hey, you're supposed to be in Chicago that weekend. Why are you looking to book a res- Yeah. reservation in New York? Like, are you sure? The agents that can do that, it's pretty solid contextual memory, right?

16:04Like, they're they're keeping track of the pace. Yeah. Uh, not all of them do that. So some of them do, some of them don't. And so it's like, how do we test these different assistants and prompt them in specific ways to really understand, like, the breadth and depth of what they can actually do and

16:18what the different gaps are and all that. I think it's a very, very interesting problem that you're trying to tackle. The the reason why is because the type of benchmarks that we talk about on the show, usually for LLM intelligence, they usually start with LLM with like no context, and then all of them get the same exact system prompt, and all of them exactly the same exact task. And most of the time, that task is verifiable with some code or LLM

16:40as a judge on the other end. So you can like test this from yesterday to today, it supposedly doesn't change, and then between different versions of the model, it changes. Right. Here, you're testing the three parts of what makes an AI in my view, which is model, harness, and context. You're testing all three of them at the same time, including context that includes

16:59memory from before. It's very interesting. How do you approach, um, h- have you heard feedback about this? How do you approach this from a perspective of, hey, assistants are also built in to learn incrementally and be better incrementally w- the more you use them, right? Like, that's what Totally. the promise of Grokbot is, like, it better the more you

17:16use it, and and instinct and memory, like, builds up for you. How do you treat this from a perspective of a clean lab, let's test it in a vacuum, versus this is an assistant that's been going with me and it's just better at learning one Right. versus the other? So, first off, we, it's not a lab, right?

17:31It's, and, and make that very clear, like, this is not a lab grade research Yeah. project. this is very use case driven, and, and that's intentional, um, because the, the everyday individual who's looking at this type of benchmark is not the everyday individual that's looking at an, an arena LM. Mm. Right, like the, the, the arenas, the, the proper LLM benchmarks are being looked at by AI nerds, LLM, you know, AI engineers that, that are into the actual technical

17:59components. The assistant benchmark is being looked at by a random Joe Schmo on the street who is like, I don't know what assistant to use, can it book a flight for me? And it's just very like basic things. Yeah. So this benchmark is, is m- meant for testing on a use case basis. Um, some use cases like memory and, you know, how persistent is it in, in context, like there's things that we're not going to be able to test for another 30, 60 days

18:25because, who knows how long the memory actually is able to maintain there. So it's, we're going to continue these tests over time. Um, Autumn Moulder, she, uh, recently was a SVP of engineering at Cohere. Um, she reached out af- like 3 days after I launched it, and she's like, this is awesome. I'm, you know, currently a free agent, just trying to figure out what, touching grass for a little bit, but I, I want to do a new, new project for the

18:48people. Um, so she reached out. She's been helping me out, which has been amazing, and she brings a lot more of that, like, technical depth and knowledge to the project, more from like a researcher lens. Um, and so we're, we're spinning up a, a new iteration of testing, a lot more focused on like the B2B non-iMessage type agents, um, to try to start testing those

19:06with a little bit more, uh, speed, like a little bit more velocity and intensity. But there's so many components to this benchmark at the end of the day, um, it's really a way to see legitimate performance of outcome, right? This isn't about how quick is your model. This isn't about, um, different, you know, AI engineering technicality. It's about if I'm your everyday person, I'm a 29-year-old guy in New York who's flying to Chicago, if I message the, my assistant, hey, find me a flight to Chicago that's under 400 dollars in a window seat,

19:45can it do it, yes or no? And that's really the baseline here of what we're comparing. That's great. Um, the thing that I would like from a benchmark like this, I think is number one. We talked about pacing the frontier before and all of the labs bringing on METR, for example, and having Yeah. their independent. The thing that, uh, would make me trust, and this is the benchmark, besides the fact that everybody's quoting you because everybody kind of needs this for

20:07themselves, Mhm. uh, is independence, full independence from the labs. I think it's very, very important. So I'll just Yeah. uh, if any lab is paying for some of this inference, I think this needs to be like clear as well if somebody's providing like an extra tier so they'd be able to test them. Totally.

20:20I think that's fair because either way, how would you test them? and also I think the very interesting thing that folks have been asking you is that, the bigger ones that started this whole category, OpenClaw and Hermes are not on this. Would you want to address this on air? Yeah, so I'll touch on two points, uh, like both of those.

20:33So first, there's no one sponsoring this, right? So I lead growth at Merit Systems, this is under my role at Merit, we're just, we live in this world of agentic commerce, and so this started as just a research project for me to really understand the landscape, to figure out, like, where can we really play in this space from an infrastructure layer?

20:49Um, so we're not being sponsored, we're not being paid. Um, any of the competitors that are out there are not able to pay us for any type of promotion or anything like that. Yeah. Um, there might be a day where we say, okay, we need to work with someone like CoreWeave and get more inference, or

21:07whoever it be. Let's talk about this after the show. Uh, totally. Um, and and we've actually recently floated the idea of like having a sponsors page, Mhm. uh, to give us a pathway to test things with, with a little more rigor, um, obviously because that that's going to cost a lot of money. Yeah. Um, so if any of anything like that does happen, there's going to be a very clear page of like, here is who is, who is paying, and here is what they are paying

21:31for, and it is specifically for the research outcome. There is, there's no profit coming from this site in any capacity. Um, in terms of OpenClaw and Hermes, so the original reason that they were not included, uh, is because the performance of them is significantly dependent on the setup from the individual. So when you're thinking about testing these and scaling it across all of them, if I run the same prompt of, book me a flight to Chicago, blah, blah, blah, um, my OpenClaw setup might do it unbelievably well, but your OpenClaw setup might fail. And this then doesn't prove as a valid use case point because if then someone says,

22:13oh great, OpenClaw is really good at this, and then they start and install and build their own OpenClaw setup, and then it fails, well, now they're going to look at the benchmark, and they're going to say, well, what the hell, it said it was really good at it. Um, so the intention here is, like, we're really focusing on out-of-the-box solutions, consumer products that are pre-built packages that you can just use, and everyone who uses them, everyone who installs it or, uh, opens it up in iMessage, whatever it is, every single person is having the exact

22:39same experience. Yep, that makes sense. And, uh, I think the world has shifted from the folks who are building their own assistant in, in their own, like, Mac Mini that they bought Right. towards the one that breaks less than the previous one, Right. towards now these labs are maintaining the machine, Right. which is, I think, a very important thing. We didn't talk about it. Uh, all of these assistants need, like, a computer

23:01environment. All these assistants need a browser with computer use. All these Yep. need access to your memories and connectors for your context. I think those also the things that signify an assistant. David, I did promise you to let you go once you, once you can, because I, you have elsewhere to be. Thank you so much for coming.

23:15Of course. And do Thank you for having me. It's very important. we are definitely expecting you to come back when the new assistant comes out or new capabilities Please. come out. Like new capabilities came out this week, for both Instinct and Muse, where they can actually phone businesses, which I wanted to talk to you about, but

23:28we ran out of time. David, thank you so much. We'll bring you on. David Poland, uh, creator of Assistant Bench. Thank you. Appreciate it. Thanks, everyone.