# SpaceX Engineer, Lauren Tan (Masterclass)

Channel: Get into AI
Video: https://www.youtube.com/watch?v=kGXnNfmvhv4
Duration: 3 hr 44 min
Language: English
Words: 43139
Transcript page: https://viewrankai.com/tools/youtube-transcript/kGXnNfmvhv4

---

[0:00] Um, personally, I think that uh the best use of you of AI is really as like a like a pair like someone you're pair programming with and not someone as not not a not a tool to just replace the act of writing code or worse outsourcing your thinking. And I think there's a tendency like, you know, because AI is so exciting, you want to put AI everywhere and be seen as someone who's very, you know, um what do you call it? Like uh on the

[0:28] ball, I guess, with with AI. Uh that you feel this pressure of like, you know, I'm just going to I got to increase my productivity. I got to ship like 100 PRs this week and barely understand what I'm doing and, you know, vibe code my way to a a million ARR or whatever. Um, but I think, you know, um one of the things that I've personally seen is that although when I use AI, I save some time, you know, writing the code, I actually find myself spending more time reviewing what the AI did and like correcting it. So, you know, like I've I've tried the the thing that a lot

[1:04] of people say like, you know, you uh instead of prompting an AI to just build the feature outright, you sort of get it to uh build a spec for you first, then you review that spec, and then you do that whole thing. Then it's like Okay, so I assume you can see my screen.

[1:18] Yes, yeah, we're good. So, yeah, today Yeah, I think I think the big theme for me as I've been using agents to write code, and I'm sure a lot of you have had the same experience as well, is how do you trust it? You know, especially if you are an engineer that's been writing code for a very long time, you have a lot of opinions and lessons that you've learned about doing good

[1:40] engineering. And when you see agents just, you know, winging it and, you know, guessing, hallucinating, uh you know, confidently stating that they found the smoking gun uh for the hundredth time, uh but it's actually not the real problem, you lose a a of trust. And when you lose when you don't have much trust in your agents, I feel like you you really can't get the

[2:01] most out of them. And for me, the parallel is like with management. Uh so, if I'm an a manager an engineering manager for a team, and I have a bunch of, you know, I have a team of engineers on my team, and I don't trust them, then the mode of operation I'm going to be in is going to be like micromanagement, right? I'll have to spend a lot of time looking over my reports' shoulders and checking that they're doing their work well, you know, that they're not

[2:27] shipping bugs to production. And so, I drew this chart because uh it's not it's not a very scientific chart, but like this is how I imagine myself and my journey through using agent. So, you know, like fast forward or back forward uh or fast back uh fast backwards like a year or so, when, you know, nobody was or not many people were using agents to code, uh I think you uh you get into this mode where you are in very heavily in the loop with one or several like a handful of agents, and you find yourself just constantly fig- you know, trying to understand what your agents are doing,

[3:10] uh and you're very very in the loop. You're watching every single output. You are sitting there prompting, um and you really can't parallelize beyond that because you don't Again, you don't have that trust, right? You can't go to 100 agents uh like on 100 agents when you don't even trust the output of one agent. So, over the past 5 months, I feel like I've really been able to uh like ascend this trust curve, and now I'm at the point where uh I actually have This sounds kind of scary to say this, and it it makes me sound like a slop artist, but I I promise I'm not,

[3:46] but I actually have my agents now um auto merging PRs for me, uh which is like a wild thing to say, but um, like I woke up today and there were like 20 PRs landed and I just reviewed them on main, like they were already landed. And they were good. Uh, so how did I get to that point? It's basically what I wanted to talk about

[4:05] today. Uh, and again like yeah, feel free to jump in if you have questions, Colin. Um, but uh, Oh yeah, of course I got to show this this chart. Uh, where uh, Don't do do not trust to be Someone requested to control my computer. Uh, probably won't do that. Uh, but yeah, so this chart I think I I I'm sharing this chart not to kind of like flex, but to kind of show like the journey. Like so you can see like the curve like it's sort of like inversely matches the contributions I've been able

[4:40] to land at Cursor. So I joined 5 months ago. And 5 months ago like I you know, my first month I was like not very productive cuz I was you know, I was learning the code base. I didn't know what the heck was going on. And as I got more confident in in my agents, I've really been able to kind of ramp up my productivity. Uh, and again like yeah, like last month I shipped a

[5:01] thousand PRs, which is ridiculous. Uh, and then this month, we're only on the 12th, I'm already at like almost 800 PRs landed. Uh, so the velocity is definitely high and you I I'm I'm sure a lot of you will definitely be questioning like how how much of this code is actually good. Um, and I think yeah, like that's definitely

[5:22] fair to question. Um, but uh, yeah, I think I think if you set up your agents well, you can definitely get to a very similar level. Um, and so I'm going to talk about how we do that. Uh, so for me I think I'm curious like I guess Colin, your experience as well, but uh, for me I think the most important skill that you should have in your toolbox when you

[5:51] work with agents is verification. Uh and by verification, I mean the ability for an agent to actually run the code uh or take CPU traces or heap snapshots or uh you know, open an iOS simulator, whatever, you know, however your application is exposed to your users, it can do the same thing and uh run it for real and actually test

[6:17] and verify it'll work. Because that's the thing that really closes the loop. Uh it doesn't guarantee your agent writes good code, uh but it allows them to at least write correct code, uh which is a big a really big step forward for being able to trust your agent. Um I will I can share one example that we have uh within

[6:40] Cursor. Uh Oops. Where let me open this.

[6:53] Let me just make this bigger. Uh let me go full screen. There you go. Uh so for the for Cursor's agent window, uh so this is actually an interesting story, but uh when I joined Cursor 5 months ago, uh they're actually uh well, I was supposed to join a different team. I was supposed to join like the cloud agent team. Uh but then since I have a lot of experience working on React and agent's window is a React application, uh I was uh I was asked to basically help out with uh the agent window work. Uh but um there wasn't really a lot of like skills

[7:33] to help me. So I just found myself like, "Okay, uh agent's window is going to launch in like a week. All right, we have a really tight deadline." And um there was uh you know, I was just sitting there like, "Okay, I'm going to open up the performance the the Chrome DevTools and just like take a trace, look at it myself, and try to make sense of this flame graph." And keep in mind, I was just like in my first week. So, I had no idea what I was looking at, no idea what I you know, I mean, I had some idea, but you know, the the code base

[7:59] was completely fresh to me. Uh and I realized like my agent had no idea either, you know, like I would take a screenshot of the trial download trace or send it to it and it'd be like, "Yeah, it kind of looks like this, you know." Uh and it would like confidently state like, "It's this thing." And then I'd try to fix that. And it turns out

[8:15] that's not the actual thing. So, this was very very slow process. And if you've ever done any like performance work yourself or you know, just even development with an agent where you don't have a verification skill, you are the verifier. All right, you you're the bottleneck. You you you tell your agent to do something, and then it goes off and writes some code, then you open up your, you know, local dev build, and then you start to say, "Oh, you know, it doesn't work." Then you got to copy-paste screen uh you know, screenshots or console errors or

[8:42] whatever. Uh and then your agent like slowly kind of like uh you know, works with that and then tries to understand it and um uh fix the thing. But then you're constantly just in the loop and and being a bottleneck. So, there's really no way to parallelize. So, the control glass goes like one of the first skills I built

[9:00] uh for Cursor. Uh and glass, by the way, is the code name for Agent Window that we use internally, but it's just Cursor, I guess. Um and so, this skill uh is I guess that the the code itself is not super interesting. Your agent can very easily make one for you. Uh where if if you're building an Electron app or a web app or even iOS uh applications, uh you can teach your agent how to use like the Chrome DevTools protocol or through uh Apple has some utilities as well for running the simulator and taking traces and controlling programmatic control

[9:37] as well. Uh so, that's really useful. Uh but, one thing I actually want to talk about is uh the this thing. Uh where is the read me? Uh so, this skill comes with this very unique feature called or not feature a unique file called a feature map. And so, the story then is like I built this skill and so now the agent was able to uh actually run the agent window and

[10:05] take traces and whatnot. Uh but, it had no idea what what the agent's window was. So, um you know, like someone would say like, "Oh, the the left side bar is like laggy or something like that." Or, you know, the right side the the PR tab is not working. And the agent would just be like kind of flailing around. It would spend a lot of time trying to like look up the code and, you know, where is this feature? How do I actually get to it on

[10:29] the UI? Which made it basically completely useless. Uh you know, like we would I would run the skill locally and, you know, it would spawn a dev build. Uh but, then it just be churning like I just try to click here it wouldn't know how to get to things. Um and it was just an awful experience. Uh so, who was putting arrows on my

[10:51] screen? Um so, uh yeah, this this feature map has been really useful uh because it teaches the agent how to get to all of the features that you have. Um and in Psec, the plugin that I I've made, uh if you search for Psec cursor on Google, you'll you'll find it. Uh but, there is a creative verification skill in that plugin where it actually helps you set

[11:15] up something like this for yourself. Um including the feature map. So, it will actually explore the code and build up this initial feature map that tells your agent how to get to all of the different features that you have. Uh and this is extremely powerful because now that you have to user reports that come in, Uh you you can actually map even like a

[11:34] vague report or even a screenshot. So, we have this uh internally at Cursor where uh we have a Slack channel with, you know, lots of people giving us feedback on the Agent's window and raw bar and whatnot. Uh and oftentimes the report is very bad. Like very low quality. Like someone will just put very often we get like a screenshot like and then someone just says {question mark} {question mark} {question mark}

[11:57] like what is this? And you know, like the without this your agent is like I have no clue. Right? But with a feature map like this, it has a lot more context and understanding of how to actually navigate, how to get to all of the different features. Uh so like, you know, example like I guess like the sidebar. Like what is the sidebar? Uh you know, like all the different subfeatures that are present in it. Um like from the user point of view, here's where to how to get to it,

[12:24] all the different keyboard shortcuts. Uh even like the the what do you call them? The DOM elements or yeah, like the attributes that you use for selecting things through the CDP uh are all there. So, um again yeah, this is like really really powerful uh for for agents. And uh and then Pystack ships uh that create verification skill

[12:47] but also a maintain verification skill. Uh so, you can keep this up to date. Cool. Yeah, I was just going to ask how you created that. So, do you mind sharing a little bit more about um that that process in the context of Pystack and maybe just what Pystack is for folks who aren't familiar. Yeah, so Pystack is pretty interesting because uh well, first of all, the name is kind of goofy. Like the P the P in

[13:08] Pystack is like potato potato sack. Uh because I um so uh the uh there is a pretty uh famous person Garry Tan who is the CEO of Y Combinator and he's come up with this plugin called Cheestack. Uh Garry Stack. And uh funnily enough, we share the last name. We have no relation. Uh but I thought it'd be funny to kind of, you know, poke fun at Gary and make P Stack my version of of of

[13:37] of his plugin. Uh but kind of just tailor it to my own set of engineering practices. Uh but I honestly actually never set out to build P Stack. Uh it just started with a bunch of skills, right? Like I started with that control glass skill, and then I started with another skill like called how, which I also noticed

[13:56] through like observing agents. Um so like, you know, in the early days of me, you know, trying to climb this ladder, I was like super in the loop, and I was basically nitpicking my agents to an extreme degree. I was uh like I would tell it, um you know, this feature has stopped working. Here's a bug report. Like, why

[14:14] isn't it working? And very often, the agent would just like confidently state like, oh, it has to be this, right? It has to be this thing. And I noticed like when I looked at the actual tool calls, I noticed it wasn't actually reading the code that I thought should be affected. And that made me just extremely suspicious. And at that point, I was like, I'm not going to I can't trust any this agent anymore cuz it's just it's

[14:37] just completely hallucinating. And I think I think it's very easy to just, you know, like build up that distrust and not and kind of feel helpless. Like, you know, you don't know how to help your agents succeed. But like again, I think the the the management analogy is super helpful because like imagine if you were a manager of an engineering team, and you had an engineer in your team who was a really good coder, no business context whatsoever, you know, they they just you just hired them and they they onboarded,

[15:06] you know, like 5 seconds ago. Uh and so how do you actually teach that person to be effective? So how you do that is through a skill. Uh skill being just, you know, it's just markdown, right? But you know, it encodes a lot of information, instructions, a lot of uh you can really draw out a lot of intelligence from an agent by Well, some people on Twitter call it like, you know, pull the agent to a different latent space, which is kind of like a fancy way of just saying like since uh you know, LLMs are sort of like they

[15:36] predict the next token. Uh when you give it some high-quality tokens uh to begin with, then, you know, it it can kind of pattern match on like a higher space that's, you know, smarter. Um so, that's a very interesting model there. But yeah, I built Pstack very, very incrementally. Uh so, uh started with just really observing how agents, you know, all the failed different failure modes of of that agents were having. And every time I saw that, I just okay, I'm just going to make that a skill. All right, like stop hallucinating, actually go and search up look up the code, use a lot of sub-agents,

[16:12] uh and yeah, stop guessing. Yeah, that makes sense. One one kind of follow-up question here, um both from myself and from a bunch of people in the chat, so I guess it's two two parts. So, one is like how do you maintain these skills? So, like the product changes over time. Obviously, there's a lot of people who are shipping against the code base. So, how do these skills get maintained? Uh and then second to that is like how do you know when your verification is is good enough? Uh like in you know, you can trust that the very verification loops that you've built are going to

[16:40] I guess you trust that the outputs uh when they're done. Uh yeah, maybe I'll talk about um I think I have something related to that. Maybe I'll start with this one first. So, like how do I maintain these skills? So, um if you're not familiar with this concept, an eval [clears throat] is essentially like a way to uh Well, I think the mental model I have is like it's like a unit test for an agent. Um and uh you can actually make your own evals. You don't need like a special framework for them. You can build you can you can build one depending on like, you know, how scientific and how

[17:17] rigorous you want to be. Uh my screen is red. Yeah, there's a little button. Um sorry. Would you mind like disabling the drawing or something? I can't see my screen. Yeah, sorry. If you guys could not draw on the screen, that'd be great, but um there's a little button in the draw. Yeah, the the little drop down.

[17:35] Um how do I clear? Yeah. Okay, yeah. You got it. Perfect. Um Yeah, so evals are a way to unit test your skills, basically. And actually in Pstack, we ship uh under potato mode, there's a playbook, if you search for it called eval playbook. Um and it's uh uh it's like not It's actually pretty

[17:59] pretty rigorous the way it's done. Uh but um essentially what I do is I spawn a lot of different sub agents. I have like my main coordinator agent uh come up with a rubric for uh what I want the skill to do. Um And then it spawns all these sub agents, and it it creates individual directories for them, uh which are cleverly named to not let the sub agent know that it's being evaluated because uh agents can actually tell, and when they do, they

[18:29] change their behavior. Uh but it does a bunch of stuff like that to um essentially, yeah, like test whether or not the skill I'm making or changing is actually doing what I think it does. Um and one of the really nice things about Cursor is that we are we have we support so many different models. So, you can actually eval your skill across all sorts of different models, um and, you know, get a sense of how well it performs across that different matrix, um especially for the models that you

[18:59] use. Uh so, I do this a lot. Every time I I modify a skill, I will run one of these, uh like the eval playbook, uh and make sure that, you know, it's actually leading to a result I want. But I will say like maintaining skills is actually pretty hard. It requires I think a lot of taste and observation. So you kind of need to

[19:22] be very good at being a backseat driver. You know what I mean? Like if you do pair if you've ever done pair programming for example and you watch a coworker code and you just like you you could probably do this better. You know you could do you know like why did you not do this, right? You You ask a lot of questions to your coworker and it's kind of a similar thing here. You like you don't want to just be a passive observer of agent. You want to be very in the driver's seat in the initial stages when you're building up your own set of skills. You know

[19:48] obviously you can use something like P Sack, but if you're building your own set of skills, it's very I think you know opening up the all the tool calls and like reading the code and reading all the the agent behavior and their thinking blocks is a really great way to see where they they fail, right? Like what what you know where are they being done and then you can go and build a skill

[20:09] for that. And then with verification how you trust it is it's I think it's also a very similar iteration loop where you know like I actually did the same process for verifying the verification skill where I actually get um So one thing that's interesting about Evals is that you can sort of hill climb them, meaning that your Eval can produce a score, right? A score that you can get your coordinator to produce, but also you can have a judge agent of a different model to kind of cross reference and make sure that the first model is not being biased, right? The model that's judging all of the sub agents that are running

[20:48] the thing. But you can also like hill climb. So meaning that you can you can use like slash loop in cursor and you can say okay, keep looping on this Eval, right? Until everything is 10 out of 10 as an example. And I did the same the basically the same approach with the control skill and so I kind of it was very it was very hands-off, actually. Uh so, you know, I uh I kind of built I built that skill that way, like the CLI in that skill. Um and over time it's gotten really good. Uh but yeah, it was definitely not super smooth at the

[21:21] beginning. It required a lot of iteration. And I think there's that analogy here for me, which is um Well, I make this analogy later in a different slide on my drawing here. Uh but I think of it like uh you know, as a as a engineer now, you're sort of more like you like maybe a manager or the analogy I like is like you're like a a chef in a restaurant. Uh you know, you're the head chef. Uh you're not cooking all the food yourself anymore. You have a team of cooks, right? You have line cooks, you have a sous chef, you have you know, all

[21:53] these different stations. Um and it's your job to really design the environment. You know, you you you're in charge of setting up the kitchen. You're in charge of, you know, like giving tasks to different people. So, um yeah, it's a very interesting way of working. Uh but yeah, that's that's how I basically built uh these verification

[22:15] skills. Yeah, just just one follow there on like I had to go try to go one layer deeper. So, are you let's say we wanted to build um an eval or a skill for for something and we wanted to kind of get better on its own, which is is what I think you're suggesting. Uh are you doing that in like a work tree, kind of isolated with like the sub agents and and then the reviewer agents and and all that? Is it happening like in some type of cloud-hosted environment? Like what's the the more the practical steps? If I wanted to go do this uh and like set up a

[22:44] verification system for something, what would I what would I do or where would I start? Um I think that uh the best place to start is local because you can observe. You can definitely observe what your agents are doing, so. Uh if you're building a verification skill for yourself, uh I would definitely start local and just have your agent bring up the application, whether it's like a CLI or desktop app or whatever. And so you can actually observe, right? You can see how the agent is interacting with the the application. You can see it, you know, how it calls like the different APIs that allow it to interact with the

[23:21] uh, the application. Um, but uh, for me personally, uh, I have basically been kind of all in, mostly all in on cloud agents because they're extremely powerful. Uh, and the really powerful thing about Cursor is the the cloud agents, actually. Where if you spend a little bit of time setting up your environment, these control skills, these verification

[23:43] skills, pay a huge amount of dividends. Because it's not just something that makes you as a single engineer better, it actually levels up your whole team. Uh, and even your whole company because uh, you can actually start thinking about cloud agents, you can start thinking about automations that automatically do things, like uh, I'll I get to I I kind of talk about this a bit later, but I'll just kind of get into it. Uh, where where, you know, for example, like I talk a lot about

[24:10] this agent we have called Benny, right? Who who uh, you know, takes all of the bug reports that we get and it automatically goes off in the cloud, opens up a cloud uh, it's, you know, it's desktop. It runs Cursor in its own computer and it uses the same control skills to interact with the application and try to reproduce the bug. Uh, or the user report, right? And this is so so powerful because at once I can immediately I I get so much information from this automatically. Like here in this example, you can see that uh, the Benny actually reproduced the bug, uh,

[24:44] but it's already fixed on main. So it actually confirms that we fixed this problem already and all I need to do is just release another build of of Cursor. Uh, so that's like huge information there that I didn't have to go off and sit with an agent, you know, and spend an hour trying to figure out is this

[25:00] fixed is this not fixed. So, you you you gain back so much time. Uh but, you know, everybody on my team benefits from this. Everybody at the company benefits from this. Uh so, definitely think that uh you know, keeping these uh using cloud agents is super powerful. Uh but, yeah, it's like a journey. You have to trust it first,

[25:19] right? Before you you get to this point. And that's it goes back to what I was saying here where, you know, it's very hard it's almost impossible and I would definitely encourage you not to try to jump from, you know, like if you're still in this zone, you don't want to jump to like I'm going to spawn 100 of thousand or a thousands of cloud agents right now because you're just

[25:39] going to waste a lot of tokens. Um and it's going to be extremely expensive. Yeah, so just kind of recap so far. Basically, the if you wanted to go on the journey that you've kind of gone on, it would be just start with verification, uh building some some skills and some some ways of determining that the agents are producing at least like correct code whether like you said whether it's good code or not is maybe a separate question, but like it's it's technically solving the problem by looking at, you know, stack traces, looking at, you know, the actual

[26:05] behavior in the app and so on. Um and then once we trust it locally, then we can start to think about scaling into the cloud and running more agents that are picking up signals I guess on their own, right? So, whether that's like a bug report that comes in or something, they can go and pick it up and solve the problem and and give us back a PR. And then maybe the last step is like auto-merging the PRs which uh is

[26:23] where you're at. That's where everyone is at. Um and then reviewing them on main, but um is that is that about right? Yeah, exactly. I think yeah, that's why I drew this this uh this this curve, right? Because that this this basically describes my journey of, you know, when I started barely could use a couple of agents and I was just observing every

[26:41] single thing. I think there's really no shortcut for going from here to there because this is really about your personal level of trust in the agents, right? Um obviously, you know, as a as engineer, you don't want to just slap code into production. So, how do you actually build up that trust? Takes um a lot of uh I guess taste and judgment. Um but uh you know, like I think plugins like PII stack definitely kind of help you

[27:06] uh get up to speed much quicker. Uh and so, I guess it's like if you trust me and you trust PII stack, then in by extension, you can maybe trust your agents. But, if you don't trust me and I definitely would not encourage people to blindly trust me. Uh uh you know, if you build up your own set of skills that you can obviously, you know, take a look at PII stack and kind of fork it, make it your own, improve the skills. Definitely encourage that. Um but, for me, it's really all about it just keeps coming back to trust. You know, every one of us here in

[27:37] this chat have a different standard for engineering. Uh and there are different things that are important for us in our code base. And uh when you are able to encode all of that into skills and you can verify that your agents are actually doing them. That allows you to really kind of ascend this curve. And um

[27:55] uh you know, start automating things. Uh there's another piece I want to to talk about. Um if there's Yeah, go for it. I'll I'll think of more questions as I go, but yeah. Yeah, I think there's a a third part to this, which I haven't talked about yet, which is kind of an interesting one, which is like refactoring and rewriting. Like one of the uh I guess most controversial one of the most controversial topics in the industry, I think, is like should you rewrite your app or not? Um because I think engineers are very prone to this, where especially when you join a company, you come in and you see like

[28:27] the code base and you're like, "Man, this is Like who wrote this code? You know, it's it's terrible. I want to rewrite the whole thing." It is a very common inclination and I think a lot of you know, before agents, um and I guess arguably even now, people will definitely discourage you from re-

[28:43] rewriting stuff. But, I'm actually here to make a case for why you might want to consider it. Um because Uh I think it really depends. Uh you know, uh brownfield applications, I think, are actually in a pretty good spot, especially if they're set up well already. Uh and like recently I've been talking to some people, but uh you know, I I I was just observing. I I just noticed this parallel, which is that a lot of big tech company problems are now everybody's problems. Um and the big tech company problem, you know, like like when I was working at Meta, like we had this giant monorepo. We had like, I

[29:20] don't know, tens of thousands of engineers just, you know, like banging on their keyboards and and shipping code. And a lot of really great engineers at Meta, uh but uh I will say like, you know, you'll be surprised that the code quality is actually not that good. Haha. Um and so I often joke that like, you

[29:38] know, before AI slop, we had human slop. Um and so uh you know, I think a lot of big tech infra, like uh like what Meta has or Google, you know, you know, really big tech companies, are actually designed for that, where you you're sort of like you're catering to the the, you know, like uh sounds this sounds so bad to say, but like the the least capable engineer on your team, right? You build you build frameworks, you build conventions, you build guardrails, you know, you restrict credentials so that, you know, your intern doesn't wipe your production

[30:07] database. Um there's uh you know, if you have that level of infra already, I think your agents can actually already do a very solid job. Right? Because they have the the guardrails are already in place for agents to not cause havoc or not cause too much havoc uh in your codebase. Um and you can always add more, you know, guardrails. Uh but I think like greenfield applications, especially, are, you know, like the brand new applications, are like the biggest risk, in my opinion. Uh and also the greatest opportunity. Because, you know, if you vibe code a project, uh a prototype, um like we did for GrokBot, you know,

[30:45] GrokBot was spun up very very very quickly. Um and if you if you haven't heard of of GrokBot, it's like our a new application we just launched yesterday. Uh it's it's really cool. Uh lets you orchestrate your create like individual agents that have their own identity and you can kind of orchestrate them. It's

[31:01] super cool, definitely check it out. Um but yeah, that was it's like a very it was a very green field application like most prototypes are. So, it's like vibe coded very quickly. Humans were not reading the code at all. And uh I had this tweet recently uh where I said something about organic architecture. Um Maybe I'll find it. Uh but the idea is that uh when you have a completely vibe coded application, you essentially have no guardrails whatsoever. So, uh your agents when you give them a task, they will just solve it in whatever method is the

[31:39] most convenient. And over time, you get into this uh situation where you have a code base that is spiraling out of control because you don't understand it. Uh your agents understand it, I guess, in a way, but like they've built something that is, you know, optimized for short for shortcuts. Uh and uh you know, it will you will suffer you'll have a lot of of

[32:00] issues with that application. Uh so, I think starting your code base with uh like very strong constraints is very much needed. Uh because like when you have a code base that you can trust, right? When you have guardrails that actually help you uh uh help your agents write good code, you can get into the, you know, like into this part of the curve where I I where I like I I said, you know, I woke up today and I had like 20 PRs merged uh by my agents. And that's because I invested a lot lot a lot of time uh over 600 PRs I

[32:36] I I calculated yesterday. When I refactored all of Grokbot to this new architecture that I've been building. Um And yeah, I've gotten to a point where I I don't really look I really don't look at the code anymore. And I say that not just you know to sell you tokens, but because I you know it it it took a lot of work to get to that point. I spent a lot of tokens to get the code base to this point where I no longer have to

[33:02] look at it. Uh but I'm very excited about it because you know of the potential where you know it's not just it this doesn't just benefit me. It benefits everyone contributing to Grokbot. And it also empowers you know designers and product managers and you know even GTM people to add features to Grokbot. And I don't have to worry you know I don't have to to wake up at night in in the middle of the night and worry like oh someone's just merged a perf regression. Right? I have a ton of constraints and CIs like it's actually very annoying to write code in in Grokbot but like agents absorb all of

[33:36] that annoyance. Um but yeah, I'm I'm happy to talk about what exactly that is. Um Yeah, I think one question um Yeah. before we get into the this part here is just around that element of like what your your your CI looks like maybe some of the constraints and then also like the average PR size. I saw a question about that earlier. Just to give people a you know kind of a a glance. It doesn't have to be like mathematically average but just a you know like what generally the size of the PR is um if it's only a couple lines of code

[34:04] or you know um yeah. Um I think it depends. Um let me I'm trying to do this in a way where I'm not going to like You you you don't have to share the actual number of you like the actual average just I think this is fine. Benchmark. But like we have So okay. This is not that interesting but uh well fun fact is that virtualization in Grokbot and in uh Cursor is actually powered by uh pretext uh which is a sort of new library that someone's built um that's really interesting. You should

[34:38] You should check it out. But, that's not really that important. Uh I think the average PR size I actually don't know my pr- I don't know if I want to click on these. Uh I probably can, but I would say [clears throat] like they can range anywhere from a few hundred lines or 50 lines to like a thousand, depending on

[34:55] what the thing is doing. Uh so, like here I'm like here I'm deleting a bunch of files, so I expect that. It's just this like mostly deletion. Uh but yeah, it it kind of varies. There's no like Yeah, there's no like hard cap or a limit or they're all like 50 line PRs. cap. There's definitely no hard cap, but I I do encourage my agents to split up

[35:13] their work into multiple PRs. Uh I do that mostly because uh I like I like the idea of I guess maybe this is much harder to do now as in the world of agents and you have like so many commits. But, I like the idea that, you know, the Git history is a very rich source of context. Uh and I like the I like each PR to sort of atomically describe what that small piece of thing is doing, which also makes it easier for me to revert changes and like figure out, you know, oh I shipped a bug and it's this it's here, right? It's not in this 40,000

[35:46] line PR where you who knows what landed in there. Uh but I I don't have a hard cap on PR size. Cool. And then um yeah, also quick question on like CI. So, again, you don't have to go into like uh the screen share of like what your CI does, but just generally would you describe what the CI kind of looks like or how strict

[36:05] it is? Uh yeah, so uh well, specifically for Grok Bot, so Dune is the is the sort of cheeky code code name for the architecture that we built for Grok Bot. Um the CI looks pretty annoying because there's checks for everything. So, like literally I have um uh well, if you've read any react for example, you know, you know that one of the biggest foot guns in react is use effect. Uh so, in uh Dune and in Graph Bot, we've banned use effect. So, Dune is a system you can the the mental model of what Dune is uh you can kind of think of it as like

[36:43] next JS for uh electron apps and it's designed for agents to write. Uh and it's like custom for, you know, our agent forward applications. Um so, the CI checks are very like specific to that like, you know, don't use use effect. It's it's it's banned so like CI will fail uh and yell at you. We have like some of the more interesting ones that people might raise eyebrows is like I actually

[37:06] banned code comments as well. Uh which is very interesting. Um but I've noticed that 99% of the time agents just write code comments that kind of describe some historical thing that is actually totally irrelevant to the code. Um like it will often say like, you know, oh Lawrence said you should never do this and it's now in the in the code comment and I'm like, what? Like why what that was I didn't say that as like a durable, you know, global rule. I just meant like your this PR sucks and you

[37:33] should change that part. Uh agents don't really understand this that well, surprisingly. Uh and or they kind of assume too much and they kind of do things in like very stupid ways. So, like yeah, we just banned everything. Everything you can imagine like the agents are bad at, we ban. Uh so, one example that we actually suffer a lot in the agents window is we have uh you know, you know, if you've used agents window, you've definitely seen performance issues and you know, we're constantly trying to fix them. Uh but it's like a it's a never-ending struggle because there's so many pull requests that get merged. Every anyone of them could just

[38:10] regress performance or stability or reliability. Uh you know, the agents window doesn't have this architecture yet. I plan to do bring this learning back there and kind of refactor everything there. Uh but uh, it just regresses super often. Uh, because uh, there's just one example is like we have very poor um, isolation between processes. So, like on you know, on on Electron you have a renderer thread that renders your UI, but you also have like a main thread that you can run other code that you know, doesn't need to block the

[38:41] renderer. Um, but we do a poor job of separating those things. And so, often times you just accidentally have code that gets pulled into running on the renderer thread. All right, and then all of a sudden you're competing with the the renderer that you know, that has a very If you want like 60 FPS, you have to every frame that gets drawn has to be done in 16 milliseconds. So, very very small, you know, deadline per frame uh, if you want you know, a very smooth product. Uh, and when you start building bringing in accidentally bringing in, you know, things that are like very computationally heavy or they have a lot

[39:14] of IO, uh, then you just get into like a lot of jank. Uh, your your FPS really get drops. You start uh, you know, losing frames. You get long tasks that take more than 16 milliseconds and you just get this really choppy experience. So, all of those patterns that we've learned basically building Electron apps, we've encoded into this framework and it becomes like a hard failure. So, I literally in in Grokbot we literally have a directory called Electron main, Electron renderer and we have uh, import uh, CI I guess where we actually check the dependency graph to make sure you're not accidentally

[39:51] code from one directory to another. Uh, so that's enforced by CI. Um, as well as Bog Bot uh, which is our which cursor's um, like code review tool that runs on CI. Uh, you know, in our agent MD it's everywhere. Like so I I I I um, I have this thing here where I I talk about like um, you know, like there are multiple layers I think for building a good code

[40:15] base. Uh, obviously the code base is one where uh if you have an architecture like this where it's extremely strict, uh you know, the the the way to build features is very conventional, that's like the strongest strongest level of enforcement because agents just love to copy existing patterns. So, uh one example of this in GrokBot is like we have this these concepts called like a feature, and we have entry points and transcript cards, like all you know, the cards that

[40:42] you see in the chat. These are all like like nouns, I guess, in in the framework, and so there's a very conventional way of creating them. And so like a feature is all in in a single directory as an example. And so all of the code that contributes to that feature lives in one directory. So, it's all co-located in one place. Makes it super easy, you know, agents don't have to like uh grep around and try to figure out like where all the things are. It just looks at the feature and like, "Oh, okay, I'm working on the onboarding

[41:10] feature in GrokBot. Uh I'm just going to work in this directory." And for 80% of the work, it's mostly just very encapsulated there. [clears throat] But uh like it's like very it's like designed again for, you know, like the dumbest agent, like you don't have to think, right? The the the one of the key principles I have for this framework is like the shortest the

[41:32] shortest path is the best path. So, uh because that plays exactly to how agents love to write code, like they like to take shortcuts, really. You know, they'll they'll find the quickest way to solve the problem. So, why not make that the best way to solve the problem? Uh so, I I I probably won't get into all

[41:50] the specific details. Um and uh this framework's really more of a collection of ideas and principles rather than something that we'll open source. Uh you can you can, you know, screenshot this, I guess, if you want, and uh tell your agent to uh do some build build something like this for you, too. Um yeah, but it's It's all about the layers. Uh you know, like the the code base is one part with features Uh uh and directories and you know, import uh blocking import dependencies uh that

[42:20] shouldn't be imported. Uh but in and it all enforces that and static analysis. So, like uh there's CI checks. We have a lot of lints for bad patterns that we observe. Uh compiler diagnostics. Uh there's also rules in BugBot which are um I think like three, four, five are more soft. All right, these two actually make make CI red, right? So, that you know, there's a hard constraint where

[42:46] the agent can't just write crappy code. For rules and skills and BugBot, your agents can still forget. All right, you can still or it may not always consistently apply them. So, I like to layer them, but I don't I don't like to rely on them as the only source of enforcement because it's very very soft, right? And if you if you only have rules and BugBot and skills and a style guide for your code, you will it's only a matter of time before your code base looks like

[43:16] complete trash. I'm sorry to say that, but I definitely recommend you know, like you know, investing in you know, things that can be hard enforced. All right, and this is why, you know, maybe the choice of tech stack that you use is also very important. Um like I think for example, Rust is sort of making you know, it's like getting super popular again uh because the compiler is so strict, right? The compiler enforces so many different things. You know, there's a

[43:42] borrow checker that you have to appease. And as as long as you make sure your agents don't write unsafe code blocks, uh you can more or less feel somewhat confident that if the code compiles, it probably works and it's good. Uh but you will you'll see it gives you that level of trust and confidence that you as a human engineer no longer need to go and check it yourself. You you know, you you rely on code and static analysis to actually make that

[44:10] a lot smoother. Um, and I guess the worst part the worst place to be in is if you are stuck in code review land where you actually enforce all of the constraints the invariants in your code base by literally the human person saying you know reading the code and like okay you should not do this, right? Every time you have to do that you should consider that as a code smell like a anti-pattern and you should say okay instead of me on the PR how do I turn this into a hard rule, right? How do I turn this into a lint rule? How do I turn this into a CI failure or how do I

[44:45] even categorically eliminate this problem entirely. I I can talk about another migration I've done but I'll probably pause here. Sure. Yeah, I feel like that's that's where I am to be honest is is what you're describing right now which is that like I don't have all of these rules. So I have some things to go do after this session in terms of being able to scale my agents. I'm I'm definitely on like the you know maybe a couple of parallel ones locally stage so like two to three locally and I'm sure most people here are on the same so yeah I know we have like a couple

[45:14] minutes left. Lauren was there anything else that you wanted to to highlight? I I was there's lots of questions so I can grab more but I want to give you a few minutes if there's anything else you want to talk about. I think I've been yapping for quite a lot so I'm maybe let's just do

[45:25] questions. Okay cool. One question that I had a couple of came up a couple of times was just around like token usage. So the the question is like is is what you're describing a realistic thing for people who are on you know a normal set of token usage they don't have you know obviously unlimited tokens

[45:44] to work with. I think that's a really good point. I mean like obviously you know I work at a AI lab where we have unlimited tokens so I definitely cannot say that you know this is something everyone should do in the exact same way that I did it. I think it's possible to get to this point without, you know, breaking

[46:02] the bank. But, you know, if you are like an engineering leader or, you know, you're you're you have a startup that you lead, um I think there may be a question of ROI. Um and it's like uh yes, you spend a lot of money on tokens in the upfront stage. You know, you like refactoring a code base is going to take a lot of

[46:21] tokens. Uh adding all these things uh is going to take a bunch of tokens. But, if we're heading to a world where agents are writing all the code, and you know, you want to be very lean, right? You don't want to have to hire you don't want to be you don't want to become like meta, right? Like I mean like in terms of you don't want to become a 10,000 person engineering org because I mean, that's a cool problem to have, but also you know, you you have so much overhead. There's like planning, you know, like you it it's it's uh personally I I wouldn't uh

[46:52] it it's not super fun. But, um I think you want to stay very nimble, right? And you want to you want to be like agents are all about allowing you to do things that you couldn't do before. That's really to me like the value of agents. You know, it's not just throwing tokens on every single little thing. But, um to me like the thing I couldn't do before is like enforce this level of constraints in a

[47:14] code base by myself. Right? Like I'm just a single person, you know, uh it would have taken me years to build this framework uh and do it all the refactoring and test everything myself and verify you know, like run imagine if there was just me, right? You know, in in pre-agent era just like running, you know, by me it would take me so long, right? And my salary is pretty high, right? Like so, you know, the the question I think an engineering leader might have is just then, you know, like what is there's a trade-off of do you hire someone to do this or do you spend the tokens to set up a code

[47:49] base so that even the the most naive, right? The dumbest agents can do a good the And when you actually get to this point, like even agents that are not, you know, Fable-size do an excellent job of writing code. And this pays a lot of dividends as well for me in personally where I have empowered not just myself, but again, like PMs, designers, engineers who are not familiar with Grok Bot to just contribute in a way that

[48:17] is sustainable. So, I think yeah, it's definitely like a trade-off for sure. You know, like nothing's like free for sure. Uh and tokens are pretty expensive. Uh but although actually, uh I I I don't know how many of you have seen this, but we actually announced Grok 4.6 today. So, very exciting. Finally out. Um so, yeah, Grok 4.6 will

[48:36] be like a great It's very, very smart. Uh it's really good on the on the benchmarks. Uh and it's the same the tokens uh I hopefully I'm not saying this incorrectly, but uh I believe the cost per token is the same as 4.5. So, you're actually getting more intelligence for the same cost. Uh I think this is an area that Cursor tries to Cursor and SpaceX AI try to really optimize for like that Pareto frontier of, you know, cost versus intelligence. Uh you know, we don't necessarily want to build the biggest model ever because that is extremely expensive to run. It's really about like how do you find that sweet

[49:14] spot, right? You don't You don't need a giant model, but it's just super smart, right? And it's not very expensive for inference. Uh but um yeah, I think uh to kind of round it up, um I think it's like a it's it's there's a If you do your own analysis, I feel like it's pretty positive. It'll be pretty positive that the ROI you get from investing in stuff like this uh just empowers not just yourself, but your whole team to be so much more productive, right? Like imagine if you have an army of engineers like me who are shipping so much improvements and and bug fixes uh you know, every day,

[49:50] all right? Like that is pretty exciting. Cool. One last question before we wrap up. This one is for the people in product on the on the call. So, let's say we do have an army of engineers for shipping like Lauren. I'm just curious like how is the product team or other functions of your company keeping up given that like if you're shipping so quickly, how are they using AI more to do their jobs? Like as much as you can speak to that. I obviously you don't have like you're not in that role, but

[50:15] just curious about how that works. Um I think this is where Grokbot has been actually exceedingly powerful where so before Grokbot like you know obviously Cursor only had Cursor like we only had agents window, we had a CLI, we had an IDE. And these are really like power user tools right like they're they're designed for developers so it's very very developer centric. You can do a knowledge work in them, but it like

[50:40] the UI is not really optimized for that. So, we actually didn't really have well I think like a lot of people like you know GTM product like they might have used Cursor to do their work, but it definitely wasn't like a delightful experience for them. Um I think now with Grokbot uh it's become Grokbot is basically like the Cursor moment for people who are not in tech in my opinion like it's like it's like a very very accessible way to use agents in a very comfortable very familiar interface. It looks like iMessage. Um and it's very fun too you

[51:15] know you can give your agent a fun name. Uh you can have you can kind of do orchestration with in a very like natural way where you can sort of you know each agent is like a person and now you have a team of agents like working on you have one one agent per account that you manage as an example. Or if you're a PM you have you know you can have an agent that summarizes all the work that Lauren did last night and then

[51:34] now you know what I did right? So, I think our PMs are leveraging that a lot. And they're shipping code too. So, you know I often times they will just say oh here's a bug I fix it. Can you look at it? And then I'll go review it and actually it's just perfect. I'm

[51:48] like, "Okay, stamp." So, I think that shows that, you know, the the Dune architecture is holding up, right? The all the the really strict constraints allow people who are not experts in engineering to contribute at a high level. So, I mean, I feel like I'm already seeing that pay off a lot where you know, designers and PMs are just

[52:07] able to to to ship features directly. And that just makes the Grokbot team super fast, right? Where we can ship so quickly. Um and we have a lot planned, so I'm very excited to, you know, to to ship more ship more stuff. But thanks so welcome. Thank you so much for joining. Thanks for having me. Yeah, very excited

[52:31] to be here. Big fan of yours. And a huge fan of yours. I think people have been talking about this like it's like the meeting of the skilled minds. Yes. The skilled Mount Olympus or something. Cuz both of us have very popular skilled libraries. Um I've not As I was saying before we started, I've not used a ton of yours and like I want to get all of the juice out of your brain so that I can go and

[52:52] use it. Yes. Properly and use it better. I think where I want to start with this is you gave a talk um pretty recently like about 10 days ago and posted it on X, which went absolutely nuts as about how I shipped 2,500 PRs last month to production. Got about 3 million views or something on X and I watched it and I loved it and I recommended it. And I kind of want to run This is almost like a Q&A of that talk, basically, of giving you Cuz I just I just had tons of questions

[53:21] about it and I wanted to dive into it. And I think where I want to start is you talk about a trust ladder with agents, where you as you trust agents more, you can get them to do better and better things. And or scale them to up to use more and more agents. So, what is your story of how you climbed the trust ladder and how did that work when like you got SpaceX and started climbing more

[53:46] and more? So, I think this the the journey sort of began even before I joined Cursor, which is now SpaceX AI. Uh so, the story is um after Meta, so I I used to work at Meta on the React team. Uh I took a month off cuz I was feeling kind of burnt out. And of course, when what do what do you do when you're burnt out? You go and start a new side project. Um and so, I started a side project, you know, I was of course using AI to to write code. Uh but then I started to realize uh you know, I was spending like so many

[54:19] hours just micromanaging one agent, right? And you know, at the time, this was back in February, maybe February early February or January. You know, people were really obsessed with this idea of like orchestration. This was like, you know, before you know, things like Cursor, you know, like the agents window was had become popular. So, people were still in like like TUI you know, in their terminal and they were all talking about okay, here you know, I built a custom orchestrator, right? And so of course, I had I was a bit nerd sniped by that. And [snorts] you know, as I was building my toy project, I got nerd sniped by oh, how do

[54:54] I make my AI coding setup more efficient? And so, you know, I I kind of started the journey there where I just you know, took a step back and realized yeah, I was spending all this time micromanaging a single agent. You know, I was creating skills and I was like finding it quite difficult to measure the output or the the results,

[55:14] the impact of the skill as well. So, I was kind of flying blind, but I was you know, iterating really fast. Um and um so that project eventually sort of became the basis of P Sack even though I didn't know it at the time. Um and a lot of some tricks I had learned like building that early set of skills actually it's still open source if you want to if anybody wants to take a look, it's on my GitHub at potato/ a noodle

[55:43] n o o d l e. Um and in there you will see some skills and a brain directory. [snorts] And so I was really interested in this idea of how do I, you know, extract my own ability, if that makes sense, and give it to the agent, right? Cuz I was I I realized that, you know, all I was trying to do was trying to teach the agent to write code more like me, you know, do do you

[56:04] do workflows more like me. So, you know, the skills were like an entry point to doing that. Um and then, you know, after I joined Cursor, uh I was starting to work on the agents window, and uh it had a lot of performance issues. Uh it was it was it was pretty laggy. Uh and so, since I had experience working on React, I was asked like, "Hey, do you want to come and help out

[56:26] uh with the agents window?" Um and so, the the beginning of the Cursor journey was very manual. [snorts] Uh I was deep in like looking at like flame graphs and heap snapshots and trying to see like, "Why exactly is the app so slow?" Uh but then coming back to the same realization that, you know, I was sort

[56:47] of the bottleneck. Uh I was doing everything manually. I was sort of the meat proxy in a way, right? I was a meat proxy between my agent and Chrome DevTools. Uh and I was like really annoyed by that. And what month of the year is that, let's say? Where are we in the timeline? Uh so I joined Cursor in March. So this

[57:04] is like early early April, probably. Early April is when, [snorts] you know, I joined and I didn't have any skills, right? I had I I sort of abandoned my personal skills cuz I didn't think they'd be relevant anymore. Uh but then working on the agents window, [snorts] uh and now working on Grokbot, uh I sort of realized that a lot of the lessons I had learned from those skill time building the the initial set of skills were very relevant, especially around things like verification, uh you know, being very rigorous in your work. Um because I think agents, from my experience, even the the frontier ones, tend to

[57:45] tend to take shortcuts. Uh they tend to do the easy thing. Uh so, uh a lot of the skills that I built have been around how do I make the easy thing the right thing? You know, how do I make that the best thing? The idea of sort of distilling your expertise and turning what you do every day into processes, that's something that feels super familiar to me. That's exactly what I've been doing with the skills. And I suppose there's something in that which is a lot of people think domain expertise is getting less useful now as people uh start to rely more on AI. Where what do

[58:21] you think about that? Just as a sort of vibe check before we start to talk to I actually feel like domain expertise is is more important than ever. You know, uh I think I wrote this on my X at some point, but you know, at times I sometimes think of, you know, AI as this like especially as the models get smarter and smarter and more capable, and the frontier models

[58:43] are just getting so good. Like I love Opus 5.5, by the way. Um uh you know, as the models get really really really good, it almost becomes like the bottleneck is no longer the agent, right? It becomes your ability to express your intent in your goals in a clear way that the agent can understand and actually carry out. And that's why I think, you know, like people with a lot of domain expertise are extremely have a have a huge advantage, in my opinion, especially if you're a little

[59:14] bit like, you know, tech techno-curious. You know, so I think of people like, you know, like uh like a doctor or a lawyer or you know, someone who who has a deep expertise in a particular non-engineering domain. And if [snorts] they're actually just a little bit tech-savvy and they can figure out how to use agents, they can actually build really, really great products, right? If they If they have a clear enough vision in their head and they can articulate it in a way that the agent can build it, you know, I think that that is really the the bottleneck these days is is like the transfer of your

[59:49] intent, right? And your vision to the agent. Yeah, I've been obsessed with language basically since agents um dropped. I'm just obsessed. I'm thinking [laughter] constantly about the the composition of words, how I can make things sharper, what um what might be hidden in the phrases that I'm using. And it's And finding what I love is when you find a word that the agent then hooks onto and then goes, "Okay, I'm going to reinforce that word. I'm going to reuse that in my thinking traces." You know, I found that with um TDD was an early example of

[1:00:22] that. A lot of chat about TDD recently. Like, you know, people say, "Should you use TDD with agents?" Doesn't matter. What you're doing is you're getting the agent to think about TDD, getting it to write tests, getting it to prioritize things in a different way than it did before. And that's why sort of grilling I think works effectively. Because grilling is a Yeah, yeah. It It draws those words out of you, right? Or it Or at least it helps the agent understand your thinking so that it can propose those words even you can pick up

[1:00:49] and say, "Yes, exactly that." Uh I've actually copied some of the the tips that you've shared as well where you know, one of my favorite ones that you've shared recently were or not or like maybe in the past couple weeks is about [snorts] uh reducing or eliminating tautological tests. Like, one of my pet peeves of agents is like all of the useless tests that they write. And [snorts] so, you know, that was one thing where, you know, the word tautology, right? Is is is I guess, you know, not many people necessarily know that if if especially if English isn't your first language. But there's a lot of meaning to that word and it's

[1:01:21] like it's almost like compressed, right? Like you compress a lot of intent and meaning into words. And so I I I totally agree with you. Like I think language I've always been interested in language, actually. Uh like programming languages, natural human languages, and how they came to be. And it's so interesting that now with agents it's sort of like this meeting of natural language with programming language, but it's all it's all language out of the it's all

[1:01:47] communication. Totally. I did a drama degree, right? So, you know, I've been thinking about language and Shakespeare and stuff for a long time. And so, this all feels very familiar. Yeah. Um so, okay. There's sort of before we get into like cuz I think the thing I want from you is like software factory stuff, right?

[1:02:04] Software factory is the big buzzword. Software factory is the thing that I'm thinking about, too. I'm sort of releasing a course in that direction, too. And is this sort of scaling yourself up to un unrealistic numbers of PRs, basically, or PR numbers that sound ridiculous to people who don't understand how this works. So, where I want to get to is sort of from people who are doing kind of like one to five agents today up to, you know, hundreds of agents running at once and how that sort of functions. And so, I'd love to hear about your metaphor of the Michelin kitchen instead of the software factory. Cuz I think that says

[1:02:38] a bit about the way you think about this stuff. Yes. Yeah, I I I've I've never really liked the term software factory. Not because, you know, it's not accurate, but I think I think the a lot of people when you think factory, right? They don't necessarily equate that with quality or craft, right? Things which are very important to me and a lot of people and technologists who work, you know, building products. We care about the user experience. We care about the

[1:03:06] things we're building. So, so while I do think software factory is an apt term, it also, I guess, maybe conjures up negative, you know, may maybe sometimes negative connotations. So, Michelin kitchen is the thing that I've sort of landed on where it's much more I feel like it's much more aspirational and I like the metaphor a lot cuz you know, I like food. I'm called potato, of course,

[1:03:29] and I like cooking. And I see a lot of parallels, right? Like with food, right? When you're cooking a meal for yourself, for example, [clears throat] it's both utilitarian that you're trying to just feed yourself, right? And and survive. But it can actually be transformed into art, right? And that's what what a Michelin star chef or even just a chef or a cook can do with food is take something very ordinary and turn it into a delicious meal that, you know, takes you back to your childhood days or

[1:03:57] something like that. Um And so it almost like mirrors that trust ladder that I talk about where you can sort of imagine your own journey as a home cook, right? As a home cook, you're doing all of the food the cooking yourself. You cut all the vegetables, you do all the prep work, you do all the cleanup. You know, you are the one-man

[1:04:19] or one-woman show, really. Um And it's an interesting thought experiment like, okay, if you were to cook a meal and then you add people, right? You your your partner enjoying to your brother, your sister, and now suddenly you have your whole family in the kitchen. I think most people would get very stressed by that, right? The thought of oh, so many people are just mucking around in my kitchen. They have

[1:04:41] no no idea where all the utensils are. I have a max capacity of one person in the Yeah, exactly. Yeah, absolutely. So, I feel like that that's really apt because when you ask yourself that question of how do I go from being a solo cook, right? To having an army or even not not even an army, but a few sous chefs, right? That that are helping me in the kitchen. How do I think about dividing the work in a way that makes sense? You know, I'm not dividing work just for the sake of it, but in a way that actually makes the sum the total the better than, you know, the

[1:05:12] total of its parts. And so the Michelin kitchen metaphor to me like works really well in that regard because, you know, as a chef, you're you know, if you become a chef, you're in a position where you're not necessarily cooking all the food yourself anymore, but you are thinking you're almost like the tech lead, right, for the kitchen, where uh you know, chefs have to think about, you know, not just cooking, but they have to basically organize the whole kitchen and they're like the CEO of the kitchen. They have to think about when to you order ingredients, how do you store them, how do you prepare them, when do they have

[1:05:45] to be prepared? You know, it's a whole it's a whole job, right, that's not just cooking. Um and I think that again it mirrors so much of how engineers, right, code today, where you are not writing the code yourself anymore, you have agents, right, but you as the human are still responsible for the final outcome, right, your name still is associated with the work that

[1:06:06] you do, your reputation. And you know, so how you set up your kitchen, right, and how you set up your skills, your environment, your code base, I think are ultimately the new ingredients that go into um building product. Yeah, I think what I love about your approach is the amount of focus that you put into the environment that the agent operates in, right? Cuz I think a lot of people they think, right, the agent is good, um probably not going to be able to make it better, let's just trust what these magic model people have put into the harness and the model combination. Uh there's nothing I can really do, like I

[1:06:44] can't mess about with Claude code's internals or something or whatever you're using. Um but what I love about your approach, and it's something I advocate for, too, is that you can change the environment the agent operates in, right? You can make changes in the code base and also give it tools for verification, as well, and allow it to verify its own work. So, the thing I I loved about watching that talk is the amount of focus you put in verification and like that is the lever

[1:07:12] that you can start to generate trust. Can you talk about that and what that concretely looks like? Let's start like looking at practical ways that people can improve their own processes, their own kitchens. Yeah, I've I've I've said this a lot actually that you know, even if you don't use P Sack or you know, your skills, I think that the single most important skill that should be in your toolkit is verification. Because without verification and for for by the way, for those watching who don't know what that means, it's this idea that you can give you can sort of give your agent uh hands and eyes in a way it that's the the

[1:07:48] analogy I like where the agent is able to run the code, right? And actually uh interact with it like a normal human user would and also do things like, you know, debug it, you know, take traces and snapshots. Um and uh funnily enough, like that was actually the first skill I built when I joined Cursor. Uh that gave me a lot of that was that was the thing that actually started to let me ascend the trust ladder a little bit in a way that some of the other skills I had looked at

[1:08:17] or built had not really let me through. Because no matter how good, you know, some of the other skills were like the how skill, the why skill, the un slob skill were, I was still [snorts] relying on me, right? As the proxy between my agent and the output. So that, you know, if the agent can't actually see the result of its work, there's no way it can actually iterate. Right? And so this is where people start to talk about loops, this

[1:08:42] idea of a loop. And really, I think the term loop, you know, seems kind of uh almost abstract. Like people are like, what what what what is a loop? What is an agent loop? But really it to me like the most important part of a loop that allows it to be a loop is the verification part. Because the agent is able to to verify its own work and uh you know, that takes you out of the equation where now I can actually do something like So the very one of the very first use cases I had for verification was you know, like the performance work that I

[1:09:13] was doing on cursor's agent window. And I want I wanted to get to the point where I could do something called hill climbing. Uh which is a term that I I think the labs I talk about a lot which is this idea that you know, you have some kind of rubric or a way to judge or

[1:09:28] score something. And now because you have a loop, you can have an agent continually try to make improvements to that. Uh I think Karpathy Andrej Karpathy also famously released uh something called auto research that has a lot of these ideas. Um but yeah, verification I would say is probably the most important skill in PSAC uh and many other you know, tool

[1:09:52] sets. Uh and I think it's the most important thing to focus on. So a lot of the a lot of the I I spent a lot of time actually you know, tuning the verification the creative verification skill. Um and also internally the the we have so many verification skills now like every app that cursor has or SpaceX AI has has a uh verification skill that is auto maintained as well. Uh and it's become critical infrastructure for our team cuz

[1:10:22] everybody uses it. And you went pretty far with that too, right? Like you had a um in your talk I saw that you actually built a custom CLI for that, too. So what does that CLI do? Like how does it [clears throat] execute things and why did you I mean that's proof of how deep you're going, right? Of how much you're

[1:10:36] pushing that. Yeah, so this is actually a tip I learned early on where um I guess you know, back in January or or or late last year, the thing that people were concerned about was context window. Right? That was the big the big topic at the time was how do I you know, manage the context window because, you know, compaction, summarization wasn't really that good yet. And [snorts] people were always people had this there was almost this meme in the community that you know, once your agent summarized or compacted once, it would become sort of stupid, right, for the rest of your session. So, there was a lot of thinking around like

[1:11:17] you know, being very efficient with your context usage. And so, that was actually the inspiration for some of the uh the CLI work inside of the verification skills. I guess that now it's less so about context because uh you know, agents are much better or harnesses have gotten a lot better with summarization. Um I still think there's some benefits to, you know, uh having a clean context window. Uh so, the CLI is really just more of a way for me to take the deterministic parts of what the skill does and encode that into a script or a CLI to reduce to to kind of take away the

[1:11:57] judgment that would otherwise unnecessarily be used because with judgment so, that I also think of, you know, agents and skills in sort of like is like a gradient. You have some parts of the work that are entirely judgment-based, right? You know, it's something that requires thought, you know, putting together multiple pieces of context, thinking. Um and then you have the more deterministic parts like, I don't know, if you wanted to uh refactor some code, right, from one pattern to another, that's very mechanical, right? You don't you don't need an agent to think about it and come up with it in a novel way each time. All

[1:12:36] right. And so, that that was really the inspiration for the CLI and you'll see this in a lot of the other skills that I built is like I try to extract out the deterministic parts and turn that into code and just leave only the parts that actually require judgment to the agent. So, in a way I think of the CLI is kind of like a wrapper, right? It's a wrapper with some light instructions or on how to use these custom tools that are

[1:12:58] inside of the skill. Um Oh, yeah. The I don't think the CLI is really that interesting in its own, really. It's not like a novel piece of software. It's just something that interacts with like Playwright and the Chrome DevTools protocol and calls a bunch of APIs. It's like It's just a bunch of glue. No, it's fascinating, cuz it's a way of hiding information from the skill, right? It's a way of conserving the skill, keeping the skill quite small, I imagine. And then you're able to delegate more of the complicated deterministic stuff into a script within the skill. So, you It's almost you're compressing information and making the

[1:13:31] agent do more consistent things more consistently. Yeah, it's fascinating. Yeah, and it also helps, I guess, if you care about context window. It does help because now that agent doesn't need to uh you know, re- reinvent uh things. Cuz uh one thing I had noticed early on when we didn't have a CLI, was that uh well, the the agent would try to verify its work, but it would basically rebuild the world each time. And then every agent did it differently. And I was starting to notice like that's very inefficient, right? I was wasting It was actually not just about context usage, but also speed, right? Like it's now an

[1:14:07] agent had to actually go off and write the scripts or the CLI and test it and you know, and it doesn't work and the last agent did it and it worked, but it discarded it. So, it was just very obvious at that point like I should just turn this into a CLI and put that inside of the skill uh so that every agent that uses it now

[1:14:24] benefits from that same piece. Um but I I also think like, you know, it's a good push for people to think about is how much of your skills and rules could actually be deterministic. Um that's like another core thing or or one of my core principles that I like to think about is yeah, how do I uh make very efficient use of determinism and non-determinism and you know, let agents shine at the non-deterministic parts, right? Because

[1:14:54] that's what they're trained to do. Um and the other parts which are much more mechanical or you know, straightforward can be just pure determinism. Um and you'll see this as well for things like doing migrations um which is another big thing that I've I've talked about is you know, going from one technology to another, especially one that is better for agents, right? And a lot of how you can do that migration is I think through things like scripts and CLIs, like the deterministic parts, like code mods, you know, like crawling the AST, the abstract syntax tree and transforming code literally mechanically, right? Like a script does it for you instead of the

[1:15:34] agent. Totally makes sense. I I mean I think what there's another thing there which is you're taking stuff away from the agents and you're kind of putting it in the environment too a little bit, which is let's say you have a a thing that you notice the agent always gets wrong, you want to make that um just impossible within the environment and that sort of comes down to code quality as well. I mean I talk about a lot like having a what a good code base means, right? What is a good code base? And I there's a definition I like which is a a good code

[1:16:10] base is a code base that's easy to make changes in, right? Easy to um change stuff without things screwing up and that means that you have a lot of guard rails, that you have a lot of um the agent or the human is constrained to very narrow paths and that's again something you talk about in your talk. And you talk about this not only on the kind of sort of automated checks side of things, so linting and type checking blah blah blah, but also in the way you design abstractions and you guys even I think built a framework uh for your agent to

[1:16:40] work in too. I think what I'd love to hear is you obviously think of that as very important, right? And [clears throat] that's how important is that compared to other things you could be doing like building features or shipping work? Yeah, I think that's a uh I almost feel like the new job of the engineer is really to to spend time on the environment. Um I almost actually wrote a tweet about this yesterday, but

[1:17:04] I but I didn't. But I think that I think that you you know, if you if you haven't really spent time, you know, building trust in your agents and building skills and tools, you can get stuck in this mode where you're very low on that trust ladder, right? You don't have a lot of trust in your agents' work. And so the only way to cope in that when you're in that situation is just to kind of lock in and micromanage your agents, and that's very time-consuming. And when you're stuck in that mode, you don't really have the luxury to think about, you know, uh higher-level things, right? Like like

[1:17:40] making yourself more productive. In the same way that uh I guess the analogy would be like if you've never taken the time to learn like your tools, right? As a developer when you're writing code yourself, and you know, you you have you've never heard of VS Code, you've never heard of Vim, you only knew about Notepad, uh and you hadn't hadn't even heard

[1:18:00] about Git. That's sort of the analogy. It's like you you haven't spent the time sharpening your own knives, right? And so of course, if you have a dull knife, then everything's going to take a long time. Um and you going to you just going to be and and especially if you know, deadlines are looming, then you don't have the Now you're stuck in this rut, right? Where where you you you don't have sharp knives, you don't have good tools, but you're under all this pressure to ship, right? And so all you

[1:18:26] can do is just focus on that. But I do think that, you know, if you can find yourself the time to actually spend time thinking about your setup, it's again going back to the cooking, you know, an analogy, it's like uh you know, if you for example, if if cutting cutting cutting the garlic is like super slow, right? There are garlic mashers, right? You can you can buy it and you put it in the thing and you like squeeze it out, right? It's super fast. Uh machines and

[1:18:53] tools were invented for a reason, right? And so, if you're operating a Michelin kitchen and your your your cooks had no tools, then of course everything's going to be extremely inefficient. Very very you know, every every every cook is going to make something up up their own. So, I think the tools and the determinism to me are you know, taking that part away. And And just like you said about constraints as well, it's the constraints are are to me as well, like uh actually a slight tangent on that is uh I I think we should talk about TypeScript [laughter] cuz like we we actually both share like a background in

[1:19:28] TypeScript where, you know, you obviously have done a lot of work with TypeScript and total TypeScript and, you know, you're a leader in that space. And I uh had adopted TypeScript pretty early and I had given like a talk or two at TypeScript Conf uh many years ago. And so, one of the the talk that I did actually was about type systems and constraining the constraining types, right? One of my most favorite things about TypeScript is actually type narrowing, right? This idea that you go from a very broad type, right? That could be anything. And then you through type guards and, you know, type narrowing and, you know, runtime

[1:20:03] checks, you can actually narrow the space and say like, "Oh, this isn't just a string. This is a very special type of string. It's a constant, right? Like I but I I determine that through the type system." And in a way it's like uh there's a lot of parallels, I think, to that with constraints in your code base where is it your your constraining the space, right? If you if you think about category theory as well, you know, you're constraining the the number of possible types, right? That can can

[1:20:32] exist. And you're saying there's only one type, right? And for for us, like that framework that I'm building called Dune, it's not an open source framework. It's the way I describe it to people, it's it's kind of like our internal next JS for our electron apps. But it comes with a lot of really really restrictive lint rules. And the code base is designed in a way that there's really only one way to do something. So we make use of a We make use of a lot of conventional patterns. So like features all go into a specific directory. Well, every feature has its own directory as an example. You

[1:21:10] know, there's like a a thing that discovers features like through a registry and like crawling the code base and stuff like that. But this conventional pattern and the lint rules make for an environment where it's actually very hard to write bad code. And that sort of frees up the It both frees up your own mental, you know, capacity as well as the agent sort of doesn't have to think about that

[1:21:38] anymore. Where it's just like, oh, there's only there's I should just if I want to add a new feature, it just goes in the feature the new feature directory. Now the code goes in there and I'm not going to append to a god file. All right, that was really actually the inspiration for those feature directories is the very first couple of versions of Godbot were composed of like eight god files which were like at least 10,000 lines long if not longer. And so I kind of had to break it up into smaller

[1:22:06] pieces. But it was just observing, you know, I think that's another important part is observing how agents fail. And then every time you see a mistake, every time you see something that could be done better, you think you step back and think, how do I turn this into a lint rule? How do I make it so that the code base makes this impossible, right? And it comes back to me from my, you know, my background learning TypeScript and types uh type systems is how do I constrain the space so that, you know, I

[1:22:36] know precisely what I'm working with. And I think, yeah, there's a lot of parallels there. Totally makes sense. And then, I mean, it's funny that you mentioned TypeScript and Go files in the [laughter] same sentence cuz TypeScript famously has a 25,000-line type uh Go file. I already knew that. [laughter] Well, I don't know if they've rewritten that in Go. I don't know. They probably

[1:22:54] haven't, have they? Um okay. So, environment is important. You should watch your agent like a hawk to make sure that any mistakes it makes, you turn them into things in the environment. And the benefit of the environment is you're not overloading your agent, right? In terms of rules, in terms of things it has to remember. It's just in the environment. And so, it stumbles into the rules and

[1:23:15] Exactly. um you know, bounces off them and hits them at the right moment. So, okay. We still haven't talked about the 2,500 PRs. Where do those come from? Like, how you you've built your trust ladder, you've worked on your environment, and you understand, okay, um I now want to scale up. So, what are the mechanics of that scaling? Are you um initiating 2,500 like chats per month? That can't be right. So, there must be Are there any kind of automated triggers that trigger stuff in your repo? Like, how do you get the software factory kind of triggering

[1:23:49] work by itself? Right. Um I'll definitely say that the prerequisite to, you know, something like uh a very high volume of of pull requests um is the environment, you know, the the stuff we just talked about, where I definitely would not have been able to do this if I had not spent the time, you know, thinking about the kitchen, right? And the knives and the tools for

[1:24:12] my agents. And so, in a way, I I think of this as I've spent the time building one kitchen and one restaurant. And now I I'm in a position where I don't actually have to be there anymore because the environment, you know, that the same analogy, right? It works really well. of restaurants, a chain of restaurants,

[1:24:33] [laughter] right? That's it. Exactly. Yeah, exactly. It's like you're Gordon Ramsay and, you know, you you've taught your executive chef like all the tricks of the of coming up with great menu and like the kitchen is set up really well. Everything is just perfect. And you're now in a position where you can open your second, your third restaurant. And I guess I I sort of see each project that I work on, like each big chat, is sort of like a restaurant, right? And and I'm I'm I have multiple of them operating at the same time. And I'm sort of like helicoptering between them, sometimes some more than others depending on how

[1:25:08] in the loop I am. But yeah, definitely I think there's there there there external triggers and context that those projects don't have that for a long time I was the proxy for that. So, uh the best example I have is like, you know, you have a project that's working on a feature uh or you're trying to fix a bug and you're getting bug reports, but the bug reports are going to things like Slack or linear or X, right? And these are external systems that aren't connected to your inner loop. So, I like to talk about this outer loop and the inner

[1:25:43] loop. Uh I don't know if I'm using the definition correctly, but to me my inner loop is like basically my engineers, my agent engineers working on the code to and building towards an intent or snapshot of my intent, right? And the thing about that is that the snapshot can go stale, right? New information comes to light that I then have to be the proxy of and, you know, transfer that context to my agent. So, you know, if you if you don't have these triggers pulling information back into the inner loop, then you sort of have to play that role where you're you're off, you know, in

[1:26:20] Slack or X or or whatever, and you're gathering context, right? You're getting context about bug reports, about feature requests, about, you know, something someone said about, you know, our back-end infrastructure has some limitation, you know, all that information, you have to ferry that across to your agent. So, that's where I think like tools like Clockwork are really good because they help you automate the the outer loop as well. And when you connect those two loops, it's very, very powerful because now all of a sudden, your agents have the ability to get

[1:26:53] context for this for themselves, right? If, for example, uh you know, either through just as a simple example, like maybe you have the Slack MCP, right? Or you have your own harness, right, that you've built a Slack subscription into for a particular Slack channel. Now, all of a sudden, you can tell your agents, okay, subscribe to this Slack channel. Every time there's a uh you know, bug report about something, go off and triage that thing, right? Go reproduce the issue, right? Using the verification skills that we've already spent time building, and all of those other skills that we've set up. So, that I have a lot of trust, right? I have a

[1:27:32] lot of trust that these agents can actually go off and understand the bug, you know, uh verify that the bug actually still exists on main, and it wasn't something about you maybe the users' setup or their data or maybe, I don't know, they didn't install a dependency or something like that. Like uh basically, [clears throat] I think uh creating that yeah, creating those two loops and connecting them is really a very

[1:27:59] important part of the job these days. Um especially if you are thinking about how to scale yourself. So, a big theme here is really just like always thinking about like what where am I the bottleneck in this process? Why do my agents need me, you know, to answer this question? I I always like to think about that. And so, I try to think about how do I actually get the agent to answer its own question, right? But not by hallucinating, not by guessing, but

[1:28:24] actually real data. And, you know, a lot of people talk about this idea of a company brain, right? Or a context graph. I feel like those terms are unnecessarily complex uh or even abstract. To me, it's just about uh how do I take information that my agent needs that I would otherwise have to go and pass it myself and just teach it how to do it, right? And that removes me from the equation. And so, how I arrive at 2,000 or however many PRs is the fact that I have all these loops set up, right? And so, uh it it allows me to open chain restaurants, right? I can I can really

[1:29:04] paralyze myself. So, yeah, I'm not sitting there creating 2,500 chats, right? Of course. It's really like these projects are um Actually, Cursor has a new feature called projects, which are these like coordinator agents. Um And so, the coordination coordinator agents are really good at sort of delegating and not doing work of their own, but they manage and supervise like almost a list of tasks and they spawn some agents to go and do them. And so, I'm just constantly feeding context or teaching the agents how to get their own context, and then they're going off

[1:29:36] and doing the work for me. Uh and really the big the last thing I'll say to this is like the big unlock for me for getting to 2,000 PRs is starting from the question and working backwards of how do I get to the point where my agent can merge its own code? Because the obvious thing people ask me when when they when I tell them, "Oh, I shipped 2,000 and 2,500 pull requests last month." They'll be like, "How did you review that, right? That that's a lot of PRs to review, right? Your team

[1:30:04] must hate you." Do you Do you mind if we go there in a second cuz I have a good question about that. Yeah, yeah, yeah. I want to like this analogy is great. I want to like deepen it a bit which is before if you're like manually initiating all those chats, it's like you're bringing the orders to your chefs manually, right? Whereas if you've got an agent

[1:30:22] Right, right. sort of like doing the expo then you're able to sort of run it yourself. What is what does that concretely look like then? You've got these sort of grok bots that are um subscribing to channels, pulling in Slack messages, and you it sounds like you have a couple of coordinator agents or like chief of staff agents that mon- like monitor that or something. Like when you look at your computer to manage your agents, what

[1:30:45] does it look like? Yeah, so so this is I guess somewhat confusing but we're working on, you know, simplifying and unifying. But so uh there's grok bot uh which or you know, you can use other tools of course as well. But I I I largely think of these tools as like your outer loop. These are tools like you know, grok bot that have connectors, right? These are connectors I guess that a lot of people

[1:31:08] call them personal agents. Um but they're connectors to things like your email, your calendar, Slack, uh Pla- Plaid, I don't know, like all these different services. And they are a great source of pulling context in to your work. So the same way that a human like you know, if I were if I was a manager and I was leading a team of engineers, um you know, like when I used to work at Netflix, one of the biggest things that managers would talk about was this idea of context not control. Which funnily enough, you know, has so much uh has so much uh carry over to the agents

[1:31:45] world uh of you know, you you know, you have you have course can drive to an outcome you want by control, right? Like by micromanaging. But what you want is to provide context instead, right? Like teach the agent, teach your engineers how to be self-sufficient. And then you don't have to micromanage them. And so I see a lot of parallels there. But yeah, Grokbot so concretely I have some Grokbots that look at my Slack channels, look at my X, or my emails, or linear and they're just constantly they have routines that subscribe. So they're constantly watching and I have I'll tell them things like you know,

[1:32:24] I'll watch for issues with bugs in the Grokbot desktop app as an example. And whenever you find that send it to my cursor project. So one of the really cool things about Grokbot is it connects to cursor. So cursor has like I I just mentioned the new feature called projects. And a project is really a again like you get a coordinator agent that's in the cloud. It has its own computer and all it really does is like it's a manager of agents. It's like your executive chef, right? Your your chief

[1:32:54] of staff. It doesn't do the work itself. It delegates and orchestrates and manages the work of other sub agents to you know, that report to your chief your chief of staff. And it basically is responsible for driving the work forward and managing things and passing context to them. So if you get a sudden burst of issues, let's say you get 30 issues at once in one payload or something or very quickly, the coordinator agent can

[1:33:23] figure it out and delegate. It gets like you know, 30 the 30 or so payloads and spawns a sub agent or a single coordinator agent. It can actually do a a bunch of different topologies of agents and it will sort of figure out the best way to you know, efficiently distribute the tasks to your team of agents. Um So I use cursor projects a lot and I also use Grokbot a lot. And cursor projects are my inner loop and Grokbot is my outer loop. Grokbot takes all the context external context gives it to the projects cuz it can actually just send messages to those

[1:34:01] projects. Right? You don't even have to open cursor. You can just tell your Okay, create a project, right, for these series of tasks. They're all related. Right? Maybe as an example, you know, you've had a a a [clears throat] big burst of issues that are all about performance, right? Your app is slow. Uh and they're all connected, right? Maybe some of them even have a similar

[1:34:23] fix. Right? And you can certainly go off and just spawn one agent per task. But then you've lost that sort of thread between them, right? And and you may duplicate work or you may not really think about the higher-level problem. You know, sometimes when you you solve bugs, you know, it helps to have multiple bug reports that are all slightly different because it helps you really, you know, zoom out and see actually, you know, the problem when I looked at this one report, I thought the bug was here. But actually when when I see the other multitude of bugs, it's actually up

[1:34:51] here. Yeah, got you. So that's why you have so many agents in that loop then, right? Because it's not just you have like you have a bug report comes in, you spawn a single agent to look at that bug report. That that single agent will be duplicating work with other other agents, right? Because if there are multiple bug reports coming in through the same thing, that could be

[1:35:10] duplicated work. Yeah, fascinating. That's really fascinating. Okay. And so this just this endless series of triggers coming from real users reporting real reports um builds up this sort of and accelerates the factory sort of adds more orders in. Other than bug reports, are there any other sources that you use for like um accelerating for pushing these PRs? Uh well, funnily enough, it's some of it comes from reading the code, too. So I guess I have sort of uh Well, I so to clarify that, you know, the 2,500 PRs, they're not obviously like 2,500 features, right? They are a lot of the

[1:35:51] work actually is spent on gardening. Like another term that I really love. Where So I guess this is more important when you have a big team of engineers, human engineers that you work with. Where And it also this goes back a little bit to what I was talking about with the environment. You know, setting up a really good environment that doesn't just help you and your agents, but everybody on your team, right? Think of a new hire who doesn't have a lot of context on all of your engineering

[1:36:18] practices joining your your team. And if you have a really good environment, they can be productive from day one, right? They can they don't have to like, you know, make open a bunch of low-quality PRs. They can start you know, they can start just churning out really good code. Um And uh yeah, I think

[1:36:37] I I I sort of lost my train of thought. I've got a I've got a follow-up which is what what are the mechanics of like triggering like how when when do you trigger a a a agent to go and look at the code, right? Cuz some people might say, well, let's just do that every hour or something or like on a cron job or what's

[1:36:55] Oh, yeah, yeah, yeah. Yeah. Uh I saw some of your recent tweets as well about like, you know, the some of the tweets you've been doing which are great for setting up these routines. Uh I have some routines like that as well. Um so uh one of them is like looking through just a another simple example is, you know, React [clears throat] has a lot of footguns. Um so uh as as I'm sure you're aware. And so I have an agent that's just constantly looking for bad patterns. And the interesting thing about that one is that I don't actually tell it to fix the issue first. I tell it to append it to a

[1:37:33] document. And then every couple of days I look at it and I see actually these are all the same thing, you know? And so that gives me you know, you almost want like a buffer, a queue. Sometimes that's actually more effective than just spawning off a couple of like a lot of sub agents to fix every single thing. Cuz when you are [clears throat] in kind of pure execution mode and just trying to like you know, you know, execute on the orders that are coming in very fast, you sometimes miss

[1:38:00] the big picture. So, sometimes having a buffer forces you to think about the big picture because you you you have these artifacts and things that you can look at as a human and sort of use your own human judgment to or I guess you can use an agent as well. But you give the agent and yourself a way to identify patterns, right? That you might otherwise miss if you're just only solving each bug at a time. And that's also really the benefit of having something like a chief of staff agent is it can see the forest,

[1:38:34] right? In addition to actually doing the execution. Fascinating. That's I mean my brain is exploding a bit there with the sort of chief of staff as the software factory, I think. I might have to change some of the course that I'm filming next week. That's amazing. [laughter] All right, let's talk about let's talk about review, right? Cuz this is the reply that you get. Is did you read did you taste all 2,500 of those dishes as they swept past you? And I assume the

[1:39:02] answer is a variety is a version of no. Yeah, I I think you you you don't want to be in a position where you're not tasting a food ever again. But you also, you know, for scale you can't be tasting every single dish that comes out of your kitchen, especially if you have multiple restaurants. So, it becomes more about sampling, right? And

[1:39:22] thinking about the processes. In the same way that, you know, if I guess maybe this is where the the factory analogy is a bit more apt is, you know, as a quality supervisor on a factory, you can't look at every single item. You sample, right? You take a you you you go in there every day and you look at the quality of the pull requests. You look at the code that the agents are writing and you scrutinize it very rigorously and you think about all the inefficiencies, the bad patterns that the agents are doing and then you think about how to course correct the environment, right? Not not

[1:39:58] that single agent because if maybe if it if it was a one-off incident, it's fine, you know, that maybe there's nothing to fix there. But if you actually notice that multiple agents are having the same issue, right? They're taking the same shortcut, they're they're propagating the same workaround everywhere. That's a sign that you should go off and think about how to amend your kitchen or your factory, right? Like thinking about your skills, your constraints, your lens, your type systems and setting adjusting it so that that problem doesn't happen again. And when you do that enough times, then you get to a place where the code base is again

[1:40:39] like the environment is so constrained and so it guides you so well that you can just you can just step away, right? That's the dream and I'll I'll definitely say it it's very hard to get to this point. I don't want to sell this as like, you know, something that you can just do easily by using Pisac. Like it takes a lot of time and effort to think about your code and where you see your agents failing and thinking very thoughtfully, intentionally and setting up guardrails and constraints so that they do the

[1:41:11] right thing by default. And you're not [clears throat and cough] like if to go back to the software factory analogy, this isn't a dark factory, right? This the lights are on, right? of is, yeah, actually. Is it? Yeah, well, it's dark in the sense that so it's dark in the sense that well, I think that my agents are merging their own pull requests, it's sort of become dark where I go to sleep

[1:41:34] and my agents are now work 24/7. Uh, have I have the equivalent of like more than 10 chief of staff, right? Each working on a different area like For example, I have one that's working on performance of the Dropbox desktop app. I have one that's working on uh, fixing bugs that users report. I have one that's exploring rewriting it in a different language just for fun. You know, like what if what if, you know, just reimagining what what it would be if it

[1:42:00] was like a native app. It's just a toy. Um But the idea is like you I I I when you spend the time setting up your environment I've gotten to a point where I review the pull request after it's landed. Right? I I tell my agent full autopilot is is something that you can do in in Pysa. And that will trigger off this very intense rigorous verification loop where I'll spawn a bunch of verifier agents for every pull request and it will fuzz, right? Fuzzing meaning that

[1:42:29] it will actually run the application. It's going to click around and try to use it like a real human. Look for regressions, look for bugs in your implementation. And um it will try to find issues with the thing and then it will fix it itself. It'll do that again and eventually get the PR to a state where it can land. Uh, so it does it does it's it is quite token intensive. You can tune this, of

[1:42:53] course. Uh, so you know, instead of like 10 verifier agents, you might do like one. Right? Or you can just tell the agent to verify it's on work. But yeah, the key thing is the verification part is really the key piece. That gives me a lot of confidence. That I guess verification plus the environment. Right? It's these the combination of these two things that allow me to step away and say, "Agents, go off and merge your thing." I'll review it in the morning by looking at my commit history. And if I see problems, I go and course correct. All right. And I'll go and revert or modify,

[1:43:26] add new lint rules, and whatever. Um So it does it does take time to get to that point. but once you get it all, it's so it feels so magical. I I I tell people like I'm sleeping so much better now because, [laughter] you know, it took it the very first day I turned on the sort of dark factory, it was very scary cuz I was like, "Ooh, what if I cause a sev, right? What if I break something overnight?" Uh and it took a lot of it took a lot of bravery, I think, to do that, but somehow I did it and now now I'm in a place where my agents are are

[1:44:00] merging their own code while I sleep. It sounds like I think it's dark in that sense. Yes, it's dark sometimes, right? You do sometimes That's true. That's true. Cuz I think of a dark factory as like almost like if you take the original definition of Copacetic's vibe coding, right? Which is the code almost doesn't exist. You forget the code might be a thing. I think your approach is totally different from that, which is that code and the environment is essential. And if the code and the environment are bad, then you will get bad outputs, garbage

[1:44:27] in garbage out. I see. So, I I think I think this is this is a different thing. It's like, you know, the I don't know, maybe there's a dimmer switch or something, right? Like, you know, some parts of the factory are dark, some parts are light. This is why that maybe the the kitchen

[1:44:42] is a better analogy. yeah. The restaurant because, you know, even as a as a as a as a restaurant restaurateur, you still might go to your restaurant every now and then to take a peek in, taste the food, right? Yeah. Uh I think that's The idea of sampling instead of blocking, I think, is really

[1:44:58] important. yeah. I think what would you say to people who are in I guess you're obviously in a pretty security-conscious environment where you're working, very security-conscious. Mhm. Um maybe there are folks working in like medical applications or law or finance or something. I think of that like some PRs are kind of like two-way doors, which is you can merge it and then revert it, right? It's easy to move back through. But there are some PRs that are one-way doors, right? That will cause data loss of some kind that will do something that can't be easily walked

[1:45:32] back. How do you deal with situations where most of your PRs, let's say, are one-way doors? Like, is this something you just wouldn't recommend? Or like, what do you think? Yeah, I think that's a really good question. I think that it all comes back to me to the quality of the verification that you're able to um

[1:45:51] get out of your agents. And I think for domains where the work is verifiable, this is easier, right? And the the one-way doors become two-way doors in a sense. But I guess I don't know. If you're working on something that is like extremely is very hard to verify uh programmatically, then I think yeah, you're definitely in a position where

[1:46:14] it's very hard to get to that point. Um so, I do think like yeah, verifiability of the domain is an important aspect to be able to do this. Um and software engineering is just one of those things where it's quite verifiable in in a lot of cases, maybe not totally. Um you know, like other domains like mathematics, I think are another example of not all of it, of course, but some aspects of mathematics can be

[1:46:41] verifiable. Like, if you write a proof, for example. Um and so, yeah, I I think it's a great question uh that I don't really have the answer to, and I think that this is something the industry and us as engineers will have to figure out. is you know, my sort of uh hope and prediction for the future is that we'll see more and more interesting new agent-oriented programming languages. And one of the most fascinating ones that I've seen so far

[1:47:07] is this one called Bend, b e n d, Bend. Um and that language is one where it kind of marries programming with proofs, right? There There to be a time, you know, where you actually had to write your proofs in a different language. And proofs, by the way, for those uh who who aren't familiar, is this idea of uh that you can sort of formally verify that some code is correct

[1:47:34] mathematically. Right? Especially if you've written your code in a very functional programming way. Uh but for the longest time you had to do that in a separate language like Lean or TLA+ or uh I'm I'm I'm blanking on some of the other other examples, but uh yeah, like languages like that where you would construct the mathematical proof and then use a solver essentially to to check, all right, that that you've covered all the cases, you don't have

[1:48:01] like a race condition or or whatever. So, yeah, I think trying to sum up the question, I think yeah, if you are in a position where you can figure out how your agents can truly verify the work in a way that gives you confidence, you can actually, you know, uh have the PRs merged, cuz if it compiles, right, if it if the proofs show you that it's correct, then why wouldn't you just

[1:48:26] merge it? Um [snorts] but of course, yeah, not all the means are verifiable. Yeah, it's a tough one. Um okay, I think we've got to think about wrapping up because we are nearly on the hour. Have you Have you got something after this? I mean, I've got something after this. I've got my son dinner, but

[1:48:45] I I can go a bit longer, up to you. Okay, let's let's go 5 minutes longer then. Um I think I just want to have one more question, which is I think I want to ask how you see P stack and how you see skills in general, like in terms of we talked about this before we went on air, which is like people think of it's like my skills versus your skills, and

[1:49:09] how do you combine frameworks together? How do you use P stack with my stuff? Like what should you take from each one? And cuz I think I see skills as sort of just derived from process, basically. Like, they're just processes turned into words, and I would love to know how you recommend people take P-STACK and take my stuff as well, and turn it

[1:49:33] into their own processes. [snorts] I think you shared a tip, actually, that I think that I thought was actually very relevant, which is this idea that you go off and look at your previous transcripts, right? And you [snorts] sort of mine for information of, you know, your own look through your own your own prompts, right? To the agents where you correct them, where you have to constantly intervene, and uh, you know, take that higher-level learning and turn that into a a reusable skill, right? So, that

[1:50:01] agents stop repeating that mistake. I think that uh, P-STACK and your skills are very complementary. I I totally agree with you that they're like a skill is really much just process. I mean, it's just at the end of the day, a skill is just English or or language. It's just markdown. Um, and I think you can you can definitely weave them, combine them in a

[1:50:23] way that makes sense to you. But, I do think that uh, everyone should have their own set of knives. I I I keep going back to the the the cooking analogy, but it's so apt because like, you know, every chef when they go to a different job, right? When they go to a different restaurant, they carry they bring their knives with

[1:50:42] them. The tools go with them. Right? And so, trust to me is really about trusting your own tools. And when you spend the time sharpening them and understanding them really, really well, you can do great things. And everybody's skills and tool set is going to look different. You know, someone might find a lot of success combining, you know, like your grill me with docs, uh, or wayfinder skill with some of the execution skills in P-STACK, as an example. Some people might use more of your skills. Some people might use more of my skills. I think that at the end of the day, it really just comes back to

[1:51:14] how much do you trust, you know, me and Matt, right? Like if you if you trust us both, of course, use our skills. But I also encourage you to, you know, look at your own transcripts. Um Tell the agent to look through, you know, some of the all of the patterns that you've used, the the times you've had to intervene, you know, suggest turning them into lit rules or new

[1:51:35] skills, right? The the past chance, I I often say is like a a treasure trove of context because that, you know, it's it's like the process materialized, right? Like it's the real process, it's not an abstract idea in your head, it's the actual thing, right? And you can actually see how it happened in practice and extract so much information from that. And there's so much so that I actually turn I have a skill in piece that called recall, which is exactly that, where this was a pattern where you know, I was working in I was working on a similar problem. So specifically, I was working on virtualization for the

[1:52:10] cursor application and there were a lot of bugs. And so, you know, every time I started a new chat, I was like, "Oh, this there's so much good context from the last one though. You know, I want to bring it over to the new chat. How do I do that?" And that's where the transcript you know, looking at the past transcript came about and then recall was just a way for me to collapse collapse and compress that workflow into a skill so that I didn't have to just say I don't have to write a long essay every

[1:52:35] time, go look at all these chats, right? And you know, the blah blah blah. So I I largely think of skills, especially as agents get more capable, as really encoding workflows. You know, skills from last year were really more about like almost like implementation details, like here are the exact script commands, you know, you should use, right? I think with the latest models, you can just delete those parts and just really focus on the workflow, right? And tell it it's more like it the skill

[1:53:01] becomes more like a series of steps. A series the a real process. And I think over time we'll see that skills get smaller and smaller, you know, more compact. And yeah, they're very compatible. Or you can you know, if you want, why not read our skills, right? And come and combine them in your of your own, right? Combine Wayfinder with potato mode and make your own custom mode, right? Like like skills are The thing I love about skills are that

[1:53:30] is are that they're so malleable. You can do anything you want. It's just language. Absolutely. There's nothing magical in them, right? They're just words. And if there if there is any magic in them, it's just the words chosen and the phrases used and the thinking that's been done to turn those like take abstract process and turn them into language. And once that thinking has been done, then it's just there. It's available. It's on the

[1:53:54] surface. You just nick it. Um Lauren, thank you so much. This has been glorious. Yeah, this has been super fun. I really enjoyed that talking to you. Hopefully we can do it again. I'd love to do it again. I'd love to do it again, absolutely. Um yeah, we'll check in in uh I don't know if once through. Yeah.

[1:54:12] Yeah, Monday. Let's do it. [laughter] Let's do it. Part two. Well, thanks so much. I'm going to close the stream here. Lauren and I will chat a little bit and stay here. But thank you guys so much for watching. It was glorious. Everyone, I am Lauren Lauren Tan. I guess not many people know my last name.

[1:54:28] Uh I am potato on Twitter. Uh potato with spelled with an e. Um and I have been at Cursor for about 5 months. Uh previously I was at Meta where I worked on the React team. Specifically working on the React compiler which was a whole lot of fun. Uh I'm still on the on the core team and and contributing to open source here and

[1:54:52] there. So that that's really nice that they still let me do that. Uh and before Meta I was at Netflix where I was both a tech lead and I transitioned to be be an engineering manager for about 2 years. So, I've had a I've had a lot of experience going between engineering management and being an individual

[1:55:16] contributor. And I think something I've noticed actually which is quite interesting is that there are so many parallels with, you know, management skills and how to like manage agents. And that's actually a big part about what I wanted to chat with you and everybody else about today. But yeah, that's that's me. I do have some like very light slides, but it's not going to be just rambling. So, let me just share my

[1:55:42] screen. And hope that I don't leak anything. Uh Oh, no. I need to allow permissions. No worries. Oh, good. Thank you. Take your time. There always tech tech tech trouble. Give me 1 second to rejoin. Yeah, go for it. I see many of you already know Lauren from the looks of the chat here. So, yeah, it's exciting to to get a chance to chat with her and and go through some of her her recent work. As you guys heard, you know, a lot of recent experience from from Netflix to

[1:56:18] to Meta and then now over at Cursor. We're going to chat a little bit about Grokbot as well. So, that'll be exciting. I don't know if you guys saw that was a recent release. I think literally maybe yesterday or the day before from the Cursor team which is kind of like let's call it like agents for everyone. You can go check it out if you want and learn a little bit a little bit more about the product. But yeah, we'll we'll explore that a little

[1:56:39] bit today as well. All righty, welcome back.

[1:56:55] And Lauren, you're just on mute there if you want hop off of mute if you're chatting. Yeah, sorry. No worries. It's 2026 and I still don't know how to use Zoom. [laughter] Uh okay, so I assume you can see my screen. Yes, yeah, we're good. Um So yeah, today yeah, I think I think the big theme for me as I've been using agents to write code, and I'm sure a lot of you have had the same experience as well, is how do you trust it? You know, especially if you are an engineer that's been writing code for a very long time, you have a lot of opinions and lessons that

[1:57:29] you've learned about doing good engineering. And when you see agents just, you know, winging it and, you know, guessing, hallucinating, uh you know, confidently stating that they found the smoking gun uh for the hundredth time, uh but it's actually not the real problem, you lose a lot of trust. And when you lose when you don't have much trust in your agents, I feel like you you really can't get the

[1:57:53] most out of them. And for me, the parallel is like with management. Uh so if I'm an a manager an engineering manager of a team, and I have a bunch of, you know, I have a team of engineers uh on my team, and I don't trust them, then the mode of operation I'm going to be in is going to be like micromanagement, right? I'll have to spend a lot of time looking over my reports' shoulders and checking that they're doing their work well, you know, that they're not shipping bugs to

[1:58:21] production. And so, I drew this chart because uh it's not it's not a very scientific chart, but like this is how I imagine myself and my journey through using agents. So, you know, like fast forward or back forward, uh or fast back, uh fast backwards like a year or so, when, you know, nobody was or not many people were using agents to code, uh I think you uh uh, you know, into this mode where you are in very heavily in the loop with one or

[1:58:57] several, like a handful of agents. And you find yourself just constantly uh, you know, trying to understand what your agents are doing. Uh, and you're very, very in the loop. You're watching every single output. You are sitting there prompting. Um, and you really can't parallelize beyond that because you don't again, you don't have that trust. All right, you can't go to 100 agents, uh, like spawn 100 agents when you don't

[1:59:21] even trust the output of one agent. So, over the past 5 months, I feel like I've really been able to, uh, like ascend this trust curve. And now I'm at the point where, uh, I actually have This sounds kind of scary to say this and it it makes me sound like a slot artist, but I I

[1:59:40] promise I'm not. But I actually have my agents now, um, auto merging PRs for me. Uh, which is like a wild thing to say, but, um, I've I woke up today and there were like 20 PRs landed and I just reviewed them on main, like they were already landed. And they were good. Uh, so how did I get to that point? It's basically what I wanted to talk about

[2:00:02] today. Uh, and again, like yeah, feel free to jump in if you have questions, Colin. Um, but, uh, Oh, yeah, of course I got to show this this chart. Uh, where, uh, Don't don't do that. Trust to Wait, someone requested my computer. Uh, probably won't do that. Uh, but, yeah, so this chart I think I I I'm sharing this chart not to kind of like flex, but to kind of show like the journey. Like so you can see like the curve, like it's sort of like inversely matches the contributions I've

[2:00:38] been able to land at Cursor. So, I joined 5 months ago. And 5 months ago, like I you know, my first month I was like not very productive cuz I was you know, I was learning the code base, didn't know what the heck was going on. And as I got more confident in in my agents, I've really been able to kind of ramp up my productivity. Uh and again like yeah, like last month I shipped a thousand PRs, which is ridiculous. Uh and then this month, we're only on the 12th, I'm already at like almost 800 PRs

[2:01:10] landed. Uh so the velocity is definitely high and you I I I share a lot of you will definitely be questioning like how how much of this code is actually good. Um and I think yeah, like that's definitely fair to question. Um but uh yeah, I think I think if you set up your agents well, you can

[2:01:32] definitely get to a very similar level. Um and so I'm going to talk about how we do that. Uh so for me, I think I'm curious like I guess calling your experience as well, but uh for me, I think the most important skill that you should have in your toolbox when you work with agents

[2:01:53] is verification. Uh and by verification, I mean the ability for an agent to actually run the code uh or take CPU traces or heap snapshots or uh you know, open an iOS simulator, whatever, you know, however your application is exposed to your users, it can do the same thing and uh run it for real and actually test and verify it

[2:02:20] don't work. Cuz that's the thing that really closes the loop. Uh it doesn't guarantee your agent writes good code, uh but it allows them to at least write correct code, uh which is a big a really big step forward for being able to trust your agents. Um I will I can share one example that we have uh within cursor uh oops

[2:02:48] where let me open this.

[2:02:57] Let's make this bigger. Let me full screen. There you go. Uh so for the for cursor's agent window, uh so this is actually an interesting story. But uh when I joined cursor 5 months ago uh they're actually uh well, I was supposed to join a different team. I was supposed to join like the cloud agent team. Uh but then since I have a lot of experience working on React and agent's window is a React application. Uh I was uh I was asked to basically help out with uh the agent window work. Uh but um there wasn't really a lot of like skills to help me. So I just told myself that

[2:03:40] okay, uh agent's window is going to launch in like a week. All right, we have a really tight deadline. And um there was uh you know, I was just sitting there like okay, I'm going to open up the performance the the Chrome DevTools and just like take a trace, look at it myself and try to make sense of this flame graph. And keep in mind, I was just like in my first week. I had no idea what I was looking at. No idea what you know, I mean, I had some idea, but you know, the the code base was

[2:04:05] completely fresh to me. Uh and I realized like my agent had no idea either. You know, like I would take a screenshot of the child download trace and send it to it and be like, yeah, it kind of looks like this, you know, uh and it would like confidently state like it's this thing and then I try to fix that and turns out that's not the

[2:04:22] actual thing. So this was very very slow process. And if you've ever done any like performance work yourself or you know, just even development with an agent where you don't have a verification skill you are the verifier. All right, you you're the bottleneck. You you you tell your agent to do something and then it goes off and writes some code. Then you open up your, you know, local dev build and then you start to say, "Oh, you know, it doesn't work." Then you have to copy-paste screen uh you know, screenshots or console errors or whatever. Uh and then your agent like slowly kind of like uh you know, works

[2:04:55] with that and then tries to understand it and um uh fix the thing. But then you're constantly just in the loop and and being a bottleneck. So, there's really no way to parallelize. So, the control glass goes like one of the first skills I built uh for Cursor. Uh and glass, by the way, is the code name for agent's window that we use internally, but it's just

[2:05:17] Cursor, I guess. Um and so this skill uh is I guess that that the code itself is not super interesting. Your agent can very easily make one for you. Uh where if if you're building an Electron app or a web app or even iOS uh applications, uh you can teach your agent how to use like the Chrome dev tools protocol or through uh Apple has some utilities as well for running the simulator and taking traces and controlling programmatic control

[2:05:47] as well. Uh so, that's really useful. Uh but one thing I want actually want to talk about is uh the this thing. Uh where is the read me? Uh so, this skill comes with this very unique feature called or not feature uh unique file called a feature map. And so, the story then is like I built this skill and so now the agent was able to uh actually run the agent window and

[2:06:17] take traces and whatnot. Uh but it had no idea what what the agent's window was. So, um you know, like someone would say like, "Oh, the the left side bar is like laggy or something like that." Or, you know, the right side the the PR tab is not working. And the agent would just be like kind of flailing around. It would spend a lot of time trying to like look up the code and, you know, where is this feature? How do I actually get to it on

[2:06:42] the UI? Which made it basically completely useless. Uh, you know, like we would I would run this go locally. And, you know, it would spawn a dev build. Uh, but then it just be churning like I just try to click here. It wouldn't know how to get to things. Um, and it was just an awful experience. Uh, so we were just putting arrows on my screen. Um, so, uh, yeah, this this feature map has been

[2:07:09] really useful. Uh, because it teaches the agent how to get to all of the features that you have. Um, and in Pstack, the plugin that I I've made, uh, if you search for Pstack cursor on Google, you'll find it. Uh, but there is a creative verification skill in that plugin where it actually helps you set up something like this for

[2:07:30] yourself. Um, including the feature map. So, it will actually explore the code and build up this initial feature map that tells your agent how to get to all of the different features that you have. Uh, and this is extremely powerful because now that you have these user reports that come in, uh, you you can actually map even like a

[2:07:49] vague report or even a screenshot. So, we have this, uh, internally at Cursor where, uh, we have a Slack channel with, you know, lots of people giving us feedback on the agent's window and rock bar and whatnot. Uh, and often times the report is very bad. Like very low quality. Like someone will just put Very often we get like a screenshot like and then someone just says, "Question mark, question mark, question mark. Like

[2:08:14] what is this?" [laughter] And, you know, like that it would without this, your agent doesn't I have no clue. Right? But with a feature map like this, it has a lot more context and understanding of how to actually navigate, how to get to all of the different features. Uh so like you know example like I guess like the sidebar like what is the sidebar? Uh you know like all the different sub features that are present in it. Um like from the user point of view here's where that how to get to it all the different keyboard

[2:08:43] shortcuts. Uh even like the the what do you call it? The DOM elements or yeah like the attributes that you use for selecting things through the CDP are all there. So uh again yeah this is like really really powerful uh for for agents. And uh and a Pystack ships uh that create verification skill but also a maintain

[2:09:06] verification skill. Uh so you can keep this up to date. Cool. Yeah I was just going to ask how you created that. So do you mind sharing a little bit more about um that that process in the context of Pystack and maybe just what Pystack is for the folks who aren't familiar. Yeah so Pystack is pretty interesting because uh well first of all the name is kind of goofy. Like the P the P in Pystack is

[2:09:28] like potato potato sack. [laughter] Um because I um so uh the uh there's a pretty uh famous person Garry Tan who is the CEO of Y Combinator and he's come up with this plugin called G stack uh Garry stack and uh funnily enough we share the last name we have no relation. Uh but I thought it'd be funny to kind of you know poke fun at Garry and make Pystack my version of of of of his plugin uh but kind of just tailor it to my own

[2:10:01] set of prac engineering practices. Uh but I honestly actually never set out to build Pystack. Uh it just started with a bunch of skills. Right like I started with that control glass skill and then I started with another skill like called how which I also noticed through like observing agents. Um so like you know in the early days of me you know, trying to climb this ladder, I was like super in the loop and I was basically nitpicking my agents to an extreme degree. I was uh like I would tell it um "You know, this feature has stopped working. Here's a bug report. Like, why

[2:10:35] isn't it working?" And very often, the agent would just like confidently state like, "Oh, it has to be this, right? It has to be this thing." And I noticed like when I looked at the actual tool calls, I noticed it wasn't actually reading the code that I thought should be affected. And that made me just extremely suspicious. And at that point I was like, "I'm not going to I can't trust any this agent anymore cuz it's just

[2:11:00] it's just completely hallucinating." And I think I think it's very easy to just, you know, like build up that distrust and not and kind of feel helpless. Like, you know, you don't know how to help your agents succeed. But like again, I think the the the management analogy is super helpful because like imagine if you were a manager of an engineering engineering team and you had an engineer in your team who was a really good coder, no business contacts whatsoever. You know, they they just you just hired them and they they onboarded, you know, like 5

[2:11:31] seconds ago. Uh and so how do you actually teach that person to be effective? So how you do that is through skill. Uh skill being just, you know, it's just markdown, right? But you know, it encodes a lot of information, instructions, a lot of uh you can really draw out a lot of intelligence from an agent by Well, some people on Twitter call it like, you know, pull the agent to a different latent space, which is kind of like a fancy way of just saying like since uh you know, LLMs are sort of like they

[2:12:01] predict the next token. Uh when you give it some high-quality tokens uh to begin with, then you know, it it can kind of pattern match on like a higher space that's, you know, smarter. Um so that's like a very interesting model there. But yeah, I built P stack very very incrementally. Uh so, uh started with just really observing how agents in all the fail different failure modes of of that agents were having. And every time I saw that, I just okay, I'm just going to make that a skill. All right, like stop hallucinating, actually go and search up look up the code, use a lot of sub

[2:12:37] agents, uh and yeah, stop guessing. Yeah, that makes sense. One one kind of follow-up question here, uh both for myself and for a bunch of people in the chat. So, I guess it's two two parts. So, one is like how do you maintain these skills? So, like the product changes over time. Obviously, there's a lot of people who are shipping against the code base. So, how do these skills get maintained? Uh and then second to that is like how do you know when your verification is is

[2:13:02] good enough? Uh like in you know, you can trust that the very verification loops that you've built are going to I guess you trust that the outputs are what when they're done. Uh yeah, maybe I'll talk about um I think there's some more really Maybe I'll start with this one first. So, like how do I

[2:13:21] maintain these skills? So, um if you're not familiar with this concept, an eval [clears throat] is essentially like a way to uh well, I think the mental model I have is like it's like a unit test for an agent. Um and uh you can actually make your own evals. You don't need like a special framework for them. You can build you can you can build one depending on like you know, how scientific and how

[2:13:46] rigorous you want to be. Uh my screen is red. Yeah, there's a little button. Um sorry. Are you mind like disabling the drawing or something? Like I can't see my screen. Yeah, sorry. If you guys could not draw on the screen, that'd be great, but um there's a little button in the troll. Yeah, the the little drop down. Uh

[2:14:06] how do I clear? Yeah. Okay, yeah. You got it. Perfect. Uh continue. Yeah, so evals are a way to unit test your skills, basically. And actually in P stack, we ship uh under potato mode, there's a playbook if you search for it called eval playbook. Um and it's uh uh it's like not It's actually pretty pretty rigorous, the way it's done. Uh but um essentially, what I do is I spawn a lot of different sub agents. I have like my main coordinator agent uh come up with a rubric for uh

[2:14:43] what I want the skill to do. Um and then it spawns all these sub agents, and it it creates individual directories for them, uh which are cleverly named to not let the sub agent know that it's being evaluated, because uh agents can actually tell, and when they do, they change their behavior. Uh but it does a bunch of stuff like that to um essentially, yeah, like test whether or not the skill I'm making or changing is

[2:15:12] actually doing what I think it does. Um and one of the really nice things about Cursor is that we are we we support so many different models. So, you can actually eval your skill across all sorts of different models, um and, you know, get a sense of how well it performs across that different matrix, um especially for the models that you

[2:15:32] use. Uh so, I do this a lot. Every time I I modify a skill, I will run one of these, uh like the eval playbook, uh and make sure that, you know, it's actually leading to a result I want. Uh but I will say like maintaining skills is actually pretty hard. Uh it requires, I think, a lot of taste and observation. So, you kind of need to be very good at being a backseat driver. You know what I mean? Like, if you do pair If you've ever done pair programming, for example, uh and you watch a coworker code, and you're just like, you you could probably

[2:16:07] do this better, you know, you could do it you know you like why did you not do this, right? You You ask a lot of questions to your coworker. And it's kind of a similar thing here. You like you don't want to just be a passive observer of agent. You want to be very in the driver's seat in the initial stages when you're building up your own

[2:16:21] set of skills. Uh you know, obviously you can use something like P set, but if you're building your own set of skills, it's very I think you know, opening up the all the tool calls and like reading the code and reading all the uh the agent behavior and their thinking blocks is uh a really great way to see where they they fail, right? Like what what you know, where are they being done? And then you can go and build a skill for

[2:16:46] that. And then with verification, how you trust it is it's I think it's also a very similar iteration loop uh where, you know, like I actually did the same process for verifying the verification skill where I actually get um So, one thing that's interesting about evals is that you can sort of hill climb them, meaning that uh your eval can produce a score, right? Uh a score that you can get your coordinator to produce, uh but also uh you can have a judge agent of a different model to uh kind of cross-reference and make sure that the

[2:17:22] first model is not being biased, right? The model that's judging all of the sub agents that are running the thing. Uh but you can also like hill climb. So, meaning that you can you can use like {slash} loop in cursor and you can say, "Okay, keep looping on this eval, right? Until everything is 10 out of 10." as an

[2:17:39] example. Uh and I did the same the basically the same approach with the control skill. And so, I kind of it was very it was very hands-off actually. Uh so, you know, I uh I kind of built I built that skill that way, like the CLI in that skill. Um and over time it's gotten really good. Uh but yeah, it was definitely not super smooth at the beginning. It required a lot of iteration. And I think there's an analogy here for me, which is um Well, I make this analogy later in a different slide on my drawing here. Uh but I think of it like uh you know, as a as a

[2:18:16] engineer now, you're sort of more [snorts] like you uh like maybe a manager or the analogy I like is like you're like a a chef in a restaurant. Uh you know, you're the head chef. Uh you're not cooking all the food yourself anymore. You have a team of cooks, right? You have a line cooks, you have a sous chef, you have you have all these

[2:18:34] different stations. Um and it's your job to really design the environment. You know, you you you're in charge of setting up the kitchen. You're in charge of, you know, like giving tasks to different people. So um yeah, it's a very interesting way of working. Uh but yeah, that's that's how I basically built uh these verification

[2:18:57] skills. Yeah, just just one follow up there and like I'll go try to go one layer deeper. So, are you let's say we wanted to build um a an eval or a skill for for something and we wanted to kind of get better on its own, which is is what what I think you're suggesting. Uh are you doing that in like a work tree? Like kind of isolated with like the sub agents and and then the reviewer agents and and all that? Is it happening like in some type of cloud hosted environment? Like what's the the more the practical steps? If I wanted to go do this uh and like set up a

[2:19:26] verification system for something, what would I what would I do or where would I start? Um I think that uh the best way to start is local because you can observe. You can definitely observe what your agents are doing. So, uh if you're building a verification skill for yourself, uh I would definitely start local and just have your agent bring up the application, whether it's like a CLI or a desktop app or whatever. And so you can actually observe, right? You can see how the agent is interacting with the the application. You can see it, you know, how it calls like the different APIs that that allow it to interact with

[2:20:04] the uh the application. Um but uh for me personally, uh I have basically been kind of all in mostly all in on cloud agents because they're extremely powerful. Uh and the really powerful thing about cursor is the the cloud agents actually. Where if you spend a little bit of time setting up your environment, these control skills, these verification skills pay a huge amount of dividends because it's not just something that

[2:20:32] makes you as a single engineer better. It actually levels up your whole team. Uh and even your whole company because uh you can actually start thinking about cloud agents and thinking about automations that automatically do things like uh I'll I get to I I kind of talk about this a bit later, but I'll just kind of get into it. Uh where where you know, for example, like I talk a lot about

[2:20:56] this agent we have called Benny, right? Who who uh you know, takes all of the bug reports that we get and it automatically goes off in the cloud, opens up a cloud uh it's, you know, it's desktop. It runs cursor in its own computer and it uses the same control skills to interact with the application and try to reproduce the bug uh or the user report. All right, and this is so so powerful because at once I can immediately I I get so much

[2:21:24] information from this automatically. Like here in this example, you can see that uh the Benny actually reproduced the bug. Uh but it's already fixed on main. So, it actually confirms that we fixed this problem already. And all I need to do is just release another build of of cursor. Uh so, that's like huge information there that I didn't have to go off and sit with an agent, you know, and spend an hour trying to figure like, is this

[2:21:48] fixed? Is this not fixed? So, you you you gain like so much time, Uh but you know, everybody on my team benefits from this. Everybody at the company benefits from this. Uh so, definitely think that, you know, keeping these using cloud agents is super powerful. Uh but yeah, it's like a journey. You have to trust it first,

[2:22:08] right? Before you you get to this point. And that's it goes back to what I was saying here, where, you know, it's very hard it's it's almost impossible and I would definitely encourage you not to try to jump from, you know, like if you're still in this zone, you don't want to jump to like I'm going to spawn a hundred or a thousand of cloud agents right now, because you're just going to waste a lot of tokens and it's going to be extremely

[2:22:32] expensive. Yeah, so just to kind of recap so far, basically the if you wanted to go on the journey that you've kind of gone on, it would be just start with verification, building some some skills and some ways of determining that the agents are producing at least like correct code, whether like you said, whether it's good code or not is maybe a separate question, but like it's it's technically solving the problem by looking at, you know, stack traces, looking at, you know, the the actual behavior in the app

[2:22:56] and so on. And once we trust it locally, then we can start to think about scaling into the cloud and running more agents that are picking up signals, I guess, on their own, right? So, whether that's like a bug report that comes in or something, they can go and pick it up and solve the problem and and give us back a PR. And then maybe the last step is like auto-merging the PRs, which is

[2:23:15] where you're at. Maybe not where everyone is at. Yeah, yeah. And then reviewing them on main, but is that is that about right? Yeah, exactly. I think yeah, that's why I drew this this uh this this curve, right? Because that this this basically describes my journey of, you know, when I started barely could use a couple of agents and I was just observing every

[2:23:33] single thing. I think there's really no shortcut for going from here to there, because this is really about your personal level of trust in the agents, right? Um obviously, you know, as a as engineer, you don't want to just lock code into production. So, how do you actually build out that trust? Takes um a lot of uh I guess taste and judgment. Um but uh you know, like I think plugins like Pstack definitely kind of help you

[2:23:59] uh get up to speed much quicker. Uh and so, I guess it's like if you trust me and you trust Pstack, then in by extension, you can maybe trust your agents. But if you don't trust me, and I definitely would not encourage people to blindly trust me. Uh uh you know, if you build up your own set of skills that you can obviously, you know, take a look at Pstack and kind of fork it, make it your own, improve

[2:24:24] the skills. Definitely encourage that. Um but for me, it's really all about it just keeps coming back to trust. You know, every one of us here in this chat have a different standard for engineering. Uh and there are different things that are important for us in our code base. And uh when you are able to encode all of that into skills and you can verify that your agents actually doing them, that allows you to really kind of ascend this curve. And um uh you know, you start automating

[2:24:52] things. Uh there's another piece I wanted to talk about. Um if there's Yeah, go for it. I'll I'll think of more questions as I go, but yeah. Yeah, I think there's a a third part to this which I haven't talked about yet, which is kind of an interesting one, which is like refactoring and rewriting. Like one of the uh I guess most controversial one of the most controversial topics in the industry, I think, is like should you rewrite your app or not? Um because I think engineers are very prone to this where, especially when you join a company, you come in and you see like the code base and you're like, "Man,

[2:25:25] this is Like who wrote this code? You know, it's it's terrible. I want to rewrite the whole thing." That is a very common inclination. And I think a lot of you know, before agents, um and I guess arguably even now, people will definitely discourage you from rewriting stuff. But I'm actually here to make a case for

[2:25:43] why you might want to consider it. Um because I think it really depends. Uh you know, brownfield applications I think are actually in a pretty good spot, especially if they're set up well already. Uh and like recently I've been talking to some people, but uh you know, I I I was just observing. I I just noticed this parallel, which is that a lot of big tech company problems are now

[2:26:11] everybody's problems. Um and the big tech company problem, you know, like when I was working at Meta, like we had this giant mono repo. We had like, I don't know, tens of thousands of engineers just, you know, like banging on their keyboards and and shipping code. And a lot of really great engineers at Meta, uh but uh I'll say like, you know, you'd be surprised that the code quality is

[2:26:33] actually not that good. Um and so I often joke that like, you know, before AI slop, we had human slop. Um and so uh you know, I think a lot of big tech infra, like uh like what Meta has or Google, you know, you know, really big tech companies, are actually designed for that, where you're you're sort of like you're catering to the the, you know, like uh this sounds so bad to say, but like the the least capable engineer in your team, right? You build you build frameworks, you build conventions, you build guardrails, you know, you restrict credentials so that, you know, your intern doesn't wipe your production

[2:27:07] database. Um there's uh you know, if you have that level of infra already, I think your agents can actually already do a very solid job, right? Because they have the the guardrails are already in place for agents to not cause havoc or not cause too much havoc uh in your code base. Um and you can always add more, you know,

[2:27:30] guardrails. Uh but I think like greenfield applications especially are, you know, like the brand new applications are like the biggest risk in my opinion Uh and also the greatest opportunity. Because you know, if you vibe code a project a prototype like we did for GrokBot, you know, GrokBot was done up very very very quickly. And if you if you haven't heard of of GrokBot, it's like our a new

[2:27:53] application we just launched yesterday. It's it's really cool. Let's you orchestrate your create like individual agents that have their own identity and you can kind of orchestrate them. It's super cool. You got to check it out. But yeah, that was it's like a very it was a very green field application like most prototypes are.

[2:28:12] So it's like vibe coded very quickly. Humans were not reading the code at all. And I had this tweet recently where I said something about organic architecture. Um Maybe I'll find it. But the idea is that when you have a completely vibe coded application, you essentially have no guardrails whatsoever. So your agents when you give them a task, they will just solve it in whatever method is the

[2:28:43] most convenient. And over time you get into this situation where you have a code base that is spiraling out of control because you don't understand it. Your agents understand it I guess in a way, but like they've built something that is you know, optimized for short for shortcuts. And you know, it will you will suffer have a lot of of issues with that

[2:29:06] application. So I think starting your code base with like very strong constraints is very much needed. Because like when you have a code base that you can trust, right? When you have guardrails that actually help you um help your agents write good code, you can get into the you know, like into this part of the curve where I I where I like I I said, you know, I woke up today and I had like 20 PRs merged uh by my agents. And that's because I invested a lot a lot a lot of time uh over 600 PRs I I I calculated yesterday uh when I refactored all of GraphQL to this

[2:29:48] new architecture that I've been building. Um and yeah, I've gotten to a point where I I don't really look I really don't look at the code anymore. And um I say that not just, you know, to sell you tokens, but because I you know, it it it took a lot of work to get to that point. I spent a lot of tokens to get the code base to this point where I no longer have to

[2:30:10] look at it. Uh but I'm very excited because you know, at the potential where, you know, it's not just it this doesn't just benefit me. It benefits everyone contributing to GraphQL. And it also empowers, you know, designers and product managers and, you know, even GTM people to add features to GraphQL. And I don't have to worry, you know, I don't have to to wake up at night in in the middle of the night and worry like, oh someone's just merged a perf regression. Right? I have a ton of constraints and CIs like it's actually very annoying to write code in in GraphQL but like agents absorb all of

[2:30:45] that annoyance. Um but yeah, I'm happy to talk about what exactly that is. Um Yeah, I think one question um Yeah. before we get into the this part here is just around that element of like what your your your CI looks like, maybe some of the constraints, and then also like the average PR size. I saw a question about that earlier. Just to give people a you know, kind of a a glance. It doesn't have to be like mathematically average, but just a you know like what generally the size of the a PR is. Um if it's only a couple lines of code or, you

[2:31:15] know, um yeah. Uh um I think it depends. Uh let me I'm trying to do this in a way where I'm not going to like You don't Yeah, you don't have to share the actual number. If you like the actual number, it's just I think this is fine. benchmark But like we have So, okay. This is not that interesting, but well, fun fact is that virtualization in Gockle and in cursor is actually powered by pre-text, which is a sort of new library that someone's built. Um that's really interesting. You should You should check

[2:31:50] it out. But that's not really that important. Uh I think the average PR size I actually don't know I I don't know if I want to click on these. I probably can, but I would say like they can range anywhere from a few hundred lines or 50 lines to like a thousand depending on what the thing is

[2:32:08] doing. Uh so like here I'm actually like deleting a bunch of files, so I expect that it's just this like mostly deletion. Uh but yeah, it it kind of varies. There's no like Yeah. There's no like hard cap or are they all like 50 line PRs? cap. There's definitely no hard cap, but I do encourage my agents to split up

[2:32:26] their work into multiple PRs. Uh I do that mostly because uh I like I like the idea of I guess maybe this is much harder to do now as in the world of agents and you have like so many commits. But I like the idea that you know, the Git history is a very rich source of context. Uh and I like the I like each PR to sort of atomically describe what that

[2:32:50] small piece of thing is doing. Which also makes it easier for me to revert changes and like figure out, you know, oh I shipped a bug and it's this it's here. It's not in this 40,000 line PR where you who knows what landed in there. Uh but yeah, I don't have a hard cap on

[2:33:07] PR size. Cool. And then um yeah, also quick question on like CI. So again, you don't have to go into like uh the screen share of like your your CI does, but just generally would you describe what the CI kind of looks like uh or how strict it is. Uh yeah, so uh well, specifically for Grokbot, so Dune is the is the sort of cheeky code code name for the architecture that we built for Grokbot. Um the CI looks pretty annoying

[2:33:37] because there's checks for everything. So, like literally I have um uh well, if you've read any React for example, you know you know that one of the biggest foot guns in React is use effect. Uh so, in uh Dune and in Grokbot, we've banned use effect. So, Dune is this the the mental model of what Dune is uh you can kind of think of it as like Next.js for uh Electron apps, and it's designed for agents to write. Uh and it's like custom for, you know, our agent powered

[2:34:08] applications. Um so, the CI checks are very like specific to that. Like, you know, don't use use effect. It's it's it's banned. Like, CI will fail uh and yell at you. We have like some of the more interesting ones that people might raise eyebrows is like I actually banned code comments as well. Uh which is very interesting. Um but I've noticed that 99% of the time agents just write code comments that kind of describe some historical thing that is

[2:34:36] actually totally irrelevant to the code. Um like it will often say like, you know, "Oh, Lauren said you should never do this." and it's now in the in the code comment. I'm like, "What? Like, why what that that was I didn't say that as like a durable, you know, global rule. I just meant like your this PR sucks and

[2:34:52] you should change that part." Uh agents don't really understand this that well, surprisingly. Uh and or they kind of assume too much and they kind of do things in like very stupid ways. So, like yeah, we just ban everything everything you can imagine like the agents are bad at, we ban. Uh so, one example that we actually suffer a lot in the agents window is we have uh you know, you know if you use the agents' window you've definitely seen performance issues and you know

[2:35:21] we're constantly trying to fix them. Uh but it's like a it's a never-ending struggle because there's so many pull requests that get merged. Every any one of them could just regress performance or stability or reliability. Uh you know the agents' window doesn't have this architecture yet. I plan to do bring this learning back there and kind

[2:35:40] of refactor everything there. Uh but uh it just regresses super often uh because uh there's just one example is like we have very poor um isolation between processes. So like on you know on on Electron you have a renderer thread that renders your UI. But you also have like a main thread that you can run other code that you

[2:36:01] know doesn't need to block the renderer. Um but we do a poor job of separating those things. And so often times you just accidentally have code that gets pulled into running on the renderer thread. And then all of a sudden you're competing with the the renderer that you know that has a very If you want like 60 FPS you have to every frame that gets

[2:36:21] drawn has to be done in 16 milliseconds. So a very very small you know deadline per frame uh if you want you know a very smooth product. Uh and when you start building bringing in accidentally bringing in you know things that are like very computationally heavy or they have a lot of IO uh then you just get into like a lot of jank. Uh your your FPS really drops. You start uh you know losing frames. You get long tasks that take more than 16 milliseconds and you

[2:36:49] just get this really choppy experience. So all of those patterns that we've learned basically building Electron apps we've encoded into this framework and it becomes like a hard failure. So I literally in in Grokbot we literally have a directory called Electron main Electron renderer and we have uh import uh CI I guess where we actually check the dependency graph to make sure you're not accidentally importing code from one

[2:37:16] directory to another. Uh so, that's enforced by CI. Um as well as Bugbot, uh which is our which cursor's um like code review tool that runs on CI. Uh you know, in our agents MD, it's everywhere. Like so, I I I I I um I have this thing here where I I talk about like um you know, like there are multiple layers, I think, for building a good

[2:37:41] code base. Uh obviously, the code base is one where uh if you have an architecture like this, where it's extremely strict, uh you know, the the the way to build features is very conventional, that's like the strongest strongest level of enforcement because agents just love to copy existing patterns. So, uh one example of this in Bugbot is like we have this these concepts called like a feature, and we have entry points and transcript cards, like all you know, the

[2:38:08] cards that you see in the chat. These are all like like nouns, I guess, in in the framework, and so there's a very conventional way of creating them. And so, like a feature is all in a in a single directory as an example. And so, all of the code that contributes to that feature lives in one directory. So, it's all co-located in one place. Makes it super easy, you know, agents don't have to like uh grep around and try to figure

[2:38:32] out like where all the things are. It just looks at the feature and like, "Oh, okay, I'm working on the onboarding feature in Bugbot. Uh I'm just going to work in this directory." And for 80% of the work, it's mostly just very encapsulated there. [clears throat] But uh like it's like very it's like designed again for, you know, like the dumbest agent. Like you don't have to think, right? The the the one of the key principles I have for this framework is like the shortest the

[2:39:00] shortest path is the best path. So, uh because that plays exactly to how agents love to write code. Like they like to take shortcuts really. You know, they'll they'll find the quickest way to solve the problem. So, why not make that the best way to solve the problem? Uh so, I I I'm I probably won't get into all the specific details. Um and uh the this framework's really more of a collection of ideas and principles rather than something that we'll open

[2:39:28] source. Uh you can you can, you know, screenshot this, I guess, if you want and uh tell your agent to uh do some build build something like this for you, too. Um yeah, but it's really all about the layers. Uh you know, like the the code base is one part with features uh and directories and, you know, import uh blocking import dependencies uh that shouldn't be imported. Uh but and and it

[2:39:53] all enforces that and static analysis. So, like uh there's CI checks. We have a lot of lints for bad patterns that we observe. Uh compiler diagnostics. Uh there's also rules and bug bot, which are um I think like three, four, five are more soft, right? These two actually make make CI red, right? So, that you know, there's a hard constraint where the

[2:40:18] agent can't just write crappy code. For rules and skills and bug bot, your agents can still forget, right? You can still or it may not always consistently apply them. So, I like to layer them, but I don't I don't like to rely on them as the only source of enforcement because it's very very soft, right? And if you if you only have rules and bug bot and skills and a style guide for your code, you will it's only a matter of time before your code base looks like complete trash. Uh I'm sorry to say that, but uh I definitely recommend, yeah, like, you know, investing in, you know, things that can be hard

[2:40:57] enforced. All right, and this is why, you You maybe the choice of tech stack that you use is also very important. Um Like I think for example Rust is sort of making, you know, it's like getting super popular again. Uh because the compiler is so strict, right? The compiler enforces so many different things, you know, there's a borrow checker that you have to appease. And if as long as you make sure your agents don't write unsafe code blocks, uh you can more or less feel somewhat confident that if the code compiles, it

[2:41:27] probably works and it's good. Uh but it gives you it gives you that level of trust and confidence that you as a human engineer no longer need to go and check it yourself. You you know, you you rely on code and static analysis to actually make that uh a lot smoother. Um and I I guess the worst part the worst place to be in is if you are stuck in code review land where you actually enforce all of the constraints, the invariants in your code base by literally a human person saying, you know, reading the code and then okay,

[2:42:03] you should not do this. Right? Every time you have to do that, you should consider that as a code smell, like a anti-pattern. And you should say, okay, instead of me commenting on the PR, how do I turn this into a hard rule? All right, how do I turn this into a lint rule? How do I turn this into a CI failure? Or how do I even categorically eliminate this problem

[2:42:24] uh entirely? Uh I I can talk about another migration I've done, but I'll probably pause here. Sure. Yeah. Yeah, I feel like that's that's where I am, to be honest, is is what you're describing right now, which is that like I don't have all of these rules. Uh so I have some things to go do after this session in terms of being able to scale my agents. I'm I'm definitely on like the uh you know, maybe a couple of parallel ones locally stage, so like two to three locally. And I'm sure most people here are on the same, so uh yeah, I know we only have a couple minutes left. Uh

[2:42:53] Lauren, was there anything else that you wanted to to highlight? I obviously there's lots of questions, so I can grab more, but I want to give you a few minutes uh if there's anything else you want to talk about. Well, I think I've been yapping for quite a lot, so I'm maybe let's just do

[2:43:03] questions. Okay, cool. One question that had a couple of came up a couple of times was just around like token usage. So, the the question is like is is what you're describing a realistic thing for people who are on, you know, a normal set of token usage, they don't have, you know, obviously unlimited tokens

[2:43:23] to work with. Well, I think that's a really good point. I mean, like obviously, you know, I work at a AI lab where we have unlimited tokens. So, I definitely cannot say that, you know, this is something everyone should do in the exact same way that I did it. I think it's possible to get to this point without, you know, breaking

[2:43:42] the bank. But, you know, if you're like an engineering leader or, you know, you you have a startup that you lead, I think that means a question of ROI. Um and it's like uh yes, you spend a lot of money on tokens in the upfront stage. You know, like refactoring a code base is going to take

[2:44:01] a lot of tokens. Adding all these things is going to take a bunch of tokens. But, if we're heading to a world where agents are writing all the code, and you know, you want to be very lean, right? You don't want to have to hire You don't want to be You don't want to become like meta, right? Like I mean, like in terms of you don't want to become a 10,000 person engineering org because I mean, that's a cool problem to have, but also, you know, you you have so much overhead. There's like planning, you know, like it it's it's personally I I wouldn't It it's not super fun. But, um

[2:44:36] I think you want to stay very nimble, right? And you want to you want to be like agents are all about allowing you to do things that you couldn't do before. That's really to me like the value of agents. You know, it's not just throwing tokens on every single little thing, but um to me like the thing I couldn't do before is like enforce this level of constraints in a

[2:44:57] code base by myself. Right? Like, I'm just a single person, you know, uh, it would have taken me years to build this framework, uh, and do all the refactoring, and test everything myself, and verify, you know, like, run Imagine if there it was just me right now in in pre-agent era, just like running, you know, by my It would take me so long. All right, and my salary is pretty high, right? Like, so, you know, the the question I think an engineering leader might have is just then, you know, like, what is there's a trade-off of do you hire someone to do this, or do you spend the tokens to set

[2:45:33] up a code base so that even the the most naive, right, the dumbest agents can do a good job? And when you actually get to this point, like, even agents that are not, you know, Fable-size, do an excellent job of writing code, and this pays a lot of dividends as well for me and personally, where I have empowered not just myself, but again, like, PMs, designers, engineers who are not familiar with GrokBot to just contribute in a way that

[2:46:02] is sustainable. So, I think, yeah, it's definitely like a trade-off, uh, for sure, you know, like, nothing is like free, for sure. Uh, and tokens are pretty expensive. Uh, but oh, actually, uh, I I I don't know how many of you have seen this, but we actually announced Grok 4.6 today. So, very exciting, finally out. Um, so, yeah, Grok 4.6 would be like a great It's

[2:46:24] very, very smart. Uh, it's really good on the on the benchmarks. Uh, and it's the same the tokens uh, well, uh, I hopefully I'm not saying this thing correctly, but, uh, I believe the cost per token is the same as 4.5. So, you're actually getting more intelligence for the same cost. Uh, I think this is an area that Cursor tries to Cursor and SpaceX AI Try to really optimize for like that Pareto frontier of, you know, cost versus intelligence. Uh, you know, we don't necessarily want to build the biggest model ever because that is

[2:46:57] extremely expensive to run. It's really about like how do you find that sweet spot, right? You don't You don't need a giant model, but it's just super smart, right? And it's not very expensive for inference. Uh, but um yeah, I think to kind of round it up, um I think it's like a it's it's There's a If you do your own analysis, I feel like it's pretty positive. It It'll be pretty positive that the ROI you get from investing in stuff like this uh just empowers not just yourself, but your whole team to be so much more productive, right? Like imagine if you have an army of engineers like me who

[2:47:33] are shipping so much improvements and and bug fixes uh you know, every day, right? Like that is pretty exciting. Cool. Uh one last question before we wrap up. Now, this one is for the people in product on the on the call. So, let's say we do have an army of engineers who are shipping like Lauren. I'm just curious like how is the product team or other functions of your company keeping up given that like if you're shipping so quickly, how are they using AI more to do their jobs? Like as much as you can speak to that. I obviously you don't have like you're not in that role, but

[2:48:05] just curious about how that works. Um I think this is where GrokBot has been actually exceedingly powerful. Uh where So, before GrokBot like, you know, uh obviously Cursor only had Cursor. Like we only had agent window. We had a CLI. We had an IDE. And these are really like power user tools, right? Like the They're designed for developers. So, it's very, very developer-centric. You can do knowledge work in them, but it like the UI is not really optimized for

[2:48:32] that. So, we actually didn't really have uh well, I I think like a lot of people like, you know, GTM, product, like they might have used Cursor uh to do their work, but it definitely wasn't like a delightful experience for them. I think now with Grokbot it's become Grokbot is basically like their cursor moment for people who are not in tech. In my opinion, like it's like it's like a very very accessible way to use agents in a very comfortable, very familiar interface. It looks like iMessage. And it's very fun too, you know, you can give your agent a fun name. You can have you can kind of do

[2:49:11] orchestration with in a very like natural way where you can sort of you know, each agent is like a person and now you have a team of agents like working on you have one one agent per account that you manage as an example. Or if you're a PM, you have you know, you can have an agent that summarizes all the work that Lauren did last night

[2:49:27] and then now you know what I did, right? So I think our PMs are leveraging that a lot. And they're shipping code too. So you know, I often times they will just say, "Oh, here's a bug. I fixed it. Can you look at it?" And then I'll go review it and actually it's just

[2:49:41] perfect. I'm like, "Okay, stamp." So I think that shows that you know, the the Dune architecture is holding up, right? The all the the really strict constraints allow people who are not experts in engineering to contribute at a high level. So I I feel like I'm already seeing that pay off a lot where you know, designers and PMs are just

[2:50:02] able to to to ship features directly. And that just makes the Grokbot team super fast, right? Where we can ship so quickly. Um And we have a lot planned, so I'm very excited to you know, to to ship more ship more stuff. Yeah, that's awesome. We are at time, so I guess Lauren, if if folks want to support you, maybe go try Grokbot, try out 4 6 and you know, get get provide some feedback. But yeah, this was awesome. Really appreciate you taking

[2:50:34] the time. Thanks everyone for all the messages in the chat. Lots of good questions. I know we didn't get through everything, but uh, as Ken said at the top, way more questions than than we could get through, but uh, yeah, really really thanks thanks for for joining. Thanks everyone for joining and uh, hopefully

[2:50:47] you enjoyed the session. Yep. All right. Yeah, I see see you on my thing. Thanks for having me and uh, if you have any more questions, yeah, just DM me on Twitter. Uh, I'll I'll open them up, I guess. So, I'll let the I'll let the You're going to get you're going to get

[2:50:58] a lot of DMs. Yeah, [laughter] I'll I'll open the floodgates. So, yeah, DM me. Maybe I'll do like a Twitter space at some point to solve for more questions, but really appreciate everyone for showing up, uh, you know, taking an hour out of your day. Yeah, all right. Thanks all. I'll see you in the next one.

[2:51:11] Okay, thanks everyone. Bye. Um, personally, I think that uh, the best use of you of AI is really as like a like a pair like someone you're pair programming with and not someone as not not a not a tool to just replace the act of writing code or worse outsourcing your thinking. And I think there's a tendency like, you know, because AI is so exciting, you want to put AI everywhere and be seen as someone who's very, you know, um, what do you call it? Like uh, on the

[2:51:41] ball, I guess, with with AI. Uh, that you feel this pressure of like, you know, I'm just going to I got to increase my productivity, you got to ship like 100 PRs this week and barely understand what I'm doing and, you know, vibe code my way to a a million ARR or whatever. Um, but I think, you know, uh, one of the things that I've personally seen is that although when I use AI, I save some time, you know, writing the code, I actually find myself spending more time reviewing what the AI did and like correcting it. So, you know, like I've I've tried the the thing that a lot

[2:52:17] of people say like, you know, you uh, instead of prompting an AI to just build the feature outright, you sort of get it to uh, build a spec for you first, then you review that spec, and then you do that whole thing. Then it's like Uh, okay, so I assume you can see my

[2:52:31] screen. Yes, yeah, we're good. So, yeah, today Yeah, I think I think the big theme for me as I've been using agents to write code, and I'm sure a lot of you have had the same experience as well, is how do you trust it? You know, especially if you are an engineer that's been writing code for a very long time, you have a lot of opinions and lessons that you've learned about doing good

[2:52:53] engineering. And when you see agents just, you know, winging it and, you know, guessing, hallucinating, uh you know, confidently stating that they found the smoking gun uh for the hundredth time, uh but it's actually not the real problem, you lose a lot of trust. And when you lose when you don't have much trust in your agents, I feel like you you really can't get the

[2:53:14] most out of them. And for me, the parallel is like with management. Uh so, if I'm an a manager, an engineering manager of a team, and I have a bunch of, you know, I have a team of engineers uh on my team, and I don't trust them, then the mode of operation I'm going to be in is going to be like micromanagement, like I'll have to spend a lot of time looking over my reports' shoulders and checking that they're doing their work well, you know, that

[2:53:40] they're not shipping bugs to production. And so, I drew this chart because uh it's not it's not a very scientific chart, but like this is how I imagine myself and my journey through using agents. So, you know, like fast forward or back forward uh or fast back uh fast backwards like a year or so when, you know, nobody was or not many people were using agents to code, uh I think you uh you know, get into this mode where you are in very heavily in the loop with one or several, like a handful of agents, and you find yourself just constantly fig- uh you know, trying to understand

[2:54:21] what your agents are doing, uh and you're very, very in the loop. You're watching every single output. You are sitting there prompting. Um and you really can't parallelize beyond that because you don't again, you don't have that trust, right? You can't go to 100 agents uh like spawn 100 agents when you don't even trust the

[2:54:40] output of one agent. So, over the past 5 months, I feel like I've really been able to uh like ascend this trust curve. And now I'm at the point where I actually have This sounds kind of scary to say this and it makes me sound like a slop artist, but I I promise I'm

[2:54:58] not. But I actually have my agents now um auto merging PRs for me. Uh which is like a wild thing to say, but um like I woke up today and there were like 20 PRs landed and I just reviewed them on main. Like they were already landed. And they were good. Uh so, how did I get to that point? It's basically what I wanted to talk about

[2:55:18] today. Uh and again, like yeah, feel free to jump in if you have questions, Colin. Um but uh Oh yeah, of course I got to show this this chart. Uh where No, don't do not trust to Someone requested to control my computer. Uh probably won't do that. Uh but yeah, so this chart I think I I I'm showing this chart not to kind of like flex, but to kind of show like the journey. Like so you can see like the curve like it sort of like inversely matches the contributions I've been able to land at

[2:55:53] Cursor. So, I joined 5 months ago. And 5 months ago like I you know, my first month I was like not very productive cuz I was you know, I was learning the code base, didn't know what the heck was going on. And as I got more confident in in my agents, uh I've really been able to kind

[2:56:10] of ramp up my productivity. Uh and again, like yeah, like last month I shipped 1,000 PRs, which is ridiculous. Uh and then this month, we're only on the 12th, I'm already at like almost 800 PRs landed. Uh so the velocity is definitely high and you you I'm I'm sure a lot of you will definitely be questioning like how

[2:56:31] how much of this code is actually good. Um and I think yeah, like that's definitely fair to question. Um but uh yeah, I think I think if you set up your agents well, you can definitely get to a very similar level. Um and so I'm going to talk about how we do that. Uh so for me, I think I'm curious like I guess call it your experience as well, but uh for me I think the most important skill that you should have in your toolbox when you

[2:57:04] work with agents is verification. Uh and by verification I mean the ability for an agent to actually run the code uh or take CPU traces or heap snapshots or uh you know, open an iOS simulator, whatever, you know, however your application is exposed to your users, it can do the same thing and uh run it for real and actually test

[2:57:30] and verify it'll work. Because that's the thing that really closes the loop. Uh it doesn't guarantee your agent writes good code, uh but it allows them to at least write correct code, uh which is a big a really big step forward for being able to trust your agent. Um I will I can share one example that we have uh within

[2:57:53] Cursor. Uh oops. Where let me open this.

[2:58:06] Let me make this bigger. Uh let me go full screen. There you go. Uh so for the for Cursor's agent window, uh so this is actually an interesting story, but uh when I joined Cursor 5 months ago, uh they're actually well I was supposed to join a different team. I was supposed to join like the cloud agents team, but then since I have a lot of experience working on React and agents window is a React application, I was uh I was asked to basically help out with uh the agent window work, uh but um there wasn't really a lot of like skills to help me. So I just found myself at

[2:58:48] okay, uh agents window is going to launch in like a week. All right, we have a really tight deadline and um there was uh you know, I was just sitting there like okay, I'm going to open up the performance the the Chrome DevTools and just like take a trace, look at it myself and try to make sense of this flame graph. And keep in mind, I was just like in my first week. So I had no idea what I was looking at. No idea what I you know, I mean I had some idea, but you know, the the code base was

[2:59:12] completely fresh to me. Uh, and I realized like my agent had no idea either. You know, like I would take a screenshot of the child download trace or send it to an agent be like, "Yeah, it kind of looks like this, you know." Uh, and it would like confidently state like it's this thing and then I try to fix that and turns out that's not the

[2:59:28] actual thing. So this was very very slow process and if you've ever done any like performance work yourself or you know, just even development with an agent where you don't have a verification skill, you are the verifier. All right, you you're the bottleneck. You you you tell your agent to do something and then it goes off and writes some code. Then you open up your you know, local dev build and then you start to say, "Oh, you know, it doesn't work." Then you got to copy-paste screen uh you know, screenshots or console errors or

[2:59:55] whatever. Uh, and then your agent like slowly kind of like uh you know, works with that and then tries to understand it and um and fix the thing. But then you're constantly just in the loop and and being a bottleneck. So there's really no way to parallelize. So the control glass goes like one of the first skills I built uh for cursor and glass by the way is the code name for agent's window that we use

[3:00:18] internally but it's just cursor I guess. And so this skill is I guess that that the code itself is not super interesting. Your agent can very easily make one for you where if if you're building an electron app or a web app or even iOS applications you can teach your agent how to use like the Chrome dev tools protocol or through Apple has some utilities as well for running the simulator and taking traces and controlling programmatic control

[3:00:50] as well. So that's really useful. But one thing I actually want to talk about is the this thing. Where is the read me? So this skill comes with this very unique feature called or not feature unique file called a feature map. And so the story then is like I built this skill and so now the agent was able to actually run the agent window and take

[3:01:19] traces and whatnot. But it had no idea what what the agent's window was. So you know like someone would say like oh the the left side bar is like laggy or something like that or you know the right side the the PR tab is not working. And the agent would just be like kind of flailing around. It would spend a lot of time trying to like look up the code and you know where is this feature? How do I actually get to it on the UI? Which made it basically

[3:01:44] completely useless. You know like we would I would run the skill locally and you know it would spawn a dev build but then it just be churning like I just try to click here it wouldn't know how to get to things. And it was just an awful experience. So who was putting arrows on my screen? So uh Yeah this this feature map has been really useful uh because it teaches the agent how to get to all of the features that you have. Um, and in P Stack, with the plugin that I I've made, uh if you search for P Stack cursor on Google, you'll find it. Uh but there is a create

[3:02:25] verification skill in that plugin where it actually helps you set up something like this for yourself. Um, including the feature map, so it will actually export the code and build up this initial feature map that tells your agent how to get to all of the different features that you have. Uh and this is extremely powerful because now that you have these user reports that come in, uh you you can actually map even like a vague report or

[3:02:48] even a screenshot. So, we have this uh internally at Cursor where uh we have a Slack channel with, you know, lots of people giving us feedback on the agent's window and raw file and whatnot. Uh and often times the report is very bad, like very low quality, like someone will just put Very often we get like a screenshot, like and then someone just says question mark question mark question mark, like

[3:03:10] what is this? And, you know, like that it would without this, your agent is like, "I have no clue." Right? But, with a feature map like this, it has a lot more context and understanding of how to actually navigate, how to get to all of the different features. Uh so, like, you know, example, like, I guess like the sidebar, like what is the sidebar? Uh you know, like all the different subfeatures that are present in it. Um, like from the user point of view, here's where to how to get to it,

[3:03:37] all the different keyboard shortcuts. Uh even like the the what do you call it? The DOM elements or yeah, like the attributes that you use for selecting things through the CDP uh are all there. So, uh again, yeah, this is like really really powerful uh for for agents. And uh and then P Stack ships uh that create verification skill,

[3:04:00] but also a maintain verification skill. Uh so you can keep this up-to-date. Cool. Yeah, I was going to ask how you created that. Did you mind sharing a little bit more about um that that process in the context of Pstack? And maybe just what Pstack is for the folks who aren't familiar. Yeah, so Pstack is pretty interesting because uh well, first of all, the name is kind of goofy. Like the P the P in Pstack is

[3:04:22] like potato potato stack. Uh because I um so uh the there is a pretty uh famous person, Garry Tan, who is the CEO of Y Combinator, and he's come up with this plugin called Gstack, uh Garry stack, and uh funnily enough, we share the last name. We have no relation. Uh but I thought it'd be funny to kind of, you know, poke fun at Garry and make Pstack my version of of of of his plugin, uh but kind of just tailor it to my own set of prac-

[3:04:54] engineering practices. Uh but I honestly actually never set out to build Pstack. Uh it just started with a bunch of skills. All right, like I started with that control glass skill, and then I started with another skill like called how, which I also noticed through like observing agents. Um so like, you know, in the early days of me, you know, trying to climb this ladder, I was like super in the loop, and I was basically nitpicking my agents to an extreme degree. I was uh like I would tell it um "You know, this feature has stopped working. Here's a bug report. Like, why

[3:05:27] isn't it working?" And very often, the agent would just like confidently state like, "Oh, it has to be this, right? It has to be this thing." And I noticed like when I looked at the actual tool calls, I noticed it wasn't actually reading the code that I thought should be affected. And that made me just extremely suspicious. And at that point, I was just like, "I'm not going to I can't trust any this agent anymore cuz it's just it's just completely

[3:05:51] hallucinating." And I think I I it's very easy to just, you know, like build up that distrust and not and kind of feel helpless. Like, you know, you you don't know how to help your agents succeed. But, like again, I think the the the management analogy is super helpful because like imagine if you were a manager of an engineering team and you had an engineer in your team who was a really good coder, no business context whatsoever, you know, they they just you just hired them and they they onboarded,

[3:06:19] you know, like uh 5 seconds ago. Uh and so how do you actually teach that person to be effective? So, how you do that is through a skill. Uh skill being just, you know, it's just markdown, right? But, you know, in in code there's a lot of information, instructions, a lot of uh you can really draw out a lot of intelligence from an agent by Well, some people on Twitter call it like, you know, pull the agent to a different latent space, which is kind of like a fancy way of just saying like since uh you know, LLMs are sort of like they

[3:06:49] predict the next token. Uh when you give it some high-quality tokens uh to begin with, then, you know, it it can kind of pattern match on like a higher space that's, you know, smarter. Um so, that's like a very interesting model there. But, yeah, I built Pstack very, very incrementally. Uh so, uh started with just really observing how agents, you know, all the fail different failure modes of of that agents were having. And every time I saw that, I just okay, I'm just going to make that a skill. All right? Like, stop hallucinating, actually go and search up look up the code, use a lot of

[3:07:24] sub-agents, uh and yeah, stop guessing. Yeah, that makes sense. One one kind of follow-up question here, uh both from myself and from a bunch of people in the chat. So, I guess it's two two parts. So, one is like, how do you maintain these skills? So, like the product changes over time. Obviously, there's a lot of people who are shipping against the code base. So, how do these skills get maintained? Uh and then second to that is like, how do you know when your verification is is

[3:07:48] good enough? Uh like in, you know, you can trust that the very verification loops that you built are going to I guess you trust that the outputs when they're done. Uh yeah, maybe I'll talk about um I think I have some more related. Maybe I'll start with this one first. So, like how do I

[3:08:06] maintain these skills? So, um if you're not familiar with this concept, an eval [clears throat] is essentially like a way to uh well, I think the mental model I have is like it's like a unit test for an agent. Um and uh you can actually make your own evals. You don't need like a special framework for them. You can build you can you can build one depending on like, you know, how scientific and how

[3:08:30] rigorous you want to be. Uh my screen is red. Yeah, there's a little button. Um sorry. Oh, do you mind like disabling the drawing or something? Like I can't see my screen. Yeah, sorry. If you guys could not draw on the screen, that'd be great, but um there's a little button in the Yeah, the the little drop-down.

[3:08:48] Um can I clear? Yeah. Okay, yeah. You got it. Perfect. Continue. Yeah, so evals are a way to unit test your skills, basically. And actually, in P Stack, we ship uh under potato mode, there's a playbook if you search for it called eval playbook. Um and it's uh uh it's like not it's actually pretty pretty rigorous the way it's done. Uh but um essentially, what I do is I spawn a lot of different sub-agents. I have like my main coordinator agent uh come up with a rubric for uh

[3:09:25] what I want the skill to do. Um and then it spawns all these sub-agents and it it creates individual directories for them, uh which are cleverly named to not let the sub-agent know that it's being evaluated because uh agents can actually tell, and when they do, they change their behavior. Uh but it does a bunch of stuff like that to um essentially yeah, like test whether or not the skill I'm making or changing is

[3:09:53] actually doing what I think it does. Um and one of the really nice things about cursor is that we are we we support so many different models. So, you can actually eval your skill across all sorts of different models um and you know, get a sense of how well it performs across that different matrix. Um especially for the models

[3:10:12] that you use. Uh so, I do this a lot. Every time I I modify a skill, I will run one of these uh like the eval playbook uh and make sure that, you know, it's actually leading to a result I want. Uh but I will say like maintaining skills is actually pretty hard. Uh it requires I think a lot of

[3:10:32] taste and observation. So, you kind of need to be very good at being a backseat driver. You know what I mean? Like if you do pair if you've ever done pair programming, for example, uh and you watch a co-worker code and you just like you you could probably do this better or you know, you could do you know, you like why did you not do this, right? You you ask a lot of questions to your co-worker and it's kind of a similar thing here. You like you don't want to just be a passive observer of the agent. You want to be very in the driver's seat in the initial

[3:10:58] stages when you're building up your own set of skills. Uh you know, obviously you can use something like P Sack, but if you're building your own set of skills, it's very I think, you know, opening up the all the tool calls and like reading the code and reading all the the agent behavior and their thinking blocks is uh a really great way to see where they they fail, right? Like what what you know, where are they being done? And then you can go and build a

[3:11:22] skill for that. And then with verification, how you trust it is it's I think it's also a very similar iteration loop uh where, you know, like I actually did the same process for verifying the verification skill where I actually get um So, one thing that's interesting about evals is that you can sort of hill climb them, meaning that uh your eval can produce a score, right? A score that you can get your coordinator to produce, but also you can have a judge agent of a different model to kind of cross-reference and make sure that the first model is not being biased, right? The model that's judging

[3:12:00] all of the sub agents that are running the thing. But you can also like hill climb. So meaning that you can you can use like slash loop in cursor, and you can say, "Okay, keep looping on this eval, right? Until everything is 10 out of 10." As an example. And I did the same the basically the

[3:12:17] same approach with the control scale. And so I kind of it was very it was very hands-off actually. So you know, I I kind of built I built that skill that way, like the CLI in that skill. And over time it's gotten really good. But yeah, it was definitely not super smooth at the beginning. It required a lot of iteration. And I think there's an analogy here for me, which is um Well, I make this analogy later in a

[3:12:43] different slide on my drawing here. But I think of it like uh you know, as a as a engineer now, you're sort of more like like maybe a manager or the analogy I like is like you're like a a chef in a restaurant. You're you're the head chef. You're not cooking all the food yourself anymore. You have a team of cooks, right? You have line cooks, you have a sous chef, you have you know, all these

[3:13:07] different stations. Um and it's your job to really design the environment. You know, you you're in charge of setting up the kitchen. You're in charge of, you know, like giving tasks to different people. So um yeah, it's a very interesting way of working. Uh but yeah, that's that's how I basically built these verification

[3:13:28] skills. Yeah, just just one thought there and like I had to go try to go one layer deeper. So are you let's say we wanted to build an eval or a skill for for something and we wanted to kind of get better on its own which is is what I think you're suggesting. Are you doing that in like a work tree kind of isolated with like the sub agents and and then the reviewer agents and and all that is it happening like in

[3:13:51] some type of cloud hosted environment? Like what's the the more the practical steps if I wanted to go do this and like set up a verification system for something what would I what would I do or where would I start? Um I think that the best place to start is local because you can observe you can definitely observe what your agents are doing so if you're building a verification skill for yourself I would definitely start local and just have your agent bring up the application whether it's like a CLI or desktop app or whatever and so you can actually observe right you can see how the agent is interacting with the the

[3:14:27] application you can see it you know how it calls like the different APIs that allow it to interact with the the application. Um but for me personally I have basically been kind of all in mostly all in on cloud agents because they're extremely powerful. Uh and the really powerful thing about cursor is the the cloud agents actually where if you spend a little bit of time setting up your environment these control skills these verification skills pay a huge amount of dividend because it's not just something that makes you as a single engineer better it actually levels up your whole team and even your whole company because you can

[3:15:08] actually start thinking about cloud agents you can start thinking about automations that automatically do things like I get to I I cannot talk about this a bit later but I'll just kind of get into it where where you know for example like I talk a lot about this agent we have called Benny right who who you know takes all of the bug reports that we get and it automatically goes off in the cloud, opens up a cloud uh it's you know it's desktop. It runs cursor in its own computer, and it uses the same control skills to interact with the application and try to reproduce the

[3:15:43] bug uh or the user report. All right, and this is so so powerful because at once I can immediately I I get so much information from this automatically. Like here in this example, you can see that uh the Benny actually reproduced the bug, uh but it's already fixed on main. So, it actually confirms that we fixed this problem already, and all I need to do is just release another build of of

[3:16:05] cursor. Uh so, that's a huge information there that I didn't have to go off and sit with an agent, you know, and spend an hour trying to figure out is this fixed or is this not fixed. So, you you you gain back so much time, uh but you know, everybody on my team benefits from this. Everybody at the

[3:16:20] company benefits from this. Uh so, definitely think that uh you know, keeping these uh using cloud agents is super powerful. Uh but yeah, it's like a journey. You have to trust it first, right, before you you get to this point, and that's it goes back to what I was saying here where, you know, it's very hard it's it's almost impossible, and I would definitely encourage you not to try to jump from, you know, like if you're still in this zone, you don't want to jump to like I'm going to spawn a hundred a thousand or a thousands of cloud agents right now because you're just going to waste a lot

[3:16:53] of tokens, um and it's going to be extremely expensive. Yeah, so just kind of recap so far. Basically, the if you wanted to go on the journey that you've kind of gone on, it would be to start with verification, uh building some some skills and some some ways of determining that the agents are producing at least like correct code. Whether like you said whether it's good code or not is maybe a separate question, but like it's it's technically solving the problem by looking at, you know, stack traces, looking at, you know, the the actual behavior in the

[3:17:18] app, and so on. Um and then once we trust it locally, then we can start to think about scaling into the cloud and running more agents that are picking up signals, I guess, on their own, right? So, whether that's like a bug report that comes in or something, they can go and pick it up and solve the problem and and give us back a PR. And then maybe the last step is like auto-merging the PRs, which is where you're at. Maybe not where

[3:17:38] everyone is at. And then reviewing them on main, but is that is that about right? Yeah, exactly. I think yeah, that's why I drew this this this this curve, right? Because that this basically describes my journey of, you know, when I started barely could use a couple of agents and I was just observing every single thing. I think there's really no shortcut for going from here to there, because this is really about your personal level of

[3:18:00] trust in agents, right? Obviously, you know, as a as engineer, you don't want to just slot code into production. So, how do you actually build up that trust? Takes a lot of I guess taste and judgment. But, you know, like I think plugins like Pstack definitely kind of help you get up to speed much quicker. And so, I guess it's like if you trust me and you trust Pstack, then in by extension, you can maybe trust your agents. But, if you don't trust me and I definitely would not encourage people to

[3:18:32] blindly trust me. Uh you know, if you build up your own set of skills that you can obviously, you know, take a look at Pstack and kind of fork it, make it your own, improve the skills, definitely encourage that. But, for me, it's really all about it just keeps coming back to trust. You know, every one of us here in this chat have a different standard for engineering. And there are different things that are important for us in our code base. And when you are able to encode all of that into skills and you can verify that your agent is actually doing them, that allows you to really kind of ascend this

[3:19:05] curve and you know, start automating things. There's another piece I wanted to talk about. There's Yeah, go for it. I'll I'll think of more questions as I go, but yeah. Yeah, I think there's a a third part to this, which I haven't talked about yet, which is kind of an interesting one, which is like refactoring and rewriting. Like one of the uh I guess most controversial one of the most controversial topics in the industry, I think, is like should you rewrite your app or not? Um because I think engineers are very prone to this where especially when you join a company, you come in and you see like

[3:19:41] the code base and you're all like, "Man, this is Like who wrote this code? You know, it's it's terrible. I want to rewrite the whole thing." It is a very common inclination and I think a lot of you know, before agents um and I guess arguably even now, people will definitely discourage you from re-

[3:19:56] rewriting stuff. But I'm actually here to make a case for why you might want to consider it. Um because I think it really depends. Uh you know, uh brownfield applications, I think, are actually in a pretty good spot, especially if they're set up well already. Uh and like recently I've been talking to some people, but uh you know, I I I was just observing. I I just noticed this parallel, which is that a lot of big tech company problems are now

[3:20:25] everybody's problems. Um and the big tech company problem, you know, like when I was working at Meta, like we had this giant monorepo. We had like, I don't know, tens of thousands of engineers just, you know, like banging on their keyboards and and shipping code. And a lot of really great engineers at Meta, uh but uh I'll say like, you know, you'll be surprised that the code

[3:20:46] quality is actually not that good. Um and so I often joke that before AI slop, we had human slop. Um and so uh you know, I think a lot of big tech infra like uh like what Meta has or Google, you know, you know, really big tech companies are actually designed for that where you you're sort of like you're catering to the the you know, like uh this this sounds so bad to say, but like the the least capable engineer on your team, right? You build you build frameworks, you build conventions, you build guardrails, you know, you restrict credentials so that, you know, your intern doesn't wipe your production

[3:21:20] database. Um there's uh you know, if you have that level of infra already, I think your agents can actually already do a very solid job, right? Because they have the the guard rails are already in place for agents to not cause havoc or not cause too much havoc uh in your code base. Um and you can always add more, you

[3:21:41] know, guard rails. Uh but I think like greenfield applications especially are, you know, like the brand new applications are like the biggest risk, in my opinion. Uh and also the greatest opportunity. Because, you know, if you vibe code a project uh a prototype, um like we did for GrokBot, you know, GrokBot was spun up very very

[3:22:00] very quickly. Um and if you yeah, if you haven't heard of of GrokBot, it's like our a new application we just launched yesterday. Uh it's it's really cool. Uh lets you orchestrate your create like individual agents that have their own identity and you can kind of orchestrate them. It's super cool, definitely check it out. Um but yeah, that was it's like a very it was a very greenfield application, like most prototypes are. So it was like vibe coded very quickly, humans were not

[3:22:24] reading the code at all. And uh I had this tweet recently uh where I said something about organic architecture. Um maybe I'll find it. Uh but the idea is that uh when you have a completely vibe coded application, you essentially have no guard rails whatsoever. So, uh your agents when you give them a task, they will just solve it in whatever method is the

[3:22:52] most convenient. And over time you get into this uh situation where you have a code base that is spiraling out of control because you don't understand it, uh your agents understand it, I guess, in a way, but like they've built something that is, you know, optimized for short for shortcuts. Uh and uh you know, it will you will suffer you have a lot of of

[3:23:13] issues with that application. So I think starting your code base with like very strong constraints is very much needed. Because like when you have a code base that you can trust, right? When you have guardrails that actually help you uh help your agents write good code, you can get into the you know like into this part of the curve where I I where I like I said you know I woke up today and I had like 20 PRs merged by my agents and that's because I invested a lot a lot of time over 600 PRs I I I calculated yesterday. When I refactored all of Graph bot to this new

[3:23:54] architecture that I've been building. Um and yeah I've gotten to a point where I I don't really look I really don't look at the code anymore and um I say that not just you know to sell you tokens but because I you know it it it took a lot of work to get to that point. I spent a lot of tokens to get the code base to this point where I no longer have to

[3:24:15] look at it. Uh but I'm very excited because you know of the potential where you know it's not just it this doesn't just benefit me. It benefits everyone contributing to Graph bot and it also empowers you know designers and product managers and you know even GTM people to add features to Graph bot and I don't have to worry you know I don't have to to wake up at night in in the middle of the night and worry like oh someone's just merged a perf regression. Right? I have a ton of constraints and CI like it's actually very annoying to write code in in Graph bot but like agents absorb all of that

[3:24:49] annoyance. Um but yeah I'm happy to talk about what exactly that is. Yeah, I think one question um before we get into the this part here is just around that element of like what your your your CI looks like or maybe some of the constraints and then also like the average PR size. I saw a question about that earlier. Just to give people a you know kind of a a glance. It doesn't have to be like mathematically average but just a you know like what generally the size of the a PR is if it's only a couple lines of code or you know

[3:25:19] yeah. Um I think it depends. Let me I'm trying to do this in a way where I'm not going to like You know yeah you don't have to share the actual number of you like the actual average just This is fine. Benchmark. But but we have so okay. This is not interesting but a fun fact is that virtualization in Grokbot and in cursor is actually powered by pre-text which is a sort of new library that someone's built that's really interesting. You should

[3:25:51] you should check it out. But that's not really that important. I think the average PR size I actually don't know I I don't know if I want to click on these. I probably can but I would say like they can range anywhere from a few hundred lines or 50 lines to like a thousand depending on what the thing is

[3:26:09] doing. So like here I'm actually like deleting a bunch of files so I expect that it's just this like mostly deletion. But yeah it it kind of varies. There's no like Yeah. There's no like hard cap or anything like that. They're all like 50 line PRs. cap there's definitely no hard cap but I do encourage my agents to split up their

[3:26:26] work into multiple PRs. I do that mostly because uh I like I like the idea of I guess maybe this is much harder to do now as in the world of agents and you have like so many commits. But I like the idea that you know the get history is a very rich source of context and I like the I like each PR to sort of atomically describe what that small piece of thing is doing which also makes it easier for me to revert changes and like figure out you know oh I shipped a bug and it's this it's here it's not in this 40,000 line PR where

[3:27:00] you know, knows what landed in there. Uh but I I don't have a hard cap on PR size. Cool. And then um yeah, also quick question on like CI. So again, you don't have to go into like uh the screen share of like your your CI does, but just generally would you describe what the CI kind of looks like uh or how strict it

[3:27:18] is? Uh yeah, so uh well, specifically for Grokbot, so Dune is the it's the sort of cheeky code code name for the architecture that we built for Grokbot. Um the CI looks pretty annoying because there's checks for everything. So, like literally I have um uh well, if you've written any React, for example, you know, you know that one of the biggest foot guns in React is use effect. Uh so, in uh Dune and in Grokbot, we've banned use effect. So, Dune is this you can the the mental model of what Dune is uh you can kind of think of it as like Next.js for uh electron apps and it's designed for

[3:28:00] agents to write. Uh and it's like custom for, you know, our agent forward applications. Um so, the CI checks are very like specific to that, like, you know, don't use use effect. This it's it's banned, like CI will fail uh and yell at you. We have like some of the more interesting ones that people might raise eyebrows is like I actually

[3:28:19] banned code comments as well. Uh which is very interesting. Um but I've noticed that 99% of the time agents just write code comments that kind of describe some historical thing that is actually totally irrelevant to the code. Um like it will often say like, you know, "Oh, Lauren said you should never do this." and it's now in the in the code comment. I'm like, "What?" Like, why what that was I didn't say that as like a durable, you know, global rule. I just meant like your this PR sucks and

[3:28:46] you should change that part. Uh agents don't really understand this that well, surprisingly. Uh and or they kind of assume too much and they kind of do things in like very stupid ways. So, like yeah, we just ban everything everything you can imagine like the agents are bad at, we ban. Uh so, one example that we actually suffer a lot in the agents window is we have uh you know, you know, if you've used agents window, you've definitely seen performance issues, and you know, we're constantly trying to fix them. Uh but it's like a it's a never-ending struggle because there's so many pull requests that get

[3:29:21] merged. Every any one of them could just regress performance or stability or reliability. Uh you know, the agents window doesn't have this architecture yet. I plan to do bring this learning back there, and kind of refactor everything there. Uh but uh it just regresses super often uh because uh there's just one example is like we have very poor um isolation between processes. So, like on you know, on on Electron, you have a renderer thread that renders your UI, but you also have like a main thread that you can run other code that you know, doesn't need to block the

[3:29:54] renderer. Um but we do a poor job of separating those things, and so often times you just accidentally have code that gets pulled into running on the renderer thread. All right, and then all of a sudden you're competing with the the renderer that you know, that has a very If you want like 60 FPS, you have to every frame that gets drawn has to be done in 16 milliseconds. So, very very small, you know, deadline per frame uh if you want, you know, a very smooth product. Uh and when you start building bringing in accidentally bringing in, you know, things that are like very computationally heavy, or they have a

[3:30:27] lot of IO, uh then you just get into like a lot of jank. And your your FPS really drops, you start uh you know, losing frames, you get long tasks that take more than 16 milliseconds, and you just get this really choppy experience. So, all of those patterns that we've learned basically building Electron apps, we've encoded into this framework, and it becomes like a hard failure. So, I literally in in Grouparoo, we literally have a directory called electron main, electron renderer, and we have uh import uh CI, I guess, where we actually check the dependency graph to make sure you're not accidentally importing code from one

[3:31:04] directory to another. Uh so, that's enforced by CI. Um as well as Bugbot, uh which is our which curses um like code review tool that runs on CI. Uh you know, in our agents MD, it's everywhere. Like So, I I I I I um I have this thing here where I I talk about like um you know, like there are multiple layers, I think, for building a good

[3:31:28] code base. Uh obviously, the code base is one where uh if you have an architecture like this, where it's extremely strict, uh you know, the the the way to build features is very conventional. That's like the strongest strongest level of enforcement because agents just love to copy existing patterns. So, uh one example of this in Grokbot is like we have this these concepts called like a feature, and we have entry points, and transcript cards, like all you know, the

[3:31:54] cards that you see in the chat. These are all like like nouns, I guess, in in the framework, and so there's a very conventional way of creating them. And so, like a feature is all in in a single directory as an example. And so, all of the code that contributes to that feature lives in one directory. So, it's all co-located in one place. Makes it super easy, you know, agents don't have to like uh grep around and try to figure out like where all the things are. It just looks at the feature and like, "Oh, okay, I'm working on the onboarding feature in Grokbot. Uh I'm just going to

[3:32:26] work in this directory." And for 80% of the work, it's mostly just very encapsulated there. But [clears throat] uh like it's like very it's like designed again for, you know, like the dumbest agent. Like you don't have to think, right? The the the One of the key principles I have for this framework is like the shortest the

[3:32:45] shortest path is the best path. So, uh because that plays exactly to how agents love to write code like they like to take shortcuts really. You know, they'll they'll find the quickest way to solve the problem. So, why not make that the best way to solve the problem? So, I probably won't get into all the

[3:33:03] specific details. And this framework is really more of a collection of ideas and principles rather than something that will open source. You can you can you know screenshot this I guess if you want and tell your agent to do something build build something like this for you too. Yeah, but it's really all about the layers. You know, like the code base is one part with features. And directories

[3:33:29] and you know import. Blocking import dependencies that shouldn't be imported. But and and it all enforces that and static analysis. So, like there's CI checks. We have a lot of lints for bad patterns that we observe. Compiler diagnostics. There's also rules and bug bot which are I think like three four five are more soft. Right? These two actually make make CI red. Right? So, that you know, there's a hard constraint where the

[3:33:59] agent can't just write crappy code. For rules and skills and bug bot. Your agents can still forget. Right? You can still Or it may not always consistently apply them. So, I like to layer them. But I don't I don't like to rely on them as the only source of enforcement because it's very very soft. Right? And if you if you only have rules and bug bot and

[3:34:23] skills and a style guide for your code. You will it's only a matter of time before your code base looks like complete trash. I'm sorry to say that. But I definitely recommend you know that you know investing in you know things that can be hard enforced. Right? And this is why you know, maybe the choice of tech stack

[3:34:41] that you use is also very important. Um Like I think for example Rust is sort of making you know it's like getting super popular again uh because the compiler is so strict, right? The compiler enforces so many different things. You know there's a borrow checker that you have to appease. And if as long as you make sure your agents don't write unsafe code blocks uh you can more or less feel somewhat confident that if the code compiles it

[3:35:05] probably works and it's good. Uh but you see it gives you that level of trust and confidence that you as a human engineer no longer need to go and check it yourself. You you know you you rely on code and static analysis to actually make that uh a lot smoother. Um and I I guess the worst part the worst place to be in is if you are stuck in code review land where you actually enforce all of the constraints to invariants in your code base by literally the human person saying you know reading the code and be like okay

[3:35:40] you should not do this. Right? Every time you have to do that you should consider that as a code smell like a anti-pattern and you should say okay instead of me commenting on the PR how do I turn this into a hard rule? All right, how do I turn this into a lint rule? How do I turn this into a CI failure? Or how do I even categorically eliminate this problem

[3:36:01] uh entirely? Uh I I can talk about another migration I've done but I'll probably pause here. Sure. Yeah. Yeah, I feel like that's that's where I am to be honest is is what you're describing right now which is that I don't have all of these rules. Uh so I have some things to go do after this session in terms of being able to scale my agents. I'm I'm definitely on like the uh you know maybe a couple of parallel ones locally stage so like two to three locally. And I'm sure most people here are on the same so uh yeah I know we have like a couple

[3:36:28] minutes left. Uh Lauren was there anything else that you wanted to to highlight? I obviously there's lots of questions so I can grab more but I want to give you a few minutes uh if there's Is there else you want to talk about. Well I think I've been yapping for quite a lot, so maybe let's just do questions. Okay, cool. One question that I had a couple of came up a couple times was just around like token usage. So, the the question is like is is what you're describing a realistic thing for people who are on, you know, a normal set of token usage. They don't

[3:36:54] have, you know, obviously unlimited tokens to work with. Well, I think that's a really good point. I mean, like obviously, you know, I work at a AI lab where we have unlimited tokens. So, I definitely cannot say that, you know, this is something everyone should do in the exact same way that I did it. I think it's possible to get to this point without, you know, breaking

[3:37:15] the bank. But, you know, if you're like an engineering leader or, you know, you're you're you have a startup that you lead, I think to me it's a question of ROI. Um, and it's like uh yes, you spend a lot of money on tokens in the upfront stage. You know, you like refactoring a code base is going to take

[3:37:34] a lot of tokens. Adding all these things is going to take a bunch of tokens. But, if we're heading to a world where agents are writing all the code, and you know, you want to be very lean, right? You don't want to have to hire You don't want to be You don't want to become like meta, right? Like I mean, like in terms of you don't want to become a 10,000 person engineering org because I mean, that's a cool problem to have, but also you need you have so much overhead. There's like planning, you know, like it's it's a personally, I I wouldn't it's not super fun. But,

[3:38:07] I think you want to stay very nimble, right? And you want to you want to be like agents are all about allowing you to do things that you couldn't do before. That's really to me like the value of agents. You know, it's not just storing tokens on every single little thing, but to me like the thing I couldn't do before is like enforce this level of constraints in a

[3:38:27] code base by myself. Right? Like I'm just a single person, you know, it would have taken me years to build this framework and do it all the refactoring and test everything myself and verify, you know, like run Imagine if there it was just me right now in in pre-agent era, just like running, you know, by my It would take me so long. All right, and my salary is pretty high, right? Like so, you know, the the question I think an engineering leader might have is just then, you know, like what is There's a trade-off of do you hire someone to do this or spend the tokens to set up a code base

[3:39:03] so that even the the most naive, right, the dumbest agents can do a good job. And when you actually get to this point, like even agents that are not, you know, Fable size do an excellent job of writing code and this pays a lot of dividends as well for me in personally where I have empowered not just myself, but again, like PMs, designers, engineers who are not familiar with Grok to just contribute in a way that

[3:39:30] is sustainable. So, I think yeah, it's definitely like a trade-off for sure, you know, like nothing's like free for sure and tokens are pretty expensive. But oh, actually, I don't I I don't know how many of you have seen this, but we actually announced Grok 4.6 today. So, very exciting, finally out. Um so, yeah, Grok 4.6 would be like a great It is very

[3:39:51] very smart. It's really good on the on the benchmarks. And it's the same the tokens I hope I'm not saying this incorrectly, but I believe the cost per token is the same as 4.5. So, you're actually getting more intelligence for the same cost. I think this is an area that Cursor tries to Cursor and SpaceX AI try to really optimize for like that Pareto frontier of you know, cost versus intelligence. You know, we don't necessarily want to build the biggest model ever because that is extremely

[3:40:24] expensive to run. It's really about like how do you find that sweet spot, right? You don't you don't need a giant model, but it's just super smart, right? And it's not very expensive for inference. Uh but um yeah, I think to kind of round it up. Um I think it's like a it's it's there's a if you do your own analysis, I feel like it's pretty positive. It it'll be pretty positive that the ROI you get from investing in stuff like this uh just empowers not just yourself, but your whole team to be so much more productive, right? Imagine if you have an army of engineers like me who are

[3:40:58] shipping so much improvements and and bug fixes uh you know, every day, right? Like that is pretty exciting. Cool. One last question before we wrap up. This one is for the people in product on the on the call. So, let's say we do have an army of engineers who are shipping like Lauren. I'm just curious like how is the product team or other functions of your company keeping up given that like if you're shipping so quickly, how are they using AI more to do their jobs? Like as much as you can speak to that. I obviously you don't like you're not in that role, but just

[3:41:28] curious about how that works. Um I think this is where Grokbot has been actually exceedingly powerful. Uh where so before Grokbot like you know, obviously Cursor only had Cursor. Like we only had agents window. We had a CLI. We had an IDE. And these are really like power user tools, right? Like they they're designed for developers. So, it's very very developer-centric. You can do knowledge work in them, but it like the UI is not really optimized for

[3:41:54] that. So, we actually didn't really have uh while I think like a lot of people like you know, GTM, product, like they might have used Cursor uh to do their work, but it definitely wasn't like a delightful experience for them. Um I think now with Grokbot uh it's become Grokbot is basically like the Cursor moment for people who are not in tech. In my opinion. Like it's like it's like a very very accessible way to use agents in a very comfortable, very familiar interface. It looks like iMessage and it's very fun, too. You know, you

[3:42:28] can give your agent a fun name. You can have you can kind of do orchestration in a very like natural way where you can sort of you know, each agent is like a person and now you have a team of agents like working on you have one one agent per account that you manage as an example. Or if you're a PM, you have you know, you can have an agent that summarizes all the work that Lauren did last night and then now you know

[3:42:48] what I did, right? So, I think our PMs are leveraging that a lot and they're shipping code, too. So, you know, like often times they will just say, "Oh, here's a bug. I fixed it. Can you look at it?" And then I'll go review it and actually it's just perfect. I'm like, "Okay, stamp." So, I think that shows that you know, that the Dune architecture is holding up, right? The all the the really strict constraints allow people who are not experts in engineering to contribute at a high

[3:43:14] level. So, I'm I feel like I'm already seeing that pay off a lot where you know, designers have PMs are just able to to to ship features directly and that just makes the Grok bot team super fast, right? Where we can ship so quickly. Um and we have a lot planned, so I'm very excited to you know,

[3:43:37] to to ship more ship more stuff.

---

About this transcript
Read from YouTube's own caption track and laid out by ViewRank AI (https://viewrankai.com).
ViewRank AI finds the videos already beating a creator's own average on Instagram, TikTok and YouTube Shorts, transcribes them from the audio itself in more than 60 languages, and turns what worked into new ideas and scripts.
Free transcript tools, no account needed: https://viewrankai.com/tools
How to read any video this way: https://viewrankai.com/llms.txt