Model Selection & Token Efficiency

0:00All right. Hello. Good morning, good evening, wherever you are. Uh, my name is Santi Garza. I'm a senior field engineer here at SpaceX AAI. And um, I'm joined by an absolute legend, Martin. Martin, if you'd like to introduce yourself. Hi, nice to meet you all. My name is Martin. I'm also a field engineer here

0:19on the cursor side or SpaceXi rather. And I'll be here more as a moderator. So, if you have any questions, feel free to put them in the chat. And I know Santi will also have his sort of Q&A sessions and time in between to break for them. Um, but you are in amazing hands with Santi and I will pass it back

0:37over to you. Thank you so much. So, uh, yeah, my job here as a field engineer is essentially to be a technical point of contact for our customers and answer any questions that they or you might have. Uh but really the main most important part of my role is to make sure that our users and our customers are real experts in how to best build with AI. Uh I'll start with a little story. [clears throat] Um earlier this year, one of the uh COOs of of of one of the world's largest tech companies came out and said that they had burned through their entire AI

1:18budget for the year in just four months. uh that is not good. Uh but it's a story that a lot of people are feeling right now, not just uh at companies or but also on small teams or even you know individual developers. And it's no wonder uh you know AI feels like a black box. And so today we're going to actually take a peek under the hood and see what is actually first of all what are tokens, what burns tokens, and how can you be more efficient and effective uh without slowing down. Um in fact uh as a result of some of the tips you'll

1:55see today, you will probably speed up. Um so I'm just going to uh make sure that um we are all set up here. Give me one second. Uh welcome. We will be talking about token optimization and model selection today. Um so uh without further ado, let's get started. All right, let's start with the basics. What is a token? So uh a token is um a unit that a model can uh input or output. And so um the closest mapping is it's like a word. Uh in some cases as you can as you can see here on the screen uh the the words or you know might be broken down into a

2:41couple different tokens but on average one word maps to one to two tokens. Um the model basically only takes in and out uh tokens. So an image a large file those all get tokenized as well. Um the reason tokens are important to us is because tokens are the unit that we're building. So um let's get into it. A lot of people talk about uh on you know X or or LinkedIn I hear a lot of chatter about the harness. Uh but for a lot of people they don't know exactly what the harness does. So I kind of want to break down the anatomy of an agent so that um

3:23we all know what is actually affecting spend and not affecting spend. So when you prompt uh cursor for example the um the first thing that's going to happen is the harness is going to uh do some prompt uh management and take a look at your prompt but also uh try to piece together all the context that might be needed for the model to properly service your request. Uh, the analogy I like to use is that if the model is the engine, the harness is essentially the car around the engine. And um, I love that analogy because um, you can take the same exact engine and put it in two cars

4:05and they you'll get more or less out of the engine depending on on how it's tuned. Uh, Cursor actually tunes the harness to every single model. So when you use cursor, you're actually leveraging um you know years of work uh in trying to get an agent to uh build and and service the requests um to your liking. As we all know now uh working with agents can be difficult and frustrating. And the most powerful thing about agents is also sort of it's like a double-edged sword. It's also the biggest weakness which is that they're non-deterministic. So software engineering um historically has been such a deterministic space. You write

4:50the code, it executes the same way every time and uh that's how massive systems that service millions or even billions of people are built. So now that we have this really powerful technology um it's it's exciting, it's liberating, but it's also uh concerning and at times frustrating because we are not going to get deterministic results. And so cursor has had to solve uh a lot of these problems with working with agents over the years. And so when you use a model through cursor, you're going to get to leverage all of those uh efficiency gains. Um the harness also uh

5:26coordinates and orchestrates tool calls. So this is like you know beyond just text in text out. Um there's a lot that you can do around MCPS file reads. It's there's a bunch of tools that the agent can can use um and uh and coordinating API requests and things like that. Now um the harness will actually communicate with the model uh kind of back and forth. So again, think of the model as the brain um and the harness as maybe the hands for example. And so um the harness will check in with the model uh and the model will you if you pop open the agent trace you can see it thinking

6:06um and and coordinating with the agent uh to take action. Eventually a prompt comes back to you and uh and that's what you see on the other side. One thing to note here is that uh what you're being built for is the essentially the tokens that go in and out of the model. All right, this is a little known fact but extremely important. So, uh the models have amnesia. Uh what I mean by that is that they don't remember what you just said in the previous turn. Uh it's like a fresh start literally. And so you can imagine uh that you basically have to feed the context back in on

6:46every turn. And you don't have to do this. Um the harness handles this. But uh this is an unbelievably important point because uh as the context grows, so as your conversation grows, as more tools get pulled in, uh as as you know, files get read, that context needs to be managed and fed back into the model in order to give the model enough context on the next cycle to make the right decision or or say the right thing. Um why does this matter to you? It matters because eventually the context hits the or gets close to hitting the limit. So

7:29um models do have a context window. You've probably heard this before. Um, and we'll get into that in a second. But, uh, when the wi when you start to approach the window, um, you run the risk of of overflowing and and then the model won't be able to service the request. And so, right around 90%, uh, cursor's harness starts to say, you know, it rings a little alarm and says, hold on, we need to start to compact this conversation a little bit and summarize this conversation. Uh that way we can free up some space to keep the conversation going. And um so if you bloat the context window

8:10yourself uh you have to realize that this you'll approach that context window uh a lot faster. And so when the compaction actually happens, which is the part where you summarize, uh you lose a lot of the details like the the important details sometimes that you wanted the model to know. So uh what happens actually uh I I promised you we'd peek under the hood. Um I'll I'll tell you exactly what happens. So um essentially the the uh the context that gets served up to the model is made up of you can think of it as being made up of three main parts. You have the prefix which is

8:48um we call it the prefix. It's where it's everything that every single model turn absolutely needs to know. So for example like the system prompt uh what rules are there that you've introduced and uh don't worry we'll look at rules during our demo at uh at the end of the presentation. So what rules have you introduced? What skills or tools are at your disposal like these things the model needs to know this every single time and so we are not going to compact the prefix that would be detrimental. So um the the middle part is like the conversation as it grows, right? And the last part is your most recent ask. We

9:26can't compact that either because you just asked that. So it would be confusing if all of a sudden the model didn't know what you just asked it. What ends up getting summarized is the middle part. And so, uh, if you ever see, you know, the context window start to summarize, um, on cursor or on any AI platform, what's happening is it's crunching that middle part and pulling out like key themes the model might need to know going forward. But if you continue to compact over and over, you eventually end up with uh, a very blurry

9:57conversation. All right, let's talk about uh, token types. So uh I mentioned that tokens are the unit you're building in. Um there are four flavors of tokens. One is the input tokens. This is the the these are the tokens that literally get fed into the model. So uh this is going to be partially your own prompting, but it'll also be like I said the system prompt and the tool definitions and the things that the model needs to know in order to uh operate. So um that input cost is over here on the left. Um, you can see, uh, I've I've kind of ordered it from most expensive to least expensive. Um,

10:38but, uh, Claude Fable, which is a very powerful model from Anthropic, um, you get build $10 for every million tokens. That's what that means. Um, Opus, $5 for every million tokens. If you go all the way down, you have Gro 4.6, um, which is a model that we relased, um, in conjunction with SpaceX AAI. So, Cursor and SpaceX are now one company. And so you're getting um uh great models uh for for a fraction of the of the price. Um so $2 for for an input token from from Grock. That is uh we'll talk about models in a second, but um that is a great deal. And composer 2.5 is just 50

11:19uh for input tokens. So um output tokens are going to be the more expensive tokens because these are the tokens that the model actually generates and spits back out. This requires um you know work on on their side. It's not just reading the the input, it's actually generating the output and that's why it is more expensive. Um and you can see the scale here. Um you know $50 for per million output tokens from Fable. So those tokens are more expensive and you can run your uh eyes all the way down to the $2.50 from um Composer 2.5. Okay. There are two other types of tokens. There are

11:56cache write tokens and cache read tokens. So, um you might be wondering why does the model need to reread what I just told it every single time? That's ridiculous. Um I don't want to get charged for that. Uh we agree, we all agree. All the model providers agree. And so we created a caching system and so did Enthropic and and GPT and most um model providers where um it'll write those in uh those input tokens into a cache and um it'll look there first. So rather than reading it fresh, it'll look there first um to see if it can pull it

12:34out of the cache. That's way cheaper. And so um you're not getting charged nearly as much for what's happening behind the scenes um in in rereading the the conversation. Now I will mention that um the way caching works is is essentially uh takes a look at the prefix and like tries to match it. Um and so if you say um change a rule for example halfway through your session that'll like bust the cache. Not a big deal. Um I wouldn't worry too much about it but that's how it works. Uh, also if you change model providers, the the caches do sit on the model provider side. So it'll rewrite to the cache. Um,

13:07in case anyone's curious. All right, let's talk about what actually consumes tokens. So um, you know, there's a myth that every single action that the agent takes costs tokens. It's actually not true. Um, the agents actions are free. Um, the majority of what you're paying for is what goes in and out of the model. And so um you know actually searching through your codebase for context um that uses uh GP and and some other fancy things on the cursor side that don't cost you any money. Um uh things like you know uh tab completions or something takes a long time that necess that doesn't necessarily cost you more

13:48tokens. The only thing that cost tokens is basically the coordination that happens with the model itself. So thinking you know reading and writing essentially. All right. Um this is a very uh quickly evolving space. Um last year the uh the kind of frontr runner across the frontier swapped hands six times across different companies. Um so uh this year it's even faster. Um Grock 4.6 six just came out and is um topping a lot of the charts um uh around different um eval specifically especially for coding but it's also general reasoning model and can do u anything all of the frontier um uh models can and um and so uh I I

14:38highly encourage you all to keep your eyes on the latest this seems to change every week uh but more importantly you know for anyone working on a team or trying to decide kind of which provider to go with. Um, what I always say is it to me in a in such a quickly evolving space, it's a little bit dangerous I would say to lock into just one provider when the frontier is changing so quickly. And so one of the major benefits of cursor uh like top five, top 10 benefits is that you get to

15:11access models across the whole frontier. All these models you see here are available in cursor and so you don't have to choose one favorite and lock in for like a year or whatever it is. Uh you can actually um you can actually leverage them all through cursor and in fact you get to use the harness that's tuned to each model as well. Um I just want to call out a few things. Uh when you compare composer 2.5 to Fable um it's a 40x difference in price uh for getting a task done. Um, now you might

15:44say, "Yeah, but Fable's so powerful." That's true. Um, so, you know, kind of comparing apples to apples here, uh, 4 Brock 4.6, which is scoring, um, uh, really high, uh, in our in our internal, um, uh, scores is 6x, you know, fable 6x more expensive. I actually have here um I want to show you all because you all should be checking this too. Not just here um with cursor but with other um with other uh uh eval um sites. But we have here you can go down and check uh each of these specifically and see the kind of the average cost uh per task. So for example, you know uh to get

16:29a task done uh Grock 4.6 on average takes $2.80. 8081 and Fable $17.32. So um at the same time uh internally we test for software engineering tasks in general not just coding. Um it's scoring basically neck and neck just slightly above Fable. Uh but you can also see uh composer uh down here which is just 44 cents per task and handles a lot of um uh more discrete tasks with a lot of definition. Um, and you can come in here

17:01and and and see this evolve over time. So, this is cursor.com. All right. Um, I want to call something out. Part of my job uh and and Martin's job is to test these models um more practically [clears throat] speaking. So, actually play around with them, see how they speak, see how they act in different situations. And I'll say that no one model wins every category. Um some of them don't win at any categories. I won't name any names. Uh but um but a lot of models have strengths and then they have limitations. Um so um you know I have here uh I chose six models for

17:43you. Let's just go through them quickly. So Grock 4.6 uh the one we just talked about which is um SpaceX AI's latest model. Um it sits right on the prao frontier. Uh what that means is that you know if you have like an XY axis so we were looking previously at like you know what the scores are um versus what the um the cost is it like sits like on the top right that's what the paro frontier is basically so it's it's the best of both worlds you can think of it that way um so you get 70% which is right around what Fable is uh for under $7 and it's

18:18the only model that does that or or is even close to that Um then there's Claude Fable uh which is um I use it sometimes. It's it's very very expensive. Um but for highly complex uh problems with like a really wide surface area where you're going to have to pull in like a ton of files, I think it's worth it. It's really good at that. Um, so complex debugging, um, visual work, things like that, things that are very complicated and and gnarly, I'll go in and use, uh, Fable and and often times get, uh, better results. It does cost me a pretty penny, but um, but that's what

18:56it's good at. Uh, Opus, on the other hand, a lot of people are like, "Oh, you know, Fable's the latest and greatest from anthropic. Forget about Opus." Uh, actually, uh, I I I disagree and so so do a lot of the evals. Um, Opus is really really good at at a lot of things, sometimes better than Fable. Um, so it's first of all, it's a great

19:16writer. Um, and it's good at execution. So, good at communicating things like that. So, if I need copy or or notes or something like that, um, or a plan that I'm going to have to read, um, Opus is very good. U,, so is Grock, by the way. But, um, I just want to say that that they're both good at that. Um, GPT has a few new models. You have uh 5.6 soul uh which is which we have found to be really good at planning and reading code bases. Uh Luna uh not as strong of a model but uh

19:50decent quality per dollar at this point. I would recommend Grock or Composer to be honest. Um they're just better bang for the buck. But um but it is nice to see GPT um or or um OpenAI come out with a a cost-ffective model. And Composer 2.5 is the model I use the most actually. um we get to see our our our internal stats and um it's a very specialized model. So it's um it's coding specific uh or specific to all software engineering tasks really not just coding. And um and what's nice about it is it's very fast. Uh it's very efficient and um you know a lot of

20:28people think of it as like a a weaker model. That's that's actually not really true. Um it's just very much scoped to software engineering tasks. So, it's what we call a specialized model. Uh, a lot of times when you use a really large general reasoning model, you're paying for uh a ton of parameters you don't need. You know, like some of these uh general reasoning models, they can teach you how to barbecue chicken and give you relationship advice. Um, that's why you're paying because there are a lot of parameters. It's like all knowing or whatever. And uh in fact, composer um is a is a model that has uh possibly way

21:05more parameters than some of these bigger models on coding and software engineering tasks specifically. So you're actually paying for more of the parameters you use. And that's um that's a good way to think about specialized models. Okay. So apart from choosing the model, you have uh a few different knobs that you get to control um as a as a cursor user. Um so one is the effort level. Um a lot of people will see like for example um gro 4.6 high or opus medium or you know um soul low. What that means what what what those words are referencing is how much effort the model is going to put into your task. So, uh,

21:49for example, um, if I have, uh, if I want to build out a feature and I turn up the effort level to extra high or max, what that means is that the model is going to actually do more loops around the thinking. So, it'll keep thinking, keep thinking, and it thinks it's done, it's going to keep thinking some more. And, um, obviously, you're paying for that. That is more token spend. Uh, but depending on what you're doing, you may want to lift that up. Uh, if you want to ensure a more thoughtful result. Uh so you get to choose that on on everyone. I'll show you how to do it

22:23in a second. Um the other knob you get to you get to play around with is fast versus standard. I just want to clarify this uh because some people are confused about what uh fast means. Fast doesn't mean that the model processes your request faster. It means that you jump to the front of the queue. So if there are a bunch of uh you know I mentioned that uh sometimes the harness has to communicate with the model and so there are a few different turns behind the scenes. Um on each turn it's basically uh it increases your time to first token so it puts you at the front of the line

22:58and um you know it costs uh quite a bit more um you know depending on which model you're using. Uh I I believe GPT is like 5x as much something like that. Um, so, uh, I would really only use it, uh, if you're in a demo like this one or, you know, if time is really of the essence. The last knob that you get to play around with is, uh, whether or not you want to, uh, kind of like delegate the model choice to, uh, an auto router,

23:28which, uh, we'll talk about in a second. And this means like you know um you don't have to go in and like change the model and you know turn all these knobs for every single prompt. Some people are nerds like me and they love doing it. Um I actually love you know finding favorites and knowing exactly which model I want to employ for each job and actually a lot of software engineers do that. Uh I would encourage you to play with that as well because it it teaches you quite a bit. Um but one of cursor's most popular features is the cursor router which lets uh you know it uses um

24:01a classifier and a lot of data to route uh your request to the best model for that job. That way you don't have to think about it. So um I just want to uh we'll we'll I'll show you this in the demo, but I just want to kind of put it face to the name here. This is the new cursor router. And what's cool about it is that it actually lets you it's got three flavors. Um, so, uh, you've got cost, balance, and intelligence. Those are your three, uh, three flavors, uh, for the router. Cost is going to

24:33optimize specifically for token spend. So, if you're, uh, more price sensitive or your company's more price sensitive, it'll make sure that's kind of like what it's optimizing for. It doesn't mean that it won't, you know, if needed, uh, you know, give a task to a beefier model. It just means that it'll kind of like the default will be to try to find cost savings on each run. Balance, which

24:54is the default, it's what we recommend. It kind of like takes all uh factors in and and chooses the right model generally speaking. And intelligence sends like the harder turns to the most capable models. Um so uh we actually do obviously we have like a lot of internal data around cursor usage clearly. And so um uh and and by the way, I just want to call out to the admins out there um

25:19privacy settings super 100% respected. So if your privacy settings are on uh I wouldn't don't don't worry about it. It's 100% zero data retention. Um but for anyone who does share their data with us or for kind of on our end our internal um use cases, we've accumulated a ton of data that we use to route to the best model for the job. Um and this

25:40isn't just true about the cursor router. like the reason people find such gains around um like you know bug bug fixing and and feature development etc is because we've like I said tuned every harness for the job. So um the last thing I'll say is that um for again for admins in the room um if you employ this as a kind of an orwide policy we see overnight teams save 30 to 60%. Uh literally overnight just by making this

26:13like a policy. So we'll talk about that. I'll show you guys um your options [clears throat] there. Okay. Um I'm going to give you the biggest tip that you're going to get today. right now. This is by far the most helpful thing you'll hear. So, if you want to uh to really nail your nail it with the with the um with the agent, you need to plan your shop. It's so easy to go in, we all do it, we're all at this point really used to using AI. It's so easy to go in and be lazy about it and just say fix the off or um you know add this feature

26:54because there's no friction to the input. In fact, I speak into it all the time and um that's fine. That is kind of one of the benefits of AI. Uh but you have to remember that the the AI cannot literally read your mind. Um it does a really good job at inferring what you're trying to say with five words, but it can't read your mind. And so what ends up happening, a couple things happen under the hood. First of all, the model has to go around and try to figure out what you're talking about. So in a codebase, that means actually like navigating the whole, not the whole

27:28codebase, but really trying to like do a wide sweep and say, "What is he talking about?" That's the first thing. Second thing is it's going to uh take a guess because it doesn't want to bother you. And um and then you're going to come back and say, "That's not what I wanted." and you're going to say, "No, no, no, no, no, no, no. This is what I wanted." Now, you have to remember uh that first slide I showed where the um the context window grows with each with each turn. You have to remember that from that point on, the agent is carrying around all of this context. So

28:02that long windy road of not this that all of those get carried around like like baggage uh with every single turn that is actually going to cost you money over time because every turn is like feeding all this extra context from like failed attempts uh turns gone wrong. Um, but more importantly, with each turn, I mentioned the model has amnesia. With each turn, the model has to like

28:29decipher that you, oh, wait, wait, wait. We decide not to go down that route, right? And so, it's really hard to get out of the weeds. Um, I'm not a golfer, so I have no idea who this poor guy is on the right here. But, uh, as you can see, it's going to take a lot more work to get out of the weeds. And so the best thing you can do by far is before

28:49prompting the agent, just plan the shot. You can plan it with the agent, and I'm going to show you how to do that uh today. But as soon as you and the agent both clarify exactly what it is you're looking for, from there, the agent will take and it'll be just like straight down the runway. I don't know. Okay. Um moving on, I want to show you some examples. So uh here we have the same tasks with two different prompts. Now these are both using uh composer 2.5 fast. Uh composer 2.5 like I said is basically it's going to be like the most efficient coding model out there. Um you get to use it

29:27through cursor. Uh it's one of our models and um and so on the left you'll see kind of a vague prompt. It says our API service is throwing errors in production. Can you fix the error handling? Okay. So, little pop quiz. Um, I don't think you guys can come off mute, but uh, what do you think is going on here? Like, what is wrong with this

29:47prompt? One thing that's wrong with the prompt is you're being very vague about what errors you're talking about. So, now the agent has to figure out, how am I going to go out and find the errors? I need to maybe hook up to uh an MCP to go look at the logs or I maybe I need to run it locally and see if it reproduces the logs. So, it has to actually think with the model about how it's going to figure out what errors you're talking about. Um then if there are multiple errors, you know, a lot of production services have a lot of errors that don't matter, uh

30:23hundreds sometimes of errors in production. It needs to figure out which ones you even want to solve. Are we solving all of these? because that's a lot of work and that's going to fill up the context window because that's like a 100 different tasks. Um, and then uh it it doesn't know where in the code the errors are coming from. So that's a vague prompt. Uh, in this case it costs like 10x as much to figure this one out um as the prompt on the right which uh has a couple of important anchors for the agent. You have an app mention of a

30:54file which you can do through cursor. This is like a super big hack is if you know the folder the file that you're talk that that you want to anchor in or or even like the log um etc. you can actually atmention it and it'll pull in that context and it'll be a great starting point for the agent so it doesn't have to like crawl the entire codebase um and figure out what you're talking about. Um we also explain exactly uh which error is being thrown. And um you know another tip I have for you is if you have the error log just copy paste it in. A lot of people will paste

31:32the entire like file with like you know hundreds of lines of of logging. Uh that is then context that the agent is carrying around on every turn. So, uh, choose like say the three lines or 20 lines that are really important for for context and you can paste those in and that helps the agent get off to the

31:51right start. So, do a little work there. Um, and uh, yeah, this uh, this was yeah 10 to 12x uh, gain. All right, let's take a look at the difference between um, Fable and uh, Composer. So on on instead of uh instead of using composer for the vague prompt, let's use fable. Um so uh again we we weren't specific um

32:17but now we chose a very expensive model. So every single token is priced you know uh you know 20x more let's say than the tokens on the right. Um you can think of each lever as being worth like 12 to 15% in gains. Uh, but together these all compounded to 175x. Um, so it adds up. Um, and just remember that these turns do cycle. Uh, so I can't stress this enough. It's really helpful to be more

32:44precise about what it is that you want. All right, I want to give you guys some examples of good prompting versus bad prompting. First, let's look at specificity. So, like I said, if you say something like fix off, that's very vague. It's very general. It's got to figure out, it's got to go in and do some digging and figure out like, well, what's broken? maybe four things are broken if you think about it. So, you know, it's it's carrying a lot of ambiguity uh through each turn. And so, a better example would be fix the null check in this file this line. You might

33:14not know the specifics. That's okay. Even something specific like hey, you know, login is failing for users with aedu address. Something like that is just as helpful. Um, okay. Focused context. So rather than saying check the whole codebase, you can actually atmention a folder um and and ask it to uh kind of scope to that. Um another big

33:38tip is to limit it to one task per turn. So uh it's very tempting to just say, hey, do everything. Um but uh that's a lot for the uh agent to carry around. Um I'll show you multitask uh during our demo, which is just for this use case. Um, so, uh, but otherwise I I highly encourage you to kind of like scope it to one task per run. Um, another thing that's really really helpful, agents are great at checking against success criteria. And so, uh, if you tell it, hey, look, you know, this is what good

34:13looks like, this is what bad looks like. um make sure all the tests are passing, make sure that we match the spec. These directives help the agent check check against um the success criteria uh as it as it works. And so um it kind of keeps the agent very focused and is more likely to result in kind of a one shot where you take a look at the result and looks exactly like what you wanted. Um and lastly, uh yeah, you don't paste like a 30,000 line file. Um, that's just too much context to be carrying around the entire time. Um, okay. Uh, couple more tips and tricks. Um, start a new

34:54chat per task. Uh, one of the greatest sins in AI development, which I see even experts doing sometimes, is like nursing a chat forever. Um, I hear friends say this all the time. This chat has like all my history. It has like all the information. Um, and so you don't want to let go of that chat because it's precious. It has all the context. And so if you keep pinging this chat for like three months, uh, you're going to end up with um, first of all, it's it's it's going to all the details you think you're keeping, you're not keeping. Um, those all get compacted. Uh, also uh,

35:33you don't have room for new context. So please, uh, for the love of God, don't do this to yourself. Um, start a new chat. And if you need to uh tip number two, you can atmention a past chat. And this gives the agent an efficient pointer to that chat. And this is such a hack. Uh it's really important. You have great context in these two chats. Start this new chat and just at mention those two chats. Pull them in. Um and the agent will be able to crawl those efficiently. Um be careful with your rules and cursor. If you're an advanced cursor user or part of a team, you

36:05probably have rules set up. I have rules set up, too. Um, just make sure that um, your always on rules are short because they're in every single turn. So, don't make giant rules. Uh, keep them concise. Make sure that your MCP servers you're still using. If not, you can, um, remove them. Uh, kind of an audit. And, um, skills are very efficient ways of telling the agent how to do something because the agent only knows the title and the description and so it doesn't pull in the whole skill body until it needs it. You can think of it as like a bookshelf where it like pulls the book

36:37off and reads it into context only if needed. It's a very efficient way um to uh give agents kind of like workflows and and um yeah. All right. Um lastly, uh a couple golden paths. Let's say you have a small direct task. Just name the scope or the file or the folder or the component. And you can usually do that with one prompt. Oftent times with composer that's easy. If you have kind of a vague unfamiliar area, start with ask mode. Get some context in uh pull in some context with the agent, then move to plan mode. Uh we'll we'll I'll show you guys this in a second. And then you

37:14can build. Um all right. Uh building out a feature. So uh first of all, pull in all the relevant context, go into plan mode, and then split into subtasks. Uh I highly recommend you do that. Um or you can and then you can use multi multitask mode a lot easier. Uh for a refactor, this is where I would say um be very clear about what success looks like. Um I use test-driven development for refactors. I make sure all the tests are passing before the refactor. If you don't have tests, just ask the agent to to you know build out a test suite and then you make sure all the tests are

37:50passing before the refactor and then you then the agent can check against those tests and um and keep working until all the tests pass uh with the with the new code. Um you can also use bugbot uh which we'll talk about briefly. Um, it's one of our most popular features and Bugbot uh checks your PRs. Um, or you can even use it locally to check your changes and look for bugs before you actually uh push them um or merge them in. Uh, for a really hard bug, um, use debug mode. Agents like to guess. Um, that's kind of built into their DNA. Uh, they're they're they're guessers. And so

38:30with a really pesky bug, it can be really frustrating. They'll be like, "Fix the bug." and then you go in, you're like, "The bug is still here." Um, it happens to us all the time. So, we built Cursor built a mode called debug mode that uses deterministic testing. So, the agent will actually have you reproduce the bug. It'll like capture logs. It's a very efficient way of fixing a bug and it'll save you a ton

38:50on tokens. All right. Uh, before we jump into the demo, Martin, uh, are there some some questions in the chat? Yeah, there are some questions. The first one here is curious when you switch a model in between a conversation, how does it impact efforts and cost? Great question. So, um I'll give you the real answer first and then I'll give you the practical answer. The real answer is that um it basically uh has to rebuild the cache, right? And the context. So um let's say you're working with um Opus and you decide to switch to Grock. Uh these model providers like live in different places. They're different

39:39model providers. So different companies um and so the caching is per model. Um so this even happens if you go you know Fable to um to Opus or Grock to Composer. And so um the cache gets busted and gets recreated. You might be saying, "Oh man, you know, I was saving so much money with the cash." Uh, in reality, there are so many turns that happen behind the scenes that it's a really small blip to be honest. Um, it's just one turn where it like, you know, writes to cash. So, uh, not going to kill you. Um, I would I would say don't let it hold you back. Um, but also it

40:14does need to feed um the context back into uh the model fresh. Any other questions? Yeah. So, thank you for that answer. Extremely helpful. Another one is more so now of instead of manually changing the model, if we use cursor router instead. This is a question from Nathan that I already answered, but could be cool to hear for everyone else already here is as far as actually routing to the specific models. How does the input cache work if you know it changes models behind the scenes uh all in one turn? If

40:51that makes sense there, Santi. Yeah. So, you're asking how does the how does it how does the cursor router affect caching given that it's it's changing the models per turn. Exactly. Yeah. Yeah. So, uh it's a similar situation. Um the cursor router is one of our priorities internally is to help people with spend. Uh it's always been kind of one of our pillars at at Cursor and now

41:18at SpaceX AI. We care a lot about this. Um it's one of the biggest problems in building with AI right now um across teams. And so uh you can rest assured that our priorities have been specifically to try to like pull down costs and you can see this um with our clients. I mean it's like overnight with the cursor router. So um it's definitely going to be the net the net uh net is

41:41going to be positive uh very positive. So uh in terms of like what's actually happening under the hood, it's what I just explained. It's basically in the event of how it's not going to like on every single turn behind the scenes like switch models like this. Um it's going to take a look at your prompt and say okay um you know uh he or she is asking for a refactor. Uh we know internally the best model for this type of refactor uh because we've run so many eval. We're going to route it there. And then let's say there's like a little bug with the refactor after rather than routing it to

42:17like you know some big model for example with a huge context window we'll say oh my god this is like this is a discrete task this can be handled um a lot more um cost effectively with a smaller model and so we'll route it there but it's not happening like behind the scenes like

42:31that. Cool. Uh there are a lot of more questions but I can also answer those asynchronously. I don't want to eat into your demo time here, so just let me know. Of course. Thank you. Um, okay. Well, we might have some time here uh at the end as well. All right. So, um, a quick orientation for anyone new. Um, this is cursor. Um, it's, you know, this is like a a pretty fresh design from this year. It puts the agent front and center. For anyone who hasn't used cursor in a couple years, you might remember like early innovations around tab and autocomplete and the IDE. That was a whole another

43:11thing. You can actually still access a lot of those features in the top right. Um, but this is the agent window. Uh, we decided to put it center stage because this is where most of the activity was living internally and we find our customers love this as well. Um, so uh couple things we won't cover today. Um, cursor is not just a way to build locally with agents. You can actually build uh with in the cloud. So, agents have their own virtual machine, their own computer, and are able to actually navigate the whole computer the way a human would, check their work, show you screenshots and screen recordings. And,

43:49um, and because we have that feature, uh, which are the cloud agents, uh, you can actually build systems called automations around this. And so um this is an incred it's one of the most powerful things that we've ever had in terms of building and I encourage you to go check out the other workshops um which talk about cloud agents and automations. Um we have thousands of automations running internally and it's the way we're able to ship uh so much code at high quality. But that is for another time. Um I just wanted to call that out for anyone who doesn't know. Um the areas we're most interested in today

44:25are if you go here, you can actually see you can pull up the model picker in full here. So um you can see right now I'm on rock 4.6 high. Uh as a reminder, high just means high effort. And if I want to change the effort, different models have different effort levels. Um generally from low to extra high or max. Um and here's the fast mode to the front of the queue. um which I guess I'll turn on since we're in a demo. Um so, uh the other thing I want to call out is this is the um model picker. If you select it, you'll be able to optimize for uh

45:04anything that you might care about. Um I usually leave it on balance by default. All right. So, um I'm going to show you uh the project we're working on. If you go up here, um, and you go to browser, I have it running here. Oh, no, I don't. Okay, one second. Give me one sec. Um, I believe if I run

45:44all right, we're up and running. So, um, over here in the top right, by the way, for anyone who's new, you have actually a ton of tools. Um, and one of the tools is, um, is, uh, a browser. And this is like a fullyfledged browser, by the way. I can like search the internet and pull in context off the internet. Um, but it's really helpful if I'm working locally um with an agent because uh I can actually point to things uh and click things and pull context in to the agent and the agent can also uh kind of pair with me here locally. So uh this

46:18project u this is a huge repository. It's an open source project called tlddraw tlddraw.com. Um it's uh I like to use it in demos because it's such a big repository. It's got like hundreds of tests, hundreds of developers, million plus lines of code. Uh because I just like showing cursor working in a in a real project. It's really it's really good at at managing that. It's not just pet projects. Um so um what I would like to build is and I already ran these for you knowing we would be short on time, but uh we'll go through and and run it so you can see. Um here you can see four

46:53other modes besides the default builder mode. Um, and so ask mode, uh, we're going to start in ask mode and we're gonna say, is there anything I should know if I want to add a custom color picker to the style panel? Now, like I said, I already ran this. Um, I'll show you in a second. Um, but uh, this is this was routed to uh, Brock 4.6 and so it's going to have that general intelligence. I just want to call out that composer uh, 2.5 is actually very very good at this as well. Um, so if you want to have like extra efficiency gains, you can switch to composer 2.5

47:28for ask mode. Uh, it's great at navigating the codebase. Ask mode is um, a great feature because it's totally safe. It's not going to uh, write any code. It's not going to run any commands that mutate anything. It's like a completely safe space to ask questions. And uh, it's not just to ask questions for your own benefit. is also to ask questions for the agents benefit because it's able to pull in context um into uh the context window which then can be

48:00referenced by another uh by another run. Um so what I like to do is I'm if I'm going to develop a feature or or fix a bug, I like to ask and pull all that in. Um, so, uh, like I said, I already, uh, ran this once and it pulled in a ton of context for me around what the current state of colors even are. Um, you know, what are some gotchas, uh, about, you know, if if I were to build out this

48:27feature, some things to look out for. And, um, this is going to be kind of your recon step. And it's like I said, it's not just a formality because uh then we can create a a new uh chat. And actually before I before I do that, I just want to show you guys something here. See if I can get this zoom thing out of the way. Okay, cool. Um if you click the bottom right there, uh you'll

48:52see a little kind of ring filling up. That is the context window. We surface that information for you to check out. Uh don't stress too much about it. um cursor is really good at managing it if you do get to 90%. Um and and so it's okay, but if you want to keep an eye down there, um it's helpful to know kind of when to switch over. Um I just want to show you practically speaking um where the context went. I mentioned the system prompt um this is tuned for each model and um by by cursor and it ensures that the model operates, you know, with

49:29the right context. Um, tool definitions are uh uh tools that the that the agent has at its disposal. So like things like being able to GP or being able to look something up on the web. It basically is a suite of features that the model can recommend employing um for for the agent depending on the context. Okay, let's talk about rules. This is important. Um I've seen a lot of people make this mistake. uh rules should be short and direct because every single turn is aware of them. It they get loaded into every single turn and um in full. So uh for example, I have a rule um if I go up

50:09here to customize and I go over to rules, I have a rule that says never use emojis uh in the place of icons in the chat. And um because I hate when a all all AI agents do that. It drives me nuts. So um that's a very short directive. It's like, hey, no emojis in the UI. Use an icon set instead. Um, that's a great example of a rule because I want it across all my projects. But you can also scope rules specifically to your uh one project as opposed to all your projects. Um, and teams can actually share or even enforce rules at the admin level. So that's super

50:45helpful. Um, so okay, skills. uh skills are um more complex uh typically more complex like recipes or or or um or kind of workflows. So uh for example um a a good example of a skill would be like you know take a plan and uh be the devil's advocate. So go back and forth a few times like look at

51:13problems from different uh perspectives. I have a skill called good UIUX design which looks at references of really good designers and tries to take uh psychology into account when designing features. Um you can make a skill or a rule or a rule with the create skill or create rule commands uh or sorry built-in skills. Uh these are cursor ships with these and they make it really easy to either create a skill just you know you can just describe it or you can take something that the agent did and say oo you nailed it. that's exactly what I want. Can you create a skill around that so we can run that back and

51:49do it do it again later? Um, so skills are really efficient because like I said, it's like a bookshelf where the agent just sees the title and sees the description and knows when to invoke it or you can invoke it yourself uh manually. So, it's very efficient. Um, and then uh yeah, MCPs, like I said, get rid of any you don't need or your team doesn't need. um and sub agent definitions. That's uh you can take a look at the workshops. That's a more advanced topic. Then your conversation will fill up and when it gets to 90%, it'll compact. All right. So, let's go uh start a new chat because like I said,

52:25you you always want to start fresh when you can. And so, um this time I'm going to hit the at symbol and I'll see that I can reference specific context. And one of the cool things I can do is reference chats. And so I can go in here and say, you know, custom color picker guidance. And this is a pointer. It's not like loading all the text in. It's a pointer to uh the conversation that uh we just had. And and so I can start there and I can say, you know, add a custom color picker. And you know, I won't you don't have to sit here and watch me type, but

52:58like I I'll copy paste some of the things I would probably add so you can see. So I would add some constraints like make sure that you know this is where the change happens. Make sure that you know we maintain one grid. Here's some output um uh that I expect. And then I can um you know pull in a skill like this. And what I'll want to do is go into plan mode for this one. And by the way I can shift tab to cycle through these. But plan mode uh is another great um readonly feature that will it's kind of like having a spec meeting with like

53:34your PM or your technical PM and like the designer and the engineer. You all gather together. It's kind of like that uh with your agent and you make some big decisions. And so the plan mode will ask you clarifying questions. You can respond to them and so you can actually let me see if I can pull one up here so you can see. It'll create a plan for you, a technical spec um with diagrams

53:56sometimes and and important decisions. Um and you can go in here and you can actually edit this plan live or work with the agent to get it to the point where where you're ready to code at which point uh you'll see a button up here. We already built this, but you'll see a button to build it and you can choose which agent to use there. Um, last thing I'll show, uh, is I have a little X-ray vision. Um, I created a skill actually that does this. Um, it shows me a context report. So, for anyone who hasn't used Canvas, uh, I don't know why it's not coming up. Well,

54:33there's a cool command called canvas. Um, I'm not sure why. Maybe it's because I already pulled it in. Um, a cool command called canvas that ships with cursor that lets you create these like really cool React um, apps. And so I created one to take a look at the context. And I can go in here and I can say, you know, okay, that's how many um tokens went into the cursor agent instructions. Um I can go in and take a look at uh the tool definitions, the rules, the skills, and so this is a great place to audit. I'm like, you know, okay, what actually all of our

55:07rules are pretty reasonable. They're not too big. So So we're good there. Um and I can check the MCPS. The la the other thing I can do is I can go down to the conversation, pop open the turns and see, you know, my message, but then also like, oh, the the agent went in and grepped super efficiently. Um, read some files into context, etc. And I can I can kind of inspect here. I al I also built a debug with agent tool um here which will um actually work with an agent to try to scrutinize and see where we could get efficiency gains like hey you should

55:40probably shorten this rule or like you know you should have been more clear with your context um that would have saved you a lot of tokens. Um you can create a tool like this too um uh with uh create skill. Um and so that for for for the real nerds in the room like me um that is like X-ray vision into the actual context. Uh the last thing I'll show here is um I want to show uh some of the tools you have on cursor.com. So cursor.com don't is our our whole team and we're encouraged to try a bunch of things. Uh I'm going to filter by just

56:15me here. Um believe this is cool. So um you have a usage tab on cursor.com um on your teams and you can actually go in and uh there's a lot of cool things you can filter by here but um you can actually go in and see some of these runs. So for example um this run right here was when I asked Fable to uh implement the feature. Um it

56:45cost me $32. Uh which I then compared to um the uh composer run which cost me 12. Uh I did this after the planning um after ask I did I did ask plan. So um I planned with Grock 66 cents I built with composer 12 cents when compared with $32. Um so kind of a big difference. Uh you

57:10can also hover this kind of a pro tip. you can hover here and see like where the hell were all these tokens burned and you can see that a lot of it was from the cache which is more efficient. Um so that's kind of one one cool thing you can do there. Um the other thing you can do from here is you can go to cursor router if you're an admin on a team. Um you can you know enable and disable different modes. Like I said, um people, you know, use they see 30 to 60% gains overnight when they when they actually impose u auto. There's two there are two

57:44ways to impose it. Uh one is like softly. Um so, uh that just like defaults to auto and the other one is it locks to auto. I would encourage you to let your engineers use the models they want to use. Um highly encourage you to do that. It's very helpful to be able to

58:00switch models and decide what you want. Um get some agency. But if you do soft, it'll at least default to auto so that someone doesn't get stuck on Fable for like two weeks. Um, so, okay. Uh, I know we're at time. Uh, if there are no major questions, Martin, how how are we looking in there? We are doing good. There are many questions. I love that everyone is active and engaged. Try my best to answer as many as possible. Um maybe just to highlight this one, uh Jonathan mentioned, sorry if I missed this, but is it better to use plan mode with a more expensive model then use a more

58:36efficient model for implementation like composer 2.5? Yes. Uh thanks for calling that out. So uh plan mode um is is great for a general reasoning model uh because when you're planning something, you kind of need to weigh a lot of pros and cons. um for a feature, for a refactor. And so um you really want to employ like a a really solid general reasoning model for that. Um try Grock, try Opus. Um and uh and that way you it it's going to pay dividends and it's also like a much shorter loop. So um yeah I would do composer for asking then I would choose grock or opus or something similar maybe

59:22GPT for planning and then I would choose uh typically composer for the buildout. The only time when I use a general reasoning model for the buildout is if it's like extremely complex and decisions are going to have to be made along the way that are really important. Like for a big refactor I'll widen the context window as big as I can. you have that option in curs in cursor. Um I'll do a million for example for the context window and then I'll um use a really

59:49beefy model and it's usually worth it.

59:56All right, I think we're probably at time. Um just want to thank uh Martin for for being a legend in the chat um and uh and for answering all those questions. Sorry if we didn't get around to your question, but um I highly encourage you guys to go and check out the other workshops we have available. And thank you all for joining. Um I know that was a whole hour and I hope that you got a lot out of it. Um if you guys uh have any feedback, feel free to send it our way. And uh we we look forward to

1:00:25the next one. Five room.