Claude Engineering Lead: "Stop Using Claude Like It’s 2024"

0:00If all we're doing is like summarize this email for me, you're not really benefiting from the new intelligence. This is Felix Reeseberg. His team built Claude co-work in about a week and a half. Now he leads engineering for Claude at Anthropic and he doesn't write code anymore. I'm just a conveyor belt for Claude. Claude needs to have access to all the crash data and then needs to like figure out itself what it needs to fix, what it

0:20doesn't need to fix. Is there still anything you would not trust an agent with? I rarely want the current session to judge itself. That I think is still really iffy. How do I protect myself from AI just confidently presenting mistakes to me? It happens to a lot of us. I have like a very practical easy answer. Felix, thank you so much. I am very excited to have you. As a user of Claude, like an avid user, we're using Claude for all the operations on this channel. Every team member has their own project for the things they're responsible for. And as someone who runs, let's it's a pretty big team for a

1:00creator. I have 35 people and I want them to be as efficient as possible. I want to produce as much as possible. But also something that I've noticed this year, we were able to launch I think three new social media channels. So we launched a newsletter. We launched a LinkedIn a year and a half ago because you know it's text based. So AI has enabled me in many many ways. And the goal of this conversation is to make me even more advanced and all the viewers

1:26and all the listeners. I mean, from what what I've seen you do, it seems like you're pretty advanced as it is. But there is no borderline, right, to to being as true as possible. That's true. Yeah. [laughter] I think I'm discovering that about myself every single day. Like I'm sort of one of the most fun things of working on claw by far is this notion of I built this product and in comparison to my other products, with my other products, it was pretty clear to me what you're going to use this for, right? Like the example I always go with is I worked on Slack and

1:51like I kind of have a feeling of like what are you going to do with this? The thing that's magical about working on Claude is that I see people really fly and do amazing things that I didn't consider possible and that's like a that's like for a product builder that's such a treat, you know, to see people

2:04fly with this thing that you made. Is there something you've seen that's super ambitious and you're like, "Oh my god, I want to copy this." I'm not sure about copying, but two days ago um there's there's I think he's actually a product manager by trade. His name is Pavo and he pointed Claude at a large amount of NASA data and asked Claude to like find them a new planet and it did. So apparently it is not confirmed yet. NASA is working on actually confirming it. I'm on a little shaky ground here because I'm not an astrophysicist of any kind. But apparently Claude may have found a new

2:35planet we didn't know of before. This is fascinating. Okay. And it's the kind of thing that I would have never considered. But if you hear the story of I think for us who work maybe on more knowledgework things, right? The idea of like point AI at a large amount of data and you're going to find interesting things that in like a in like an in isolation seems pretty reasonable, right? Like something we do every single day. But the idea of like looking at all of the public data we have about space and considering at a similar problem of like go and see if you find anything in

3:02here we haven't noticed before. So beautiful. Can you talk to me about your day? You help build claude co-work but you said you're not coding anymore. So what happens in the morning? What do you do? Is that like your agents are briefing you on what's coming or what is it? That is true. My agents do brief me a

3:19little bit on what's coming for the day. I've really leveled up in sort of altitude on the problems that I'm thinking about. Right? Like sometimes sometimes I feel like these days I'm I'm much more concerned with human emotion, empathy, trying to understand how people work in a way that I think I was less so as a as a classic engineering manager a couple of years ago. So I think for me what I'm really discovering is that the amount of times I have to reach out to other professions to get their help has really dwindled down. I'm now this weird mix between like a product manager, an

3:48engineering manager, just a traditional people manager, a data scientist. Right? To answer your question very specifically, most days I'm I'm busy with a lot of prototyping, right? We just try a lot of things. We sort of like think about stuff. We sleep over it for a couple of nights, try to figure out if it's a good idea or a bad idea, and then we just go tell Claw to like build all the various variations and ideas we had. And then we need to do critique, try to figure out which ones of these have good ideas, which ones are bad. But I think prototyping is really

4:14what I spend most of my day on. How do you decide what is a good idea? Like what are the signals you're looking at? This has been difficult for people for a long time now and I'm not sure that I have all the answers to like what is a good idea, what is a bad idea. I think generally speaking, the way we look at good or bad ideas, the calculus has changed, right? Like if you think about products we've built in the past for the last 1015 years, we we always

4:37made implicit cost decisions, right? Like we sort of had to figure out, okay, especially for software, can I build this? What is the maintenance cost? What is the ongoing cost? Um how much how much time do I invest in this? What is the opportunity cost? It's all these decisions that engineering managers make to figure out, do I go and build this? Yes or no? And I think now we can fast forward to the end state much quicker and be like, all right, just build me a prototype. Let me play with this. Let's try it out. I think human attention is still rather limited. Let's say you, for

5:03instance, you have a lot of taste in AI, right? Like I can probably show you maybe one thing a day. I can easily build a thousand. I can't show you a thousand. Even if you're willing to sit down with me for all day, like at some point your eyes are going to glaze over and you're like, "Okay, I'm kind of done with this." So, what what makes a good product? I think especially now that we live in this AI world, there's this old saying that like perfect is sort of when you can't take anything away anymore and less so about like building a product that has every single feature under the

5:27sun. Like we can make as many buttons as you want us to, but I think good products actually have fewer buttons. I kind of know whether or not a product really works if we put it in front of a few people and you see things click for them rather quickly and they can do

5:40things that they couldn't do before. That that's really the signal I'm looking for. And how do you think about the underlying problem that you're solving? So for example, if you're came to my company now, how would you think about the tasks that you would fully automate with AI versus just leave it to humans versus maybe just automating parts of

5:56it? The first thing I want to say is that I think about automation and augmentation in slightly different ways. Automation can be really empowering if you automate away tasks that sort of keep you from doing work that that you enjoy or if it's like expensive and automating it a way like saves you a lot of saves you a lot of money or time. In those cases, automation is is useful. And I think it's really where AI currently probably provides most of the economic value, right? Like when we evaluate, is this worth it or not? We usually look at automation. The area I find most interesting though and I think where

6:26people really do things that they weren't possible with AI before is augmentation. It's this idea of AI is helping you do things that you couldn't do by hand because maybe you were lacking skill or techniques or um just the raw knowledge to do something. And when I go to company, the thing that is often most impressive to me is less so what the tool does and more so how you

6:49change your workflow. Does that make any sense? Yeah. AI is an interesting product because to really harness it, you kind of have to rethink how you do things. You have to like think about your outcomes. You have to figure out what does good mean, what does bad mean, like how do I evaluate whether or not this thing that I'm asking for, like whether the result is good. And the people who impress me the most are usually the ones who are very good at coming up with like very good at

7:11reflecting on their own workflows. Yeah. But it's so hard when I'm thinking about my work for example and a lot of people use AI for drafting texts like how do you define a good text like is it still possible for like knowledge work cuz for coding I guess it's possible right cuz you know what you want you don't want any bugs but when it comes to knowledge work and like the result of human creativity how do you even define what's good for the agent? Yeah, it's such a tricky problem. Like the the example I often go with is let's say your job is to like report on the

7:41weather, right? And you're like, "Agent, please figure out what the temperature is outside." There's like two ways to look at that because you have to rethink what good means, right? Like if the model just guesses a temperature and is correct, that should not be a passing grade. That should be bad. We should like rethink what we're doing here. But if the model is actually off by like half a degree or something, but it like um really figured out the right location, figured out some reputable data sources and like sort of the the the way it got there was reputable and good, that should actually be better

8:07than like a gas good result, if that makes any sense. But for writing and like all the other outputs where you feel like a lot of taste is involved and it's like a little hard to like prove or verify whether or not the result is like better than another. It is tricky, but I think I really enjoy talking to very good writers who are also a little AI

8:26friendly. What do they say? It is it is fascinating to watch them work. My friend Adam Davidson, who was a staff writer at a number of like pretty good publications, he has all of these various prompts for Claude where he's like, "Okay, like take my argument and put it to like magnify it by 100. Um,

8:42just mess with my text a little bit. Like what are the things you disagree with the most? Like what is the strongest counterargument?" It's interesting because I think to me as someone who's not a writer I think the naive approach would be given a text claude make this good and I think what I'm discovering more and more is that for knowledge work the secret sauce is like less and asking for a good output and sort of doing one step in between which is the best in the field how do they work and can we automate those steps this would kind of mimic their workflow but still the taste isn't there that's

9:15why I think you still need a lot of yeah I think That's true. I think that's true. But the interesting thing is that if you talk to really good writers, most of them would disagree that the text is really good just because they have like the magic touch. They would often say, "No, no,

9:28no. This is the result of hard work." Interesting. Right. Like it's this idea of like most artists have a process and many of them would actually probably find it I don't want to say insulting but like if you the more I talk to artists and the more I explore this idea of like is your stuff just good because it comes natural to you and it just like happens to be good or do you have a process that you like refined over the years a lot of them are quite proud of their process and like this is very very hard work like there's a lot like a lot

9:54of like manual individual steps go into making this good. It's not like I sit down and suddenly there's a book that's great. And if you go to like writer studios, right, there's like a lot of white boards and like a lot of information. And I'm I'm noticing this across a number of different domains where Claude is quite receptive to being told not I want a result that looks like huh, but Claude, I want you to take the same steps that a professional in this field would take. This is like very high manage. So, let me bring this down a little bit. Like I think we're currently

10:23in the public seeing this quite a bit with like motion, video, and 3D where I think Opus 55 has made a pretty big jump. But the big jump that the public has made is to just tell Claw to use Blender, right? Just like Claude, please use the same tools that professional animators use. And now the results are

10:39quite good. The unlock really is here in the media and the technique and the tools and is less so like raw understanding of what makes a good animated movie. Let me pause here for a second because a lot of the work behind this podcast happens long before we hit record. For every guest, we already have a pretty heavy research process. Our internal clot agent does the first pass, past interviews, articles, public posts, and the topics the guest always gets asked about and the angles people usually miss. Then my producer goes deeper. She watches full interviews, reads industry pieces, fact checkcks everything, and

11:13builds the first version of the episode. Then it comes to me. I do my own research from my point of view and usually we go through the questions a few times before we record. That process gives us a lot of information what people are actually saying about the topic right now. So recently we added dig AI to that workflow. It gives us a layer we didn't really have before. The real human reaction around a topic what people are asking arguing about and still trying to understand. Before an interview, we can look across similar creators and ask, "What questions keep showing up in comments but still aren't

11:45being answered? What parts of the conversation are suddenly growing? What are people confused about that nobody has explained well yet?" And then we can go straight to the source. Ask Dex scans through billions of videos monthly and sources the actual videos, the actual comments, and the language people are using so we can see the pattern for ourselves instead of relying on another summary. That helps us sharpen the questions before we record and avoid asking the same things every other podcast has already covered. So now we're not only looking at what the guest has said before, we also know what the audience still wants answered. If you do

12:20content marketing, research, PR, and honestly anything where you need to understand what people are actually talking about, ask Dig is worth trying. The link is in the description. And we're back to the video. But I wonder for people like you mentioned this prompt act as if you're a professional in this field right and it's going to go on the internet and research but a lot of information is just not there. How do you trust the output? Like for example if I want to start another medium [snorts] or like do discover stars right and I have no idea about that but I just

12:50tell it act like an asset researcher. How do I protect myself from AI just confidently presenting mistakes to me? Yeah that's a good point. I think it still happens to a lot of us. Um I I have I have like a very practical easy answer which is adversarial reviews um which has proven quite effective in coding math and like various other areas where we can give you numerical results on like how effective it is. Adversarial reviews basically boil down to the idea of you've given given one claw a task spin off another 10 that try to like break this down. Their whole task is to attack this and prove to you why it's

13:24wrong. It's essentially the same concept as red teaming or short selling. Are you using the same model for that or are you want to use another model? Kind of depends on the task. This is a little context dependent. Um but usually the adversarial review. It is it is sometimes easier to find flaws than to create new things. Much like as it is with humans, right? It's much easier for me to say whether or not the new phone is good. It's much easier for me to do that than to design a new phone from scratch. So usually you can like take a step down in intelligence and still get

13:50very good results. That's the very practical advice. How would you prompt that? Like if there is a process already, what's the additional prompt that you add? So you have some kind of output, right? Like let's say it's a slide deck or a document. Um or in my case, it's you talked about this blog post that I wrote this morning. I just tell Claude very I'm I'm using Claude Projects, which is this new product that we have that allows you to do work in multiple threads with a little bit more ease. And I basically tell Claude, "Please fire off five threads and task each of them with like proving that this this post is

14:20wrong, that I'm thinking about it the wrong way, that I have my facts wrong, that I misunderstood something, and then they go off with a given goal of like break this down, you know, prove that I'm wrong." How do you then deal with all the output? Because I do the same, but I use I just use different models when I'm prepping for the interview. like tell me if I'm digging too deep into this or that. But then I'm giving so much different feedback and I'm like okay now I have to decide cuz this model gave me this this gave me something else and gave this gave me something completely

14:49different. How do you then decide [laughter] what to use? I think you're going to enjoy this because you can see where this is going. The answer is once again clawed. What we use in coding context is something we call a judge where you get the output from the people who are like the the agents whose job it was to attack your output and they're like all right here's our main points of criticisms and then you use yet another cloud and their job is your your job is to validate and verify and like opinionate whether or not this is valid feedback. M um and then what you can sometimes do is

15:19that ultimately um depending on how you set up your workflow the feedback that is valid should probably go back to the clot that actually made the thing and that claw too that agent too can be discerning and whether or not to act on feedback right because sometimes um and we see this in a coding context fairly often I also see it in like the knowledge work output that I make sometimes the feedback is like maybe reasonable but um has missed a point or it's actually not that actionable or is not that useful and That's usually when

15:47I know that my output is pretty good. That's one way. That's when you are really quite uncertain whether or not the output is good yourself and you can't easily check. But for a lot of a lot of things that I create with AI on the other hand, I actually don't need to fully understand the entire flow because as I mentioned like to me as a person, it is I do have opinions about almost anything I use every single day. I've remade my website with all of these 3D assets. I have no idea how to use Blender. I've never used it. But I can sort of judge whether or not this looks

16:13good, feels good, and that might be enough, right? like you sort of put yourself much more in a client position. For so many other things in our life, we don't really have a good way of verifying whether or not we get a good output other than like giving feedback to the person we're using, right? Like for a contractor or a lawyer or like for many people a doctor, right? So you sort of just put it in one of these two buckets like either you really have no idea what goods like good looks like there. My answer is more AI just throw more tokens at it. or you do have you do

16:41have an understanding what good looks like and then maybe that's enough and you don't really need to like have opinions about the process itself. Yeah. I think it it comes down to uh making sure you select only the top projects where you want to understand every single step instead of just want to control everything. Yeah. Yeah. It's like the old problem of like sort of being willing to let go, right? Like how to not micromanage. It's

17:03really hard. Yeah. Comes Yeah. Now we have this problem with agents. Is there any knowledge process that you have nailed with agents like you completely trust them? They just go I don't know maybe it's yeah fact checking like editorial factecking I fully trust now the output I get 100% if I ask claw to verify that all the facts that I'm stating in the slide deck and this presentation um it doesn't even need that many rounds right like it's not like I have 50 agents doing that I really trust one agent I'm like I need you to find primary sources for every number I mention I want you to

17:33verify that it's a good primary source I fully trust that 100%. When you say agent, do you have a special agent for factchecking that you've trained or is just an one agent that you're using for everything? It's just just a cloth thread. I feel like this is going to be a spicy take which I think you'll like. I think we're kind of over rotating on this agents thing. And I can back this up with a little bit like I think models are really good generalists and we've seen this now time and time again. Like when we first started looking at math as an industry, I think the the first company

18:03to really look at math like with with a very serious lens was Google Deep Mind and they trained a custom model and a custom harness and they unleashed it on the math Olympiad and they got a silver medal which at the time was state-of-the-art and then four or five months later all the major generalist

18:18models come along and win a gold medal. I understand why people use agents. I have empathy because it's easier for us humans to rationalize what AI can do if we sort of like put on a little hat like you're a data scientist, right? But that same data scientist has all the intelligence as like a fieldleading biologist or mathematician or what have

18:35you. And I think we're we're sometimes limiting ourselves in our use of AI if we sort of force ourselves to first like make up this cast of characters. So for a lot of these things that I just meant, I don't really have any agents who are being told who they are. But does the data matter because for every agent you upload certain data set or just because your cloud has access to all your tools it doesn't really matter what you feed to each particular age but then isn't it a lot of data for it to consider when answering each particular question that is true but it's quite discerning

19:06in what kind of data it wants to use right like I think the right approach here is not to make these slices of data and then give every single agent the slice of data that it wants because many of us if we want to be quite effective at our job we do actually need the full

19:19view on the entire business, right? We're not quite effective if we only get a small slice of data. Like the as an engineering manager, if you only give me code access, I'm probably going to be much less useful to you as opposed to if I also have customer data, access to customer calls. Sometimes I do want to sit in like a like a customer call like when we're trying to sell something, I

19:37want to have deep usage information. Ideally, I get business data about what is working, what isn't. And you want your agents to have the same range of data. Now, how do you deal with the fact that it's probably a lot of data? tools. You give them tools and you document really well what kind of data is behind this tool so that they don't actually

19:55have to ingest the whole thing, right? You're just like, if you do want business data, here's the tool to get business data. So, when you're setting this up, cuz I'm thinking like when you mentioned this, should I move away from all my projects that have access to all the like specific data? For example, I don't know, there's a project for ranking a guest and we have the whole uh description of the process and then all the old scripts there so that we know how each guest performed. But now that you're saying this, does that mean that

20:20I should move away from that project? Tell my claude, okay, we have this is a folder with all the scripts. If you want to look at them in notion, we have all the intelligence on how every podcast guest performed. And uh this is how we rank guests and just leave it there. And whenever I want to rank a guest, I just start a new chat without telling it anything and just tell it rank the guest and it's going to Yeah, I'm careful to be prescriptive in

20:42workflows. Like I always say, if it works for you, it works for you and I don't think you should change it. The problem is the bottles are getting better and I want to get better. So this been working for this year but a year is a lot. Yeah. Okay. Well, I mean if if you're looking for the recommendation that I

20:57give for like what do I do? Again, everyone should do whatever makes them happy and comfortable and what makes them powerful. But it is true that the models have gotten so much better at tool calling and discerning tool calling. Um and tool calling is this process of like the model knows when to use a tool, how to use a tool. Doesn't

21:12like get stuck in tool call loops. doesn't accidentally say, "Oh, I found a data tool call. Let me call this tool until there's no more data left." Right? Like um the models have gotten particularly good at efficiently and effectively using tools. And for that reason, I don't really make this a priority decision anymore and

21:29what kind of data do you need. Um it also allows me to like have more threads going and more sessions going without easier process. Much easier process. Do you still use skills? We do use skills quite a bit still. Although um our official anthropic guidance as well as my personal guidance is be really careful with skills. We often tell models things that were sort of relevant for like the last generation of models which time is so compressed in AI right that the halflife of a good skill is very short and sometimes what we see is if you completely remove the

21:59skill you actually get better output. Yeah I think journey said that like just forget about that just just prompt just prompt and see what happens. Yeah it is it is very effective. This is obviously most useful to people who have some kind of evil and have an idea of what good and bad looks like. But I'm pretty careful and very discerning and when when I hear skills and when I

22:17don't. Okay. You know, you know what a what what a great way to think about this is I like to consider the model experience as opposed to like user experience. And model experience is you pretend for a second that you're the model. You're claude, right? What kind of things would you want if you had fulfill whatever task you're being given? And you very quickly discover that you actually do want access to the raw data. You want access to a computer. You probably want

22:38to be able to install some software. I want to make some decisions myself without somebody telling me what they think already. I just want to see exactly what you said, raw data, and make my own decisions. Like I think the upfront cost is higher. Like if you like even a year and a half ago, I was in this world of like telling Claude, I found another crash, please go and fix the crash. Right? But it very quickly dawned on me, I'm just a conveyor belt for Claude, right? Like I go look at our crashes, pick one up, move it over, drop it off in front of Claude. I'm like,

23:07"Please fix this." I'm like, "Okay, this is this is no, I'm just going to claw it needs to have access to all the crash data and then needs to like figure out itself what it needs to fix, what it doesn't need to fix." The upfront cost is really worth it, but it's also like important, I think, to consider that there is upfront cost, but once you set this up, um, it it opens up so many doors for you, and that's where I find a lot of like real hard value coming out

23:27of AI. If you've been hearing about AI agents and wondering how you'd actually set up one, I put together a guide that walks you through your first one in about 20 minutes. It helps you choose what to automate and get it working with no coding required. You can get it through my free newsletter. The link is below. For someone who just who's listening to this conversation and who thought they would need like a major setup to start using AI correctly, would you say uh if somebody is starting just today, they're using it for knowledge work. They don't

23:55need all these tone of voice things. Like my business strategy, you just give it access to your email. It learns from it. From your tone of voice, you give access to your newsletters or whatever. So you don't need any of that. The best way to get started with claw today, I think, is step one, download the desktop app. It has a built-in browser where you can log into some system you're using and forget about connectors, forget about like downloading all the data, forget about like all kinds of things. Just let

24:21Claude use your computer a little bit. Sort of give away the keyboard. That gets you so far and really shows you what works and what doesn't work. And then at some point, you might discover, oh, Cloud is really helping me with whatever tool you're using, whatever internal or external tool you're using. And once you see that a workflow really works for you, you might consider, okay, can we give Claude something better than like having to manually click in this dashboard around? Can we give it some

24:43kind of connector? But just starting with a browser gets you so far. Okay. And when it just to wrap up with the agents, when you say agents, that that means you have just basically chats, right? And that's it. There there's no like specific agent who does research, no specific agent who does your tax returns. No, no. I I mean I do have projects so I do think about I do think about the different work items but I don't think

25:06about different roles. Mh. Right. So I don't I I never tell Claude you're an excellent whatever. Um I think this was this was very invogue sort of like 2024 we were like Claude you're an excellent engineer please go fix whatever problems are here. Um, but you still use that in prompting now. Act like something that we started with.

25:26Act like a top whatever. I specifically implore claw to like tell me what tools someone top of the field in this field would use and then I'm like I want you to use the same fields. Um, a problem most models still have today is that if you simply tell it data science is maybe a good example or like accounting. A lot of the models still have the problem that if you tell them to act like a very good accountant that they engage in a little bit of role play and role play is not really what we're going for. We want the techniques, the tools, the knowledge that a professional

25:55would use. We don't really want just the vocabulary, right? Like I don't I don't need claw to sound like I don't need to put on a costume. I actually wanted to talk to me like I'm five, but I wanted to use all the top tools in the field. Is there still anything you would not

26:09trust an agent with? That's a tricky question. we can just keep layering more validation on top. So I think the question for me is much more it's less about it's just about what do I not trust Claude with and much more about uh what things do I not trust one single session with. And I think the the list of things where I don't trust one single session is actually much larger than the list of like things I wouldn't trust Claude with. I rarely want the current session to judge itself. that I think is still really iffy like the current cloud session that is running wherever you have it in cloud

26:42code in cloud co-work in chat in even in a project the current session tends to be quite bad at being a being a judge of whether or not the work it did was quite good. So as much as I can, I try to find that outside signal and the outside signal in a pinch can be another claw session that has proven to be quite effective, but my my preference is to find some hard outside evidence for what works or what doesn't work. And sometimes you need to like trust the process a little bit and give Claude a little bit of time. Like for instance in your field, I'm kind of making this up,

27:11but I assume that maybe like the number of views or subscribers is like an important metric that you Oh, like the number of views that's that's what we're whenever I give a task to Claude, I tell it like we need to hit this number of views. Yeah. Go and try. But you that and that's that's [snorts] something that we're still like struggling with is this feedback loop, right? Because it's me giving feedback to my team and then my team actually going and posting

27:34something else, not what Claude gave us. It's like maybe 70% the same, but then the 30% that I've added is actually my taste. Yeah. How do you close that loop? A thing that has worked for us is there's probably some kind of place where this feedback usually happens. Is it usually in person or does it usually

27:50happen? It's for us is on Telegram. There isn't a native Claude connection. We have a bot that scrapes every chat, but it's still very manual. I have to trigger Claude to look at the bot, look at the feedback, and it's still not it's it's not there. even like looking at something we've sent, right? Yeah. Because we do we do obviously again in a coding context we have a lot of engineers who you sort of know which engineer you want to review a certain thing that you're working on and the

28:19kind of things that they look for. Claude is very good at figuring out from the feedback. It's pretty good like given all of this feedback that I've read feedback on Slack, right? Yeah. On GitHub, Slack, wherever else the feedback might happen. And you didn't tell it to look at feedback specifically. Just because you gave it access to Slack, it's looking for feedback cuz yeah, that is much easier to do. Like if it has the tools and you don't have to like as a human yourself figure out, okay, let me assemble a list of all the

28:43feedback I've ever given. Um, but if you do this like three-step process of step one, assemble all the feedback you've ever given. Step two, tell a claw to like condense that down into a list of rules or principles that you care about. Sometimes it's useful to like then again do a third round of like okay what are the principles that maybe you might believe but you don't actually know you believe and then take that list as like an input for your ongoing reviews that is that has been quite successful in our attempts to I want to say like specialize the feedback that individual sessions give um that has really worked I think for

29:18for work that is a little different than raw coding or not very textbased I think the problem is still bigger Um but I think similar things can be done right about like give me feedback on a customer call. How did that go? If you read the transcript or like maybe you're I'm doing a lot of interviews for anthropic which is like a typical knowledge work problem but also like quite interesting but then it's teaching me I wanted to teach it. So and I think a lot of people who started using I experienced this you want to do more. You see how productive you are but you I don't there is no agent that I

29:50trust 100%. Every output goes through my team and then eventually through me. So I become the biggest bottleneck in the company. Honestly, I feel like our products would do that for you pretty well today. I'm actually quite confident like if Claude has the ability to watch you give this feedback for a while. Like the first thing you would probably do is you would sort of run it side by side. This is an experiment I recommend to almost everyone. It's like really fun to do. If you currently have a trust problem, you will probably not be super excited to

30:15immediately give over the reins, right? Like from one day to the next. But sort of run it side by side. like you read something, you write down your feedback and um you do that for a few rounds and just to check what kind of feedback claw would have given and like where do they differ where are they the same that also gives you something where you can actually sharpen the workflow a little bit because I think models will still be imperfect right there's still going to be things that where where the feedback is like a little different and like even for most humans if we take the same

30:42human and we ask them to review the same document like on a Monday versus a Friday the feedback will be different but this like sideby-side process is rather powerful because it shows you so much. And how do I build this agent that talks to my team? Is it just something that like a website where they upload stuff

30:57and it gives feedback? I would recommend claw tag which is cla inside slack. Uhhuh. And you build you give it access to my taste and Yeah. It's like it's like currently our core multiplayer experience, right? Where multiple people can talk to Claude at the same time. Yeah. Yeah. Yeah. This is all getting very interesting

31:15when it comes to building things, right? because you can build so many things easy. What becomes valuable when you're building software and I'm talking about uh people who are beginner entrepreneurs. I'm thinking about myself to what is it 2015 we came here with a booking platform for study abroad experiences amazing like a modem was building this whole platform because we hired a lot of engineers was th hundreds of thousands of dollars we automated stuff now I can do this in two days by myself like what is my entrepreneurial mode when you think about building software is there any spicy opinion I think it actually hasn't changed that much I think the cost of

31:56investment like sort of the upfront capital cost was much higher. That is true. And like I think we're now entering this interesting area where like single person teams or like two person teams can build billion dollar companies. But I think the secret to building a good product is actually not changed. And I believe it's empathy. Empathy for the user, empathy for the use case, empathy for like trying to figure out is this actually valuable for the person who's ultimately going to pay for it. I think that has again and I think this is maybe a spicy opinion but I think this has always been the key

32:24ingredient in any entrepreneurial team right and I think especially in startups like before we started taping you and I talked a little bit about 500 startups I think this is even like sort of common knowledge within VCs that whatever you show up with doesn't really matter it's probably going to change anyway but you as the founder you're the key ingredient and we can we can go one level deeper what does that mean is it like your talent your skills I think empathy and like an ability to really build something of value is probably the core thing that matters in a founder and I think that has not changed. It has just

32:53gotten much easier to tap into that empathy and like to to execute on it. I totally agree with that. But also sometimes I'm thinking like I have this idea but then I see something on X where people say oh this new model just killed this and we see people vip coding their smaller tools etc. Do you think it's a trend that we're going to v code a lot of tools versus paying for them? like what would us make us pay for a certain tool if we can spend an hour just

33:17building something similar. I think software that is hopefully going to go away is bad software. Like the software that doesn't really serve your use case and you're only using it because you don't have anything better. I think all of us have some tools that come to mind where you're like begrudgingly okay I guess I'm going to use this because I have to and I don't have the time to like look for something else or use something

33:35else. I think that will come to an end. Um however at the same time we talked about how hard it is to to build good products. I think there's still going to be tremendous value in like figuring out how to best deliver value and like to have empathy in a way where most of the products I use that I'm like a big fan

33:51of I wouldn't have come up with myself. Mhm. Right. Like I'm on record as like having written blog posts about how the iPhone is stupid because obviously you need a keyboard. Right. Oh, really? Yeah. Of course. I think for most of the products that I really enjoy, someone else came up with an idea, executed on it, and then gave it to me. And I think the main reason I use those products is rarely because it was very expensive to build or because it took them a long

34:11time or because they needed a big team. The reason I use the product is because it solves a need I have really really well probably in a way that was better than I thought of. The gap that I think we're going to completely fill is this garbage software and I only have to use it because I have no other choice. I think that will hopefully go away and

34:26I'm very excited about that future. Before we keep going, I'd love to get to know you a little better. I share a lot about my work here and I'm curious about yours. What do you do and what you'd like more help with? We've put together a little survey. Your answers will help me decide what to cover in future conversations and what resources to put together for you. I'll leave the link in the description. Please, please, please take a couple minutes and tell me about you, about your business, about your ambition so I can make better content for you. I really, really appreciate

34:56you. Thank you so much for doing this. You mentioned something about Windows computers that quietly run work and that they are a big business opportunity. Can you talk to me about that? I build AI products. I work on cloud. When I go out into the world and I look at what are the computers and the machines that are really running the economy. I think it's really easy, especially for like us in Silicon Valley to like sort of consider your typical startup with like bunch of the latest generation MacBooks and that's all that exists and then you go to an airport and you sort of like peek around the corner

35:26and you see that they have not actually yet moved to graphical operating systems. It's still like the black and white like oh yeah the transportation authorities the Yeah. And I'm doing this for fun now sometimes that I just go to all of these places that I that that run so much of the economy. Like the last place I did this was a hotel. Like I love doing this at a check-in where I'm like, "Hi, I work in software. I build tools. Can I just like can you just show me real quick like what you do on your little

35:49machine right there to check me in?" Harrowing. Like all of the all of these devices have software that is so clunky, so bad. Truly awful. And I think it is probably an untapped opportunity for many businesses to go in there and like try to clean that up. Well, before AI became a thing in 2010, I I joke that if I ever wanted to retire, I would build really

36:13good software for dentist offices. Now you can build it in a day. I guess it's it's the relationship part. Yeah, exactly. That's becoming Well, this is the hardest, right? Not not just the software. Any other businesses you wish existed? I still think we're barely getting started on software that feels a little bit more personal and has a little bit of a touch of AI that is not just here's a here's a sidebar with a chatbot in it. I think we've we've sort of permeated the entire industry with low intelligence cheap models that operate sort of at the level of I've read the FAQ page and if you give me a

36:47question I'm sort of like spit out the answer. We haven't really like figured out what can we do with topline intelligence across many of our applications. Businesses that I specifically wish existed. I'm just going to double down on this empathy thing. The industry moves slowly, right? Like even VCs move kind of slowly and it's like really hard to adjust to a new world. And I don't think we've properly adjusted to execution is cheap. We've always said ideas are cheap. That was true. And we said execution kind of expensive, but it's gotten cheaper over time. And now execution has become extremely cheap. I think we need to like

37:18really double down on empathy, connections, who's capable of like going into a business and really understanding what the needs are and how to can how can you actually optimize things here. So this might actually be a nice opportunity like the bigger revenge are the MBAs, right? Like I think MBAs can now make a bunch of software startups

37:33that might be quite successful. I was just talking to a Nobel Prize laurate in economics and he was telling me that we have such powerful models but we really lack the application layer. Do you agree with him that there is this huge gap between what models are capable of and what average user is using them

37:49for? Oh yeah, for sure. Yeah, we definitely have like a capability overhang, right? Like the idea of you have an amazing model in front of you and if you're essentially using it to to ask, hey, what's the capital of France? Again, you're probably not really tapping into what is possible there. Our founder and CEO, Dario, talks about this concept of like a data center of geniuses. My job really is to figure out okay like let's

38:11assume we have a data set of geniuses. How do I empower you the user to actually make use of that? And it requires creativity. It requires like a rethinking of like your needs and like what you think is possible. Um it's a it's a really hard problem that I don't actually think is new, right? It's this notion of people have always been held back what what by what they thought is possible rather than what is actually possible. Yeah, I do agree that we have like we've just scratched the surface of what's possible. How can one close that

38:39on an individual level? On an individual level, I thought about this like all weekend. I think on an individual level, the thing that you see on X a lot is this this rallying cry of be more ambitious. We ought to be more ambitious. Like what being more ambitious really means is I think so context dependent. But I would like people to shed all the shackles of like what they believe is possible and rethink why are they doing things, right? like what like I'm doing a certain task in my business. What I what

39:08am I hoping for is going to happen next. And you just keep giving the model that task. Like whatever outcome you're hoping for, give the model that task. You can't play this forever. You will eventually find the limits of the current top intelligence. Like a good counterargument here is okay like the purpose for most businesses is to make money. And if I just tell clude go make money, it's probably going to like not make it very far. So we we haven't reached like the top level of altitude yet. is somewhere in the middle. But once you bump against that ceiling, you

39:38do know that you're at the forefront. You're at the frontier and you're going to really benefit. You're going to benefit from the next model generations. Because if if all we're doing is like summarize this email for me or like make this email sound as angry, you're not really like benefiting from the new intelligent. What would be your advice to someone who's just starting out now as a software engineer in this ever evolving in this ever evolving like landscape of

39:58like how to do software engineering? I think it's I think it really depends on what you want as a software engineer, right? like what does it what does it mean to be a software engineer? I think we're like as engineers are we're currently in this like moment of crisis where we're like what does it mean to be a software engineer? Um does it mean that I'm I've completely memorized the entire API service in React? Is that what I'm valuable for? Is that what I do? Um and I think that's probably less the case now. To me, this is a pretty welcome change. I've always been of the

40:24opinion that a good engineer versus a bad engineer. I measure I've always measured my engineers on their impact and the impact is like does the user like this enough to buy the software. This there's always for me been relatively little value in like really elegant beautiful mastery solutions where we can be really proud of the engineering craft here but the user

40:44doesn't feel any benefit. How do you develop that? What do you double down on now in order to be relevant in three years if like every position is suddenly a senior position? What would be your advice there? The way I got my start in this career is by like building a strong portfolio and I think that would be the same advice today. I think building a portfolio that is not just like a mock is much easier. So my recommendation for anyone young would be go and build some real things. Prove that you can do it. Prove that you have the empathy. Prove that you have the

41:14chops, the ability to talk to customers. Like figure out what is necessary versus not necessary. Um I do think there's like a slightly new definition of what software engineering is. I personally think that's good. I do see the young people adapt to this new style of thinking much quicker than some of the older engineers who maybe enjoyed the luxury of primarily working on on tickets without having to ask why too

41:35many times. I've seen so many people be quite successful in that. Um the things that I would maybe to contrast not recommend anymore. It is true that I think like completely memorizing individual API services I think has become less valuable. What is still valuable is being able to combine different systems in interesting ways. I think the value of that is amplified now because you can so quickly combine things, right? You can just like point AI like three different systems and be like I want these three like I want these five ingredients mashed together. Go and do

42:05it. Is that something that you look for when you hire people? Yeah. This ability to Is there something else that you're particularly looking for? I mean, I don't know how you hire. Do you still do you do you care a lot about the resume? I do. When it goes because when you mentioned relationships, they're very important for my industry. I am looking for someone who knows people. I am looking for someone who knows how other businesses operate because this is not public information like how do I learn how to what what do I charge as a podcaster right what do I charge for I want someone who knows exactly what's

42:33going on in the market so we do care about relationships but when it comes to the creative part actually with creative part we still look at how many views their videos have produced because yeah they know the art I don't look at their degree of course yeah yeah exactly exactly and I think this is I don't know if this is still controversial I guess we both work like in a you know like tech adjacent, but I've never cared too much about which degree do you have or which school you went to. It was always much easier for me to look at what are the things you

42:59made. Are those things I can touch, feel, try and I can feel like, oh, you either were the winning player on a team or you at least worked on a winning team. That is sometimes useful for me to know. I think that's really valuable information. And the thing is, if I didn't have that, I might go take the

43:15shortcut and look for schools, degrees. I don't really have to. I will always pick someone who's proven themselves to build something amazing over someone who hasn't built something amazing but has an amazing degree. What would you tell your kid when she grows up and says like, "Dad, I wanted to coding." What would be your advice? I mean, she's 9 months old right now, so

43:32I think things will change a little bit. I think you have to be in line for your daycare already. [laughter] Just saying. And it all happens quicker than you think. What would I tell my daughter? Um, I think I would tell my daughter to very early on try to understand how businesses operate, how people collaborate, um, how to really find value. I've not gone to the University of Wateroo, but I'm such a fan of the

43:51University of Wateroo's process. All the amazing minds, all the amazing minds, but I think the University of Wateroo has always had this approach which was quite unique within within computer science engineering, which was this is an outside perspective. They might disagree with me on this, but my outside perspective was always coming from Germany where computer science is very

44:09much computer science. It's informatics. It's a lot of math and very little like only a sliver of how do we actually build a good product. Um, let alone computer engineering. JavaScript was in Germany for a long time universally considered to be like not what real computer scientists use. And the University of Wateroo's approach has always fascinated me so much because I would get these interns or junior engineers who've worked at five different companies before in their internships. It was pretty common especially in San Francisco that you get an intern and it's their like fourth internship and they previously interned at Tesla, Google and Apple. And they

44:43have so many interesting things they learned about how companies operate, what are the upsides and downsides, how do you actually advocate for an idea, how do you get resources, how do you convince legal to get something done. I think those are the skills that really like for me made University of Wateroo's engineers almost immediately effective. It's their first job maybe with me, but they're in a completely different category than someone who has maybe even gotten a PhD in computer science, but has never worked in a company. They might be very smart what they do. But as we all know, like building a good product and getting it out there is

45:13about much more than just knowing what code to write. You need to be able to talk to people. You need to be able to like work together with other functions. You need to have like a little bit of a of a sense for how does this company run. So I would tell my daughter to

45:24probably like really invest in that. Love it. What do you tell your kids? What do you think your kids should do in the ever evolving economy? I tell them to try all the things. And I tell them you can be anything cuz sometimes they're like oh robots. You can be the queen of like if I tell her like you can build a robot, she's like not very excited. I tell her you can be

45:44the queen of robots. Yeah. And she's like oh I like queen of robots. And then she says and we we bought this self-driving car that you can program yourself. Oh, that is fun. She was building it with her dad and like just telling them they can be anything and there is no thing that they cannot touch because it's I don't know out of this or whatever cuz I came here with a notion that everyone here is from another planet. Yeah, if you went to Stanford that was like something that I couldn't even fathom but seeing that these are all normal people and like giving this idea to my

46:18daughters that anything is possible is just hard work and passion. Oh man, I love that. I think the reason I love that so much is because in AI circles we constantly talk about this idea of ambition and the reason people are not ambitious enough is because we're sort of programmed as humans to like only consider certain things possible. And if we don't consider them possible, we don't even try. So it's like real like what we believe is possible is the real limiter. Like a much bigger limiter than what the models can do or what we can do. Maybe that's another reason like really believe in the new generation is because they will

46:47never have experienced all these limitations that we have. Okay. I think we covered so much. I just wanted like your final thought on our future. How do you think our life is going to be different in 5 years? That is such a big question. And I'm not sure you have the right person for like I I know that like going on any podcast is like there's a there's an occupational hazard of having to make predictions. Biggest thing that I've noticed so far is that we're all so wrong about the intelligence exponential coming out of AI. It's really hard to like wrap your finger around what does

47:16it really mean for for the exponential to exist. And I keep going back to math, right? Like when when we won the silver medal at the m period when I say we I mean Google um as an industry I was well aware of scaling laws. I was well aware of like how much intelligence progressing and I think if you had asked me at what point will AI solve what we would otherwise consider unsolvable math problems I don't think I would have given you like the middle of 2026 I would have been like that's that's in a year and a half. What are you talking about? So I think instinctively having a

47:52good feeling for how will what will the future look like is really really tricky. That's the first part. And I think the second part that is rather tricky for me is the one thing I can tell you with a decent amount of confidence is that I think the products that we use today will all look completely different. I think I think we're currently we've hit this point where I becoming so mainstream and it's becoming so popular that it's like easy to imagine or we're maybe like this this is sort of the end state. Um, but I'm quite convinced that all the AI products we use today are roughly in the same

48:20phase as all the funny phones that we had right before the smartphone came along. So I think we're still in the pre iPhone era, which makes it very exciting for me to work in. Wow, this is really fascinating already like from in our preI era. Uh there are so many things that are changing and you made me rethink my whole process. I won't even show you what what it looks like now cuz I think it's fine if it works. very outdated [laughter] cuz cuz we give it all the data and tell it exactly what to look at. But I totally understand how giving it access to

48:48everything is just so much more powerful. It's such [laughter] a tricky thing, right? Like all the advice I've given you today in about 10 weeks, we're probably going to have a new revision on all of it. That is also true. So being an always on agent when it comes to new models, [laughter] I think that's the that's the thing. And trying new things. Thank you so much, Felix. If this conversation with Felix made you want to try Claude in your work, watch my interview with Alli Miller next. Ally works with teams on using AI and in our conversation, she showed me her real workflows and built a

49:17Claude morning brief life. It's a practical place to start if you're wondering what to do