Stop Shipping AI Nobody Can Verify with Hamel Husain

0:02Hey there, Hamill, and welcome back to the show. Yeah, nice to see you. It's always interesting. I like I can look between your headphones and your head and like guess what your real background is. Exact. I mean, it's like it's like fall fall leaves or something. That's It's a It's an indigenous Australian painting that is reminiscent of all types of uh flora, including fall autumn

0:24autumn leaves. So, you nailed it, man. Um, what's up, Haml? Really nice to see you. Yeah, really, really nice to be on pod. I What uh part is this? Or what? How many times have you been on the pod? Do you know? Do we know? Do you have a count? I think nine. Eight or nine. You're you're the most recurring guest. And I mean, in the early like a couple of years ago, you were coming on once every couple of months. And it was so it was so much so that you were like, "Hey, if I'm going to come on this much, I should just be co-host." And then we started

0:55doing slightly slightly different things. But man, it's been a wild a wild journey, right? And I do think and hopefully we get into some of this, but I was reflecting we met over a decade ago in Boston uh when you're at data robot. I was at at data camp in what were like early to middle data science days and it

1:13feels like it was a simpler time. Yeah, it was a it was a fun time back then like data science was just starting to take off. There was a lot of excitement around it. It was that was around the time where you know there was a lot of buzz around data science like it was the quote sexiest perfection or something like that. Harvard Business Review had an article everyone was trying to be a data scientist. They're getting paid through the nose. Uh it was

1:41really crazy. Exactly. And I do think it would be interesting at some point to for us to talk through what's happened to data science and how important data science as a throughine in our conversations is in AI engineering as as well. But before jumping in, I just want to welcome everyone to watching the live stream, everyone watching afterwards. This is Vanishing Gradients. I'm Hugo Ban Anderson. Um, and put all your questions in Discord. I've put a link in the description and in the chat. Uh, that's just so we can continue the conversation afterwards. If you do uh like this type of stuff, please give it a like and

2:12subscribe and and share with a friend as well. Um, I'm also really excited. I hope to get into how your strayer and your course has changed uh Hamill. um particularly with what everything that's happening in a with agents AI agents now but um Hamilton and Shrea's next AI evals course for engineers and PM starts in a couple of days and they're very kindly giving all viewers 50 uh 25% off so I put the link to [laughter] that in there 110% off um but they're very generously doing uh uh doing that um I am maybe we could just jump into I am interested Hamill in I've taken your

2:49course several times now very excited to participate in the upcoming uh cohort and I am wondering how how you've had to change it and wanted to change it with everything that's been happening with agents now. Yeah. Okay. So, one kind of key sort of change or one of the most important changes how do you do eval bottlenecks in doing eval is looking at data. been saying looking at data for a long time on this podcast and people don't want to do it. It takes a lot of time, takes a lot of effort, sweat, tears, blood, whatever you want to call it. You know, you have to sit there look

3:31at data. You know, as a data scientist, you go, you probably don't have any issue with looking at data. You know, instinctively that it's a high value activity. I'm also addicted to it and in some ways like in in in good ways but also in ways that like I kind of can like hyperfocus and forget to eat and stuff like that right so there is a certain trait of people who've been working in data for some time I think that maybe that's tough to learn yes and so we know that that is a bottleneck so for example so what we've tried to do is think really deeply about

4:06eval to help you. And at first we we resisted it because we were like, well, everyone's going to just turn off their brains. They're not going to look at any data at all that is going to tell Claude just to like do everything and it's going to go off the rails. But then we kind of said, okay, you know, we can't really do that. We can't just

4:28tell everybody to eat their vegetables. And so, you know, in reality, you're going to be using a hybrid approach where AI helps you just like you do with anything else in life. And so for example when it comes to looking at data so in machine learning we have this idea of active learning. It's like if you're going to label examples can we surface high value examples to the annotator to label. For example, uh examples that the model has low confidence in are other signals about these are useful things to label in addition to some random sampling, some trade-off between, you know, exploration, exploitation, so on

5:13and so forth. Active learning is a really deep subject. But can we do something like that for when we need to look at the data? Can we, you know, do we just have to just do straight random sampling? Do we have to sort through a lot of data? How can we give people a little bit more power tools without getting them locked into a vendor? Because your agents, your coding agents are super powerful, right? And so what we can do is say, okay, like when it comes to looking at data, how do you do effective sampling? How do you look at the data and then get an agent to

5:51sort of with you after like learn from what you're doing and what you're labeling and surface other really high value examples to you so that you can label faster so you can annotate faster and the chances you're like your your learning rate as a human I'm not talking about machine learning your learning rate as a human is is much faster like you're learning more because the examples you're seeing are teaching you something new about your problem. And so there's ways to go about it. I mean, so there's not there's no like rocket science per se behind it. I mean, we can do things like, you know, clustering the

6:30data in various ways. Um, various types of sampling. Um, what we show for example is a workflow where you have the agent monitoring your annotations in real time and learning from what you're doing and then helping surface more examples of any failures that you that you notice, but also doing a little bit more of a meta analysis and learning from what you annotate and saying, "Okay, like perhaps these other types of things might be interesting." in addition to exploration. And it's tricky. There's no I mean it's not bulletproof. It can go wrong just like any just like a normal sampling can go wrong. But it is

7:15something that we're teaching. We're teaching you how to navigate it. How do you do it? How to think about it. Uh and then this kind of shows up in every part of the workflow of okay um when it comes to making LM as a judge, how do you how do you hill climb your LM as a judge? How do you make it to where um yeah, how do you get an agent's help to help you climb the element as a judge, but also make to validate that judge to make sure it's not overfitting or uh is actually agrees with human labels, etc., etc. How do you acquire human labels? How do you

7:51do that like more effectively? uh basically like how do you create tools that will allow you to source those labels um or borrow them from other parts of the process. We also teach you also even before it comes to like error discovery which is the process I described right now is like how do you design your products to begin with with evals in mind which is what we're going to a [clears throat] large part of what we're going to talk about today because like even so um the reason this comes up is like how do you how do you uh design your product with evals in mind because

8:29a lot of students come to us say hey you know like I'm having trouble evaling this thing. A lot of times we have to zoom all the way out and and realize that hey like this product isn't designed well for to be evaluated. Meaning your users won't be able to understand if the answer is right either. And we can show some examples. We can do some screen

8:58sharing and show you what that means. And so basically we just redid the entire course from scratch to make it to modernize it essentially to make sure that we are incorporating like all the different ways that you can go about evals and use AI effectively in the workflow and it does draw on a lot of

9:21data science. um you know but it's like okay the data science is yeah you have to like put the data science on guard rails in a sense and we try to teach as much of the data science as possible but um but yeah it's this is a way more agentic in nature and they also we expanded the scope a little bit like we also go into red teaming now and other things so uh if you want to go to the course page you can go to if yeah if Google Haml Eval's course or something

9:55like that, you can see the course page. Um there's a link to the syllabus on there and you can see the full syllabus. Awesome. And I'm going to put the link in Discord. It is in the um YouTube description as well. I am really interested. Um so we are talking about data analysis, data science plus for the purposes of AI engineering uh for lack of a better term. Um, and it is kind of bordering on agentic engineering as well. It's using agents to explore things, but to help verify along these lines also. And as we know, claude code, codecs, these types of things aren't necessarily great out of

10:38the out of the box at doing these types of things. Um, so how do you think about constraining or steering uh the agent to be able to do the do these things well as a human partner? Do you think about equipping them with skills or tools in MCP servers that that you've built and maintained or um agents markdowns or Yeah. How do you how do you coax agents

11:01to perform well at these types of tasks? Yeah. So it's really So we have skills and those skills are open by the way if you we redid all the skills um just recently and I've been making updates even as recently as yesterday of like okay eval skills but it's just like anything else it's just like coding or anything else you need to educate the human and you need to give them some skills and the the human needs to understand the process of like what good looks like what are we trying to do? The

11:35skills can only help you so much. There's a there's an upper limit to how how skills can help you because I mean at the end of the day it's a bit generic because it's meant to apply to a lot of different use cases, but you have to put a little bit more thinking into it when

11:51it comes to customizing it for yourself. And so it's it really comes down to like learning how what good evals look like plus using the skills um and then customizing the skills for yourself. Makes a lot of sense. Um I also I there are some skills that I think it's very much worth sharing. I think Marimo pair uh grill me there are a handful of skills. um some of the impeccable skills for making stuff look really really nice design-wise. But I have I actually some of our conversations among other things led me to to this. But I think um we should be sharing the shape of skills to

12:33use an LLMism rather than skills themselves. And I'll give one example. When you came on our show us your agent skills show and said your skills and mine too. Um for reasons you've just alluded to. Uh you also showed us a skill several skills that you find useful and one was um being able to populate stuff in the in your Maven course landing pages using their content management system which can be a bit gnarly. Um and so after that I was like oh yeah that's friction that has some friction for me so I want to do the same. And I nearly messaged you to say

13:05hey um can you share your skill with me? Then I was like, actually, I've got I've got the shape of Haml's skill literally and I actually have a skill which goes through the transcripts from all those videos and then pulls down the shapes of skills and whatever it may be. So, I had the shape of your skill from how you described it. And then I literally just built it out for myself with my agent pretty quickly. And it was actually I realized it was better than it would have been had I asked you directly for your skill because your skill encodes all types of personal things to you and

13:36your operating system uh that are totally irrelevant for me. So that's why I think for the I know there are as I said there are some like MIMO pair that are very much worth sharing a scale. Um but most skills I think we should be sharing shapes rather than skills themselves. Yeah. Um I am a fan of meta skills. So,

13:54let me let me uh let me share my screen. I can show you one. Yeah, great. And let's just let's make sure to narrate as much as possible for those listening and not watching. Yeah. Okay. Good reminder. So, what I'm showing on my screen here is a GitHub repo. It's Haml SMUR reverse-engineer site skill. Whatever. I'm getting carried carried carried away with the narration, but like okay, the skill I'm showing a skill on the screen. And basically what it is, it's a reverse engineering uh site skill. And it's basically for it's very simple. It's like any website you or any SAS tool you use uh that doesn't have an MCP like

14:35Maven. Um what it does is it helps you reverse engineer it while you're using while you're trying to do something like you know because you don't want to be clicking you don't want to have the AI be clicking around using computer use necessarily. It's very inefficient. Um, and while while your AI is clicking around, it can record the network requests because like you know there's like a post request or a get request or whatever and then your AI can document the the internal API of that site and basically create a create its own API like it can just reverse engineer it very you know simply. So this just like

15:15helps you with that. So this is like what I I use to like do that Maven skill as I just have make sure I I tell the AI what I'm trying to do is like hey I'm trying to like add a student to this class and do some other tricky thing u and then you know it clicks around it records the network requests it makes along the way and it's like oh okay next time I do this is it going to it just do it programmatically like very efficiently. Um, and so that's kind of, you know, I'm a big fan of these like meta skills that either help you create

15:46skills. Um, or do do things like across a wide range of of things. And I think like yeah, impeccable is is a good example of that cuz it like applies to a lot of design and like I love the grill me skill as well. Uh, another one that our mutual friend and your former colleague Greg Cesarelli has built at Specs Story Law, L O R E, which I don't know if you played around with that one, but it mines um all of your threads and conversations for things that you may want to turn into skills. So, it looks for repeatable patterns that you spend too much time asking your agent to do

16:24essentially. Um, turn them into skills. Uh, need some some SEO. Yeah, need some SEO. Um, I probably I probably misspelled his name in a weird way as well. Um, yeah, but yeah, I can I can link to that in the show notes as well. I um the other thing Hamill that we're talking around you you have we've been talking about using agents to help you with evaluation um but we haven't uh been talking about evaluation for agents and as we know a lot more people are building aic systems now so agents can help with evaluation but when we're building agents uh just the act of building an agent can

17:20add a lot more complexity over you know a basic series of deterministic LM calls workflows when we have a lot of reasoning loops the different types of steps they they can take so I'm wondering how you're thinking about this now with the advent of so many agents in production yeah I mean the process for evaluating agents like I guess like agents have existed for so long at this point that I don't even know what evaluating without agents necessarily looks like um but we we recommend the same process that we teach

17:50is like start with error analysis. We call it error discovery though so we change the name um so start with error discovery and you there's some analytical tools that you can avail yourself of. There's something called like a transition failure matrix that allows you to see like different steps in your agentic workflows and identify hotspots. But you don't have to go there right away. It's still really good to start with like essentially data analysis of your traces to find

18:21patterns. Absolutely. And I do think the transition matrices that um you and Sha and Brian really alluded me to it I found super useful. Um more useful I think when I have a sense of the different steps the agent can can take. Um, but if it has too many tools and too many different paths, it it can do that can actually really mess up um the way I evaluate things and my ability to evaluate. But that's that's one of the reasons I invited you on again this time. Not of course I I love chatting with you always, but your blog post like when I build these systems or consult or

18:59teach people building these types of systems, we often come and say, "Look, this is actually hard to evaluate now uh because of all the different choices the agent can can make." Um, and you wrote a wonderful blog post that I'll link to called called It's Hard to Eval is actually a product smell. So, I'm wondering if you can tell us a bit about why when you hear someone say it's hard to eval, your Spidey sense tingles with respect to wait, what are we building

19:24and how? Yeah. Um, I think it might be helpful to start with an example uh just to make it really clear. Uh, should I share my screen and kind of do something? Okay, please. So let me do that one second. So this one will be fun since you're a data scientist and you will immediately get into it. Okay. So one of the most like this is an example. I'm going to zoom in on this blog post and this this example and I have seen this product being built at companies more than any other product like everyone is building this. is some kind of agent, internal agent that will

20:08answer business questions. It's like what everybody thinks about doing with an agent, right? And they have like some kind of chatbot they've been built internally that can go query your database and do and basically try to replace a data analyst to some degree. And the dream is like oh like you can ask a question like what was the net

20:30revenue of product day last quarter? Okay. And then you'll get you get a response like the net revenue is X. [clears throat] Now, a lot of people get really excited when they see this. They're like, "This is amazing." Of course, like it seems like an obvious idea. But this is actually really bad. Like, as pictured on the screen, this is one

20:52of the most This is a really bad design. Do not make something like this. Um, and you might wonder be wondering like why? And so what happens in reality when you have some kind of internal tool like this? What happens is you ask this question, you get this answer and I don't know like you don't really know if it's right like how do you know right and if you don't know and so what ends up happening is they like ping a data scientist so let's say there's a PM his name is Doug and Doug runs this thing in the new this new fangle tool this data

21:34agent and gives it to you Hugo and he's like, "Hey, like this data agent gave me this answer." What would what would you say? Like what would you what would be your reaction? Like how would you even what would you start to do to try to know if this is a good answer? Aside from being pissed off that you have these like new questions you have to think about. I' I'd say show show me the working and some of the code or some of the analysis or I'd want to I'd want to know what the path was to figuring this out or the SQL query like

22:06whatever. Show me something. And the guy like you know again is like he doesn't have any of that. He just shows you the screenshots like I don't know I don't even know what SQL is. I' I'd then waste my time reproducing it probably. Yeah. Yeah. you would be reproducing it and specifically like I mean what would you do is you'd be like well this question is actually super complicated like what the hell is net revenue what does that mean what table does that come from what filters need to be applied to get to net revenue is there a agreed upon definition of net revenue and like what things need to be

22:48joined to be able to come to that And what the hell is last quarter? Like what are the dates? What what kind of physical calendar are we on? And what what is it? And um you you should be like sweating bullets. Like if you have this number, it's like net revenue number. You're like you don't want to give somebody a wrong number for something like this. Like how do you know if it's right? Well, the one of the best things you can do is try to try to like see if it's consistent with

23:20other numbers. Like maybe there's a report someone has elsewhere. You can kind of see like whether this number makes any sense. Um how was this? And then like like you said, you want to look into how maybe it was calculated like what how did we get to this net revenue? You know, what was the gross revenue? uh and then kind of like walk it does and do all those numbers make sense like you would start going through this thought process right and it's actually it might take you all day to figure that out normally and so basically what you have to do is when you this is just quick I mean I don't need

24:03to tell you this but this is one of the reasons notebooks and literate programming has been so useful for decades right yes yeah I mean that's exactly I mean So that's what a data scientist would do is they would they would take notes. They would like kind of write down what is their thought process to come into this number because like if you're especially if you're not a data scientist and you're designing a product like this, you would take for granted like all the stuff I just said. You'd be like how

24:31hard can this be? Uh net revenue, how hard is that? I see it on a dashboard. Seems like I don't know, maybe it's fine. Um but you have to be like really careful, right? like you have to and so the the thing is like if you have even if you have the trace it might not be enough to like know if it's cor if it's correct like yeah you could know what the SQL query is um and maybe some tool calls but like was that enough like I don't know like maybe but probably not like I want to know like how I need to have some evidence that I

25:12can trust this number. I need to do I need to think like a data scientist. Basically, I need to think like the domain expert and I need to design the product so that the user or the expert can actually evaluate it. Like how would the how does the domain expert want to see this information? They can quickly sort of get confidence in it. And so if you were to redesign this, okay, to be evaluated, what might it look like? And so, um, it might look something like this. It's like, oh, uh, you have what was net revenue for product A. And then it would have some

25:54sort of kind of thing. It would say like and this is just a mockup but just to drive home the idea like you might have some something from your semantic layer that returns a definition. You might have some intermediate calculations might be reference a notebook you know so let me click this notebook. Oh, it says, okay, getting carried away with my own example, but like yeah, I'll link you to a notebook um where this kind of thing was spelled out and you can see like what is the AI trying to do. It might even adapt that notebook from another analysis someone else did. These are all design

26:32ideas that you might want to do. And the idea is like there's a really large distance between what I showed initially and this kind of design that is more like a notebook which shows like the thinking and the steps and the provenence of like where the data comes from, but also why, you know, give us as many reasons to trust the number as you can think of. Like why should we can we make it super easy to verify or can we can we make it a lot easier to verify? And this has a lot to do with eval because if you design your product to make it a lot easier to

27:09verify, you're going to have a way easier time doing eval because you will collect you'll be able to collect way better signals from your end users about like what went wrong and what's wrong. Not to mention, you will log the signals that matter because you're surfacing the same thing. And you'll make Yeah, you'll just make it easier to label things either implicitly or explicitly, but also you'll make it a lot easier to um to automatically like classify errors because like you you are now sort of having all the data and then you'll get like way better feedback this way. And so like this is not science fiction be

27:54like oh that's that's cute Haml you mocked up what this thing could look like better you know uh that's not realistic like how in the hell am I going to make application looks like this right like how how are you going to make an application that you maybe like looks something like this and then you know you can open this notebook goes here right like how like whatever but there are applications that kind of do this like this is this is a real data product that kind of does something similar it's called hex it's pretty cool. But it's kind of like this is the, you know, a

28:25way better shape for this problem. It's like it's a [clears throat] no it's a notebook. Like you ask a question, it gives you an answer, but it shows you evidence and it also has a notebook and it like has a SQL query in it and all these other stuff and you can go and iterate on it. And so this is not science fiction. is actually like good design and you can find you know a lot of times if you feel that a product is hard to eval or you feel like I don't even I don't know how to eval this or you have you know you are

28:57coming from the position of like yeah eval sound great but like you just can't do it it's a strong smell that your product isn't good um and so you should think about that and this is not the only example there's other examples Uh I started with this data one because we have an infinity torance data but uh you know and I can show you other ones if you want to get into that but I'll stop here for a moment in case you have

29:23questions. I love it. Yeah, I've got so many questions and I've linked to the blog post and we will in the show notes. Um the other examples are the uh physical education curriculum builder and the workers comp medical report that we may may get to. But I am I am interested something we're really talking talking about is how thinking through verification can uh help us design and build better AI products. And I'm wondering just what main questions should we be asking to guide AI product

29:56design for verification? Yeah, I mean the main thing to do is simulate in your mind what a domain expert would want and how they would what they would want to see to trust the output. Um, and I'll show you an example in a bit where the idea the product started with, you know, giving you a deliverable, but it but after thinking about it, we realize like the deliverable is not what's important at all. It's it's just the proof. It's just the proof of how you got the

30:36deliverable. That's the important part. We don't care about the deliverable. Um, and so and this it was a it's a research agent. And I'll show you what that looks like. Um, but it's really like simulating. It's like getting really in tune with your domain expert like the topic at hand of like understanding the user and what they need because the user is not going to tell you

30:59directly like what they're looking for. Um, you know, if you've read like the mom test or uh or if you've read about, you know, discovery, product discovery, you'll know that like people are not good at telling you what they want. You have to kind of observe their behaviors and you have to get really in tune with your your users and say like what are they doing with your AI product? like are they, you know, if are they getting an answer and then have to do a whole bunch of work to like verify it cuz they they don't even know to tell you about this, right? Like you have to somehow

31:39discover it. But it does come down to like you really need to understand your user. I do want to have a look at your the the research example you you just mentioned. I am I do want to go on a slight tangent though. Um when Vincent Varmadam, our our mutual friend and colleague who is a MIMO now, but just wonderful like generative creative thinker and builder in the space came on our show us your agent skills show. He um he he talks about using ao notebook. There are other ways to imagine this though um as having a canvas where you and your agent can collaborate. Um, and he's even built

32:22things where he's got sliders which communicate with the agent in the MIMO notebook. Um, among other things, when when our friend and colleague Isaac Flaff um, came on, he he showed us how he actually gets his agent to create a lot of editable HTML, right? Um, so that's another form of of a shared canvas and that's something we're kind of talking around here. So, I'm wondering just what you see is like currently happening in the space, but the future of where we'll be working with agents to to move as quickly and interestingly and verifiably as

32:57possible. Yeah. Um I I think having these canvases where you work with your agent is really powerful. There is a blog post from my friend who works at OpenAI. Um it's a it's talking about a new kind of notebook that is used for DevOps and it uses web something called web MCP which is very newish um that allows the agent allows you to expose something like an MCP but through your browser directly. um that allows the agent to talk directly to the application in the browser like allows it exposes tools so you don't have to go

33:43through a server back end. Um and basically it's a notebook that allows you to I can share I can find it if you want want to see it real quick. Let me let me pull it up. Let me pull it up.

34:01Oh, here we go. Found it. Uh let me share my screen. Um, so this is this is written by my friend Jeremy Louie. You should maybe have him on the podcast. Really cool guy. And I've uh he's done a lot of machine learning work in the past in ML infrastructure, but he has this sort of like thing that he created. It's basically a notebook as

34:24you can see. I know it's kind of small. Um, basically it's a static site that has a backend. It has a well I wouldn't call it static site. It's a It generates an HTML file from you have to read it. I'm not I don't want to butcher it, but basically it exposes web MCP. Um, and you can edit cells and run cells. Um, and you can embed this stuff inside things like codecs also. So you could just, you know, just work with it right there. Um but you know DevOps is has a lot of has a lot of commonalities in with data science workflows in the sense

35:05that there's state that you want to keep track of and a lot of tasks take a really long time and you want like to know what the intermediate state of is of any of like that task and you want to be able to play around because you're like debugging you're like okay I got here after like a few hours but then like what is happening here and you want to experiment a little bit and you going to write notes. And so it's kind of like this notebook. It's a

35:30little bit different kind of notebook. It's on RunMe. But basically, this is what he's using at inside OpenAI to do like a lot of infrastructure work and particularly uh this person is doing a lot of works with work with like evals and like running lots of uh like models through like various evals which is a non-trivial task at the OpenAI scale, right? And so,

35:55um, yeah, he uses something like this. And you can see, you can read more about it here. You can see like here's an example where he has some notes. He has like, you know, you can see like run a terminal bench against 5.6 soul. Um, and then he talks about like the notebooks and the web MCP and how how it sort of what works. So, like I would pay attention to this. I would pay attention to WebMP. Check out this blog post. Um, and yeah, I think there's something here. I think this is like a really fun way to like build things uh that you can expose to other people

36:33as well. Super cool. Well, I've linked to that in the Discord and we'll link to it in uh the show notes as well. Um, we have a a great question in Discord and just a reminder anyone watching if you have any questions for Hamill, I mean you get, you know, get answers straight from the Oracle here. Um we have a question from Lakvia which is uh software engineering is moving at breakneck speed. Agree. Um Lakia feels like uh he's never had the time to do proper evals like he wants to. Um and he's wondering generally how you see this like how much of not having time for this is organizational

37:09alignment towards having an eval policy first. You know, I wouldn't get too academic about it or even try to get um too dogmatic about it of like evals. You should do evals. If you're not doing evals, you are in the wrong. You should be guilty. No, like you should be doing evals. If you need evals, if you want to do evals and I would start with,

37:38okay, let's forget evals for a second. Just start with looking at some data. That's all. Just look at some data. even like 20 traces, even 10 traces, and you know, use the eval skills that we discussed earlier. It was up on the screen, but I know you put a link somewhere already. Um, you can get your coding agent to help you look at traces really fast. They can have it render like a nice interface for you to look at those traces, help you sample those traces, and just look at 10 traces. Just

38:08start there. Just look at your data. Just look at the data. If you look at the data, it will motivate what you need to do next. you'll automatically figure out like, oh, there's some really nasty errors in my application that I didn't know about. How am I going to make sure that doesn't happen again? Or how do I like even solve this problem? I have no effing idea how to solve this problem right off the bat. I have some maybe some theories and okay then maybe an eval would help you cuz you like work through that. But like I wanted to make sure that people don't have this idea like eval is a

38:52separate thing like this separate activity like go do evals. I'm going to go do eval. No, it's like part of your software development. It's part it's just part of what you're building. So like at the point where you want to start investigating is your thing good like if that's something that you care about like is it working then start just start with looking at data and then the

39:18rest is motivated from there. Totally agree and I do think even if you're sometimes people don't seem motivated to measure if it's good or not but there a forcing function is when a new model comes out right and that's a that's a question like what's happening in your business if you're building stuff that you're not motivated to figure out if it works or not but that aside I've seen people forced to do this when new models come out and they're like oh I got to switch to 5.6 six or whatever, whatever, right? And then they're like, "Oh, how do I tell if it regresses on some stuff or or or

39:53works better." So, even, you know, things like this can force people to do it as well, I found. Yeah. I mean, you don't necessarily need to go into like rigorous evals too early either. Like, if you're just building some tool for yourself and you are the only user, then you are the eval for all intents and purposes, like you don't need to do all this like stuff necessarily. I mean you can look at almost you look at every example. Um and so there's certain situations where you don't necessarily need to but okay if you're building a product for other people especially or is being used a lot or is being done in

40:32critical workflows or anything like that then that's when you're going to want some idea of if it's working well. Yeah. I um Pastor has a great question in Discord about how you've seen the space change the past couple of years in terms of what what people are building. So I mean for example, have you seen a movement to from people building basic retrieval systems to throwing lots of tools at agents to less tools now or what what have you seen happen um the

41:02past couple of years? I mean I think the main gradient of change is coding agents. So everybody is writing code and everyone's creating stuff and prototyping stuff and so um that's why we have to have these discussions like make sure you design your products with evals in mind or design your products with verification in mind. So like let's like we don't even have to get dogmatic about evals like you have to do just like think you got to think about the user and like how are they going to trust what you're

41:38giving them. M. So, were there any other examples you wanted to Yeah. Yeah. Let's Let's talk about some other ones. Yeah. Yeah. Great. Um, so I'm going to pull up the blog post again. Yeah. Um, and go here and let me let me go to the blog post here. Um, by the way, there is this blog post, the revenge of the data scientist, which is something we we touched on. We might want to touch on it later, but um let's go to uh it's hard to email as a product smell. Okay, so we talked about this data agent already. So let's let's look at a different example. Um let me

42:20come down here and say okay. So like there's this workers comp medical report thing and this is like a real example. It's anonymized. So like this data is shown on the screen is not real data but basically um there's an agent that ingests an entire patient chart and history along with a bunch of other information to make an assessment if that patient or that candidate is eligible for uh workers comp. And workers comp is sort of this thing in the US where if you get hurt at in your job then you know your employer or the insurance company will compensate you for your medical expenses and maybe

43:08some other things. And so it's a pretty arduous thing normally like you have to have a doctor review all the evidence and sort of come to conclusion within the guidelines. um and there's multiple parties involved so on and so forth and you have to then generate a PDF or sorry generate a report it can be a PDF or some other formats and this thing takes a long time is like very expensive to do and so this product that I was advising on they were doing that and um you know they ingested all of this stuff and then they produced a report and you see like like this they

43:52they produced this report now and then they said like well how how do we know if it's good? So it's great. It's like it's you know again and a lot of AI products are like this and hopefully you know as we go through more examples it'll click. It's like you can't just give the user an

44:14output and say voila here's your output. you just don't don't ever do that because what what you know the user what are they going to do with this and this particular person sort of sat with this form of the product for quite a while and they're like spinning their wheels like how did they how do you know if this is good like and you can feel stuck if you're working on something like this you're like well how do you what are you going to do keep updating the prompt like what are you going to do and so the first question in your mind when it comes when you start to eval

44:47something and you and you don't know if it's good or like you're like, "Well, how do I begin?" Well, think about how would an expert begin? And what is an exp what is the process an expert goes through to arrive at this output? So, if you had to do something like this, like what do you imagine someone would do? they would um they would do a factf finding you know so they would try to like go through sort of the all the documents and like find key facts but then they might also do things like find contradictions or open questions like oh it's not clear whether XYZ happened or this person was

45:35hurt at this time this document says uh the person was hurt on this say but then this document says like it happened in a completely different place and there's an inconsistency. These are all kinds of things like if you know when you dig in to the particular use case and understand like what is going on here you understand like okay like what is the process of coming to the output and so you can imagine like you want maybe something like this right it's like you have your sources but you have like some kind of things that you might want to review um oops sorry um this is just a mockup

46:15up. So I also click on it just out of curiosity sometimes. But like um you know so you might want things to review, you might want to add notes, you know, you see like open questions, key facts, things like that. And you know, if you think about it, this is actually really useful to any person in that field. They're like, "Oh, like if you go to demo this and imagine demoing this and you demo, oh, here's the

46:49output." Now, at first blush, you might think, "Oh, demoing that, giving the person the output, like we've done your work for you. That sounds compelling, right?" But actually, what's more exciting to the expert is like, "I'm going to help you do your job. I'm going to take the user along for their journey and I'm going to involve you in the research project pro process, but I'm going to try to help you. I'm going to augment you as a human and I'm going to find things that have a high likelihood of being interesting. Some of them might be wrong, but that's AI is exactly good at this is like sorting through lots of

47:27information, surfacing things that might be interesting. Um, basically like you're improving recall for the human, you know, like can you somehow like augment the human so they're not so they don't miss something or the likelihood that they miss something goes way down and that's you know so it requires like when you understand the process um and you can almost reframe the product and say like absolutely you know this is what the product is doing this is why it's valuable and now you're not stressed out. You're not as stressed out. And now you can actually evaluate these kind of things. You can say like, okay, is it actually showing

48:09me re like are these contradictions actually contradictions? Are these key facts actually key facts that I need to know? Are these open questions actually open questions? See, we can we can now like get really concrete. We can think about how to evaluate this. whereas before like I have no idea how would you going to evaluate I mean it would be very messy it would it would not be tractable but now we've broken down the problem um and it's it's similar to the data problem like we're showing the steps but you see it's a little bit different as well we are you know essentially what we're doing is

48:48like we are trying to emulate what the user is doing in their process and their thought process and we're bringing them along because at the end of the day it's Like the main thing that you want the user of this product to get is you want to inform them. They want to feel educated. They want to feel like you have increased their level of knowledge because ultimately the doctor has to

49:11sign off on the report. And so the doctor is never going to sign off on a report that just created for them they have no idea whether or not it's correct. They just won't do that. But here they can you know do their normal thing and then also use this tool or some hybrid approach of both and they can look into things and okay now you can sort of now they feel more informed faster and like this is not a perfect I'm not a designer I'm just have this mock here as a discussion point to start the thought process of like hey when you're designing products you don't

49:51you feel like you can't evaluate it. Again, it's a big smell and you should think about what are we trying to do? What are the goals? It requires a bit of product thinking. Um, but it's not that big of a leap. Like I think it's fairly straightforward to sort of sort of see, but it it just it just requires like empathy with the

50:10user. Absolutely. And also I think something I like so firstly I agree with with with all of this and [clears throat] your formulation and that the mockups you've done I think are wonderful ways for people to dive into the things that they need to in order to trust the system and use the system. It actually reminds me of when we like you know will vibe code agent trace views or whatever and look at the you know end to end whatever happened but then be able to click in and see different tool calls and whatever it whatever it may be. Um, so I suppose iterative ex exposure of or

50:44exposing of of details of interest. The other thing it it really reminds me of I've been doing a lot of work trying to help people understand when you know in agentic engineering when to have deterministic verification when to have agentic verification when to have human verification and people are loop engineering this graph engineering that. Of course, there was the joke on Twitter, next it's going to be unemployment engineering. But I do, one thing I think people don't appreciate there is that sometimes or a lot of the time part of your objective function of what you're optimizing for is human understanding. Like when you're doing exploratory data analysis, for

51:20example, you don't want to automate all of that because one thing you really want there is for the user to understand significantly more about their data, about their business, whatever it may be. [clears throat] And that's something I love in here. Like you described as to be able to sign off on something for a medical practitioner. Great example. But there are so many examples where human understanding is actually a a a first principle of what we need this system to

51:44help with also. Yeah. And okay, there's another example that has a little bit of a different goal. And you bring up a good point that maybe it's the goal, the human's goal that we need to really pay attention to. And so, okay, like let's talk about this one. So this one, this particular product I have on the screen is a lesson planner for physical education. So like physical education is like, you know, gym, class, whatever you want to call it. And teachers want to be able to generate lesson plans. This is also a

52:16real product um that that I advised on. And so what the initial design of the product that the person was stuck with is like they put in the criteria of like what the grade level is, which is basically like what age the students are, the class length, the location, the equipment, so on and so forth. And it would spit out a like a it would spit

52:39out a lesson plan for a teacher. And you know they based it on like sort of lesson plans like some repository lesson plans and kind of had some rules so on so forth. They're like how do you eval they were stuck like how do you evaluate this? And so one thing that so when we dug into it what are teachers like what do they want to know? Like if you're a teacher and you're doing like physical education, especially if you're you want to you're you're new or you want help, what's like a use what's like a really useful thing is you would love to know if there are

53:23teachers like you that already are doing this same physical education you are that you trust that it's going well for. That would be amazing. If you could quickly find other teachers like you um that you trust with like lesson plans, you would love to see those lesson plans. And if there there's something different about those teachers, like maybe their class is not as long or they have different equipment than you, maybe you could like tweak something, but you would still want it like you would feel a lot better to you number one to like understand what everyone else is doing and people you

53:59trust. Um, but also you want to learn as a teacher. You're trying to learn like what does a good lesson plan look like. And what's one way good of learning is what other teachers other expert teachers. It's like any learning anything else. And so if you were to put that frame on this product, it might look something like this. is like okay now instead of generating just a lesson plan you're like okay you can find like other lesson plans that you trust like okay this is like some other person's lesson plan use that these like you can and then you can think about like what are some trust

54:43signals okay like okay this particular wrestling plan has been forked 14 times and this particular thing is like okay maybe I'll click on that and then it'll show you like what the plan is and how that original plan was edited to suit you. So now you are like doing this, you're trying to like um have more empathy for the user and you're trying to really think hard like why should this user care about your AI's output? Like how do you number one trust it, but number two like what

55:16is the whole point of this AI anyway? Like what is it trying to do? And like by designing it this way now we can evaluate it. Now we have something more concrete to say like okay number one is this recommendation useful is for example is the model lesson pulled is that good? We can evaluate that. We can evaluate the retrieval. We can evaluate the matching between a teacher's query

55:46and the exemplar lesson plan. we can we can start to evaluate these dips and we can start to get we can have something concrete but also when you think about it this way it makes the product a lot better. And so to drive home like what we're talking about here is like we talked about three examples is like okay you know before you even think about evals make sure your product design is good because if you try to come at eval with a bad product design you're going to have a bad time and this this is happening at an increasing rate more and more as it

56:27gets it's like free to make anything you want like making something like this you can do it with cloud code fairly easily. So so you have to think a lot about this now. Um and so and so the whole motivation for this blog post is like to get people to stop and think very carefully especially because my blog is really focused on evals. is like, hey, before you get into evals, let's design

57:00with evals in mind. That's one thing I loved about it's it's a product blog post, right? Um yes, which I which I really appreciate and it actually we've got a great question in in Discord from Rahul. Um which I think speaks to the design aspect of things to a certain extent, but it is a technical question. So, Rahul asks, "How do you get to a transition failure matrix when you have an agent that's dynamically planning with a react loop or dynamically choosing sub aents, tools, that type of stuff? How do we do error discovery and iteratively improve such a product?" Um, so maybe if we are going to talk about

57:35transition, failure matrices, you can remind us what that is, Haml. But then this kind of goes back to a part of our conversation earlier. If you've got agents going yolo with lots of different tool calls and sub aents, how do you even do this? Yeah. Okay. So, let's pull up what this uh transition matrix looks like so people can have a visual. Uh so, there's this FAQ page. Um let me see if I can find it. Okay. So, this is like a transition failure matrix now be so if we were to the question that was asked is like I have all this crazy Hamill, you told me about a

58:14two-dimensional chart. Two dimensions, but I have 5,000 dimensions. How am I going to get my 5,000 dimensions into this two-dimensional flat piece of paper that I can look like look at? One principle from data science is you need to simplify your data. You need to find ways to reduce the dimensionality of the problem and figure out like what are the most important things so that you can visualize it so that you can sort of make sense of it and it requires a lot of massaging right you need to like so that's where the data analysis part comes in is like okay in the transition failure matrix

59:00you know these rows and columns can be whatever you want in this particular example that I'm showing on the screen. These are various tools like you know uh gen SQL XX exe execute SQL plan calculation execute calculation but that might not be the right shape for what your problem is like you need to figure out like what is the kind of shape of

59:25that makes sense for you to dig into. Um, so there's no like one-sizefits fit-all answer. Basically, it requires a lot of exploratory data analysis of your traces to figure out like what types of signals are most important. Is it like tool calls? Is it transitions between various states? Is it some other kind of latent signals like number of turns, latency? Is it something else? Is there other some kind of latent signal in your data that allows you to like start

1:00:00digging around? And um and so that is like very exploratory like there's no like button you can push to automatically make sense of everything. It requires like kind of fiddling around a bit. Now you can fiddle around with an agent like you don't have to build these visualizations yourself. You don't have to write all the code to like summarize the data in this way, but you have to like develop some hypothesis. You have to have some business understanding and some systems thinking about like how does your system work and then like how can you simplify it? Um and so the way you get to this intuition you're like okay great you

1:00:39told me like I need to do this and whatever is to do error analysis. So you need to start with error analysis of looking at your data and it's through looking at your data that you start to develop hypotheses about what is important. I love it. And in particular, I I do think of it as, you know, failure-driven

1:01:00development in in in some ways, right? That you you really look at all the failures and allow that to to guide what, as you said, what what states transitions you're talking about here and notice where the hot spots are to build build out something like this. Yes, exactly. Awesome. So, we are going to have to wrap up in a minute. I do we've been talking around what's what the state of data science agentic data science um how data science can help uh AI engineering but I I do want to say um Kareem Carr on the platform formerly known as Twitter recently wrote and I get why I get why um he'd write this he

1:01:44wrote data science feels so dead now that it's like it never existed at all and then he wrote AI seems to have killed it. Um, and I I understand why he would write that, but I think from this conversation kind of seems like the opposite in some ways. So, I'm wondering if you can tell us where you feel the state of data science is now in the age

1:02:03of agentic AI. Yeah. So, first of all, I think that people should never get caught up on a job title. We should always think beyond our own job titles and think beyond job titles in general. We should think about skills. what skills do we have and how valuable are those skills? So the kind of what skills does a data

1:02:31scientist have? A data scientist has the skills of figuring out how to make sense of very noisy signals. Okay, these signals where they're not always consistent, they're not deterministic things happening in the world, patterns that we don't even know the answer to. They're coming out of a black box of some kind. Um, and then like how do you know uh what the root cause of those signals are? How do you get insight into you know the impact of those? And uh how do you debug systems that emit

1:03:16signals that are very noisy? Data scientists have all the tools to be able to tackle that problem. Now all the things I just said, this is what AI output is. AI output produces nondeterministic output text. um how do you measure this text of that is it good? Uh the thing that is producing the text is a bit of a black box like we don't know it's coming from an API or even if it's coming from model weights like might as well be a black box is fairly complicated. How do we know like whether the LM is doing what we want? How do we debug that? How do we sift through all

1:03:59this data? We just got a question like I have a crazy aentic process is producing lots of tool calls, a lot of traces everywhere. How do I know what the hell is going on? Data analysis. Data analysis helps us find patterns. And so it's the statistical thinking when it comes to you know measuring and thinking about noise and what that means, but also um you know like quantifying uncertainty um knowing about all the different traps and biases. But all the other tools, data tools of like cleaning data, data pipelines, um, and you know, all the muscle around sort of data analysis, exploratory data analysis, like thinking about signals,

1:04:47um, all of that is super valuable in AI. I can't think of anything that's like more valuable than that. Um, I think that people for whatever reason have don't see the connection They're like, "Oh, the job title data scientist." We don't see the job title data scientist anymore advertised on LinkedIn or whatever. So that the data scientist is dead. But let me ask you the question like all these skills that are needed for AI, who is going to do that? Like are you just going to start a new and just like people are just going to forget about

1:05:27what they learn as data scientists? No. those data scientists need to be doing this. They're going to be doing this thing that we've been talking about all day long uh or in this podcast in in and there's a lot more that we haven't talked about is okay. So that's data science. So I think it's I think it's wrong to say that data science is dead. like AI has made data science way more valuable uh than ever before because now you had just have like way more data and way more noisy signals that you need to reason about and debug and think about

1:06:00if they're working. What do you Well, I I totally agree and I do think that's one one side of it. We kind of have a a double-sided a double-sided coin like all good coins here. Um where these skills, as you said, are so important now that we're building. um AI powered software. Um on the other side of things to what you spoke about earlier how you're thinking about using agents to help with data analysis and eval um the ability of agents to be thought

1:06:32partners and co-builders. um when doing [clears throat] data science I think is moving us to you know we've seen what is h where agentic engineering is is going now and I think we're starting to really see the starts of what's happening with what we're calling agentic data science and premise being that claude code not necessarily good um at data science nor should it be and how do you build your harness around it so it can you know you can run par like parallel analyses and then have adversarial um analytical path code review and and that type of stuff and what's the role of the human in specification and verification

1:07:08when doing data science workflows where agents are helping you with a bunch of the stuff you know the silly examples being like you know writing and executing pandas code or polers or whatever it is but the more meaningful ones are you know having them as partners in designing an analytical paths and experiments and analysis and then adversarial review so yeah I think it's a really exciting time both for building AI powered products and dealing with the massive unstructured data we have out there and on the other side having them as partners in helping us do

1:07:42all of that as well. Yeah, absolutely. And I don't think like just you know the fact that AI can help you design an experiment of course like write your mapplot lib code. I mean I haven't memorized 100% of mapplot liib there's probably like rare there's probably like five people in this world that can do it from scratch. um you know write pandas code like even like some of the obscure stuff like okay like that saves a lot of time but even if you wipe all that stuff out there's all you know there's all of this judgment and skill about like what tools to apply at what time even if you zoom

1:08:20into something very specific like rag so the r you know of rag retrieval so retrieval is a search problem and search this information retrieval is a very deep data science field of like how do you make search good? How do you measure search? How do you know if uh how do you like tune the search to sort of do better? And how do you measure that? There's there's so many search metrics that you know and the search metric you apply depends upon your product goals and so and it can be very tricky to

1:08:57apply the right search metric. Mhm. um you know depending on what your goals are. So like you have to kind of command the tools and until we have AGI the AI is not going to command itself like that well at a very high level. Um and so you'll still need a data scientist. And also this kind of comes full circle

1:09:22to something I mentioned at the start. you like you like looking at data and I was like I'm I can be addicted to it dude and I I think bringing back to human curiosity and what we want to be doing with our time as well when people say to me hey can I just get an agent to look at all my traces I don't always say this but I almost always think maybe you shouldn't be building this product if you don't want to look at what's up in your data right so okay like I want to talk about that like people might hear you say you're

1:09:53addicted to data and they think like Hugo is a maniac. And I would say no, Hugo is not a maniac. You feel that. You feel that way because you have been rewarded so many times by looking at the data. You think [clears throat] you're addicted, but you have been rewarded. Every time you've looked at the data, there's been like an extremely large payoff. And you also have learned that that is an activity that is like a lot of people not there's not that much competition in there like you will do it and no one else is going to do it. And so those things combine you're like this

1:10:28is just insane. Like I'm just going to do this. It's like so much fun. Not not necessarily in the process. I mean we were mixing it up like this. The payoff is so high that like it's the whole thing is fun. But I was just want to point that out for anyone listening like Yeah, that's true. So there's a reason

1:10:44you were addicted. That's true. And there are two other things as well. The first is it's I I genuinely find it interesting. There is a there is a huge payoff of of course, but I I genuinely find it interesting particularly now like you know we talk about non-determinism of stockasticity of LM outputs and stuff. Sure, you get all types of funky weird stuff. Looking at human inputs is fascinating and strange and seeing what people do with your product is one of the some of the weirdest weirdest highv value stuff you can ever do. It's it can be truly surreal. Um and that's why I really love

1:11:17what you and and Shrea and a lot of us do in the space is, you know, tell people just to look at 10 10 or 20 traces, right? Because we know that once they do that, more often than not, they'll be like, "Oh yeah, I want to look at look at more now because this is

1:11:29really helping me ship good shit." Absolutely. Yeah. It's a gateway drug to basically everything else. It's like look at look at 10 examples. Data science is back, baby. The sexiest job of 2026 is what I'm feeling feeling about it. Um actually to to and we'll wrap up in a second, but um you also mentioned search and as you know I'm teaching this course with Doug um our our friend and colleague at the moment on building enterprise agents and

1:12:01search agents and that type of stuff. But one of the reasons I wanted to do this is in job listings like search is becoming really more and more prevalent. And one of the reasons to your point search kind of used to be like like oh re R rexius and feeds and that type of stuff or like manually looking through data. Um and and now with with AI any data scientist can step outside that loop and start building the engine that does search with with retrieval now. So I do think search with a capital S, a big S, is going to be an increasing core competency for any builder in the space

1:12:38actually. Absolutely. Yeah, search is really important. Um, search tends to be the Achilles heel of a lot of AI products. It's like, hey, we're not finding the right context. And so it's super important. Totally. The other thing that actually came to mind why I think um data science is as as a skill set is going to take take off even more um because AI actually allows people who aren't as technical but they're very interested in data ways to access it now. And sure we need guardrails. We need to make sure not hallucinations that retrieval is grounded well. All of these these types of things. But if you've got like an

1:13:18chief marketing person or an executive who wants to use an agent to reason crossorganizationally across an entire company that they work for or or their company, I think that's that's some of the most powerful stuff we can use AI for these days. Yeah, I do think practically speaking, maybe there's going to be a new job

1:13:38title. We placeholder we can call it AI scientist. I have no idea what's going to be called, but you know how it is. Every kind of epoch of things, there needs to be a new job title being created to encapsulate these skills. And I think Kareem, as we discussed, like he's a statistician, I think, by training, and he he maybe saw the word data scientist and always had suspicion

1:14:05towards that. Yeah. Yep. Cuz it it is natural to have suspicion of like job titles as we go through epox, but it's also important to see like this is all the same. This is all very similar. Um, and you know, let's not get caught up in job titles. But I do think there is going to be a new job title.

1:14:21May maybe you should coin it. I I think you just coined AI scientists. So, but I actually love that as a title. And it does it makes me think um before when you were talking about forget about titles, think about tasks, what you do, the these types of things. It does actually remind me at Cypi um many years ago before the plague that we seem to have you know not talk about in anymore at Cypi in Austin I was sitting with Anthony Scopats um who um old friend big guy in in in Python and I said to him I said to him do you miss being a

1:14:54scientist because we both used to be research scientists and he said what do you mean I said I kind of miss being a scientist and he said Hugo you are a scientist and I said you know what I mean and he said no I Hugo, because science is something you do, not something you are. And and my take and I was like, and I still do employ the scientific method. Um, a lot of businessmen are some of the best scientists, business people I've I've ever met as well. But taking it back as a doing word and not a being word, I think, can help us reinvigorate what it

1:15:22means to be a scientist through doing data science and doing AI science. I really love it is what you do is not what you are. We should we should plaster that somewhere where it's we we can't stop looking at it. Well, I think definitely with my um image generation skills and you as like the meme lord that you are, we can

1:15:44definitely figure figure something out. Haml, um I always love chatting chatting with you and um I've appreciated your your friendship for years, but thanks for coming on and sharing all your new hardearned wisdom and everything you're thinking about with respect to emails, agents, and good product design and data science with us today, man. Yeah, a

1:16:00pleasure being on. Can't wait to take your course. I'm sharing the link once again. It's in the YouTube description. I put it in Discord. Um Shrea and Hamill kindly giving the community uh 25% off. And um I'm excited to see you all in the course and learn more and build more emails with you all. Um so thanks everyone and

1:16:19see you in the next episode. Thank you.