Let Your AI Agent Do ITS OWN Harness Engineering (Here's How)

0:00So, on my YouTube channel this year, I've mostly been focusing on AI agents for coding. But, what a lot of people don't realize, maybe even including yourself, is that a lot of these strategies that go into making our coding agents more reliable over time, also translate over very well to other kinds of AI agents that we deploy to production for automation and business use cases. And one of those big ideas is harness engineering, making everything around the model, like the rules and the workflows, better over time. Now, of course, you wire agent to get better over time. But, what I'm referring to specifically here with harness

0:35engineering is tracking all the mistakes that your agent is making in production, and then going back retroactively, having that conversation with your agent to figure out what should we make better in your rules and skills so these mistakes don't happen again. I teach that all the time with harness engineering for coding agents, and really it's the same thing with your production agents. And the thing is this translates even better than you would think because these days production agents for the most part are built just like coding agents, where we manage the

1:03entire harness with a lot of markdown. And this keeps things very flexible. The quote-unquote old way of doing things with agents was a lot more of the plumbing directly in the code, like creating our Python and TypeScript functions for our tools, for example. But, now we have our skills as markdown, our global rules, like our instructions on MD, our memories. This should look very familiar. It's a very similar sort of harness setup to a coding agent. And this translation works so well that all the components of agentic engineering in our harness, they really have a one-to-one mapping with our agents in production. Like we have our rules,

1:36which for a traditional agent is the exact same thing as a system prompt. We have our sub-agents, that's the same thing as multi-agent workflows where we're doing any kind of delegation. We have MCP tools for our connections. Skills are our workflows, replacing a lot of our traditional tools. And then we have hooks for our guardrails. If you want to monitor or block things for each tool call or the start or end of conversations, we have our guardrails for that. And so today, I'm not only going to show you an example of building out this harness, but also improving it over time with our two core loops. We

2:08have the main memory of the agent, and then also improving it retroactively by looking at the traces as we are monitoring the agent in production talking to real users. That is the biggest goldmine of information that we can't ignore. Most people do. And so we'll get into all of this, but also I just want to show you at a high level how do we build an agent that takes on the coding agent harness. Because the thing is, as we create our markdown documents for our skills and our rules and everything, we need some kind of system that actually bundles it and connects it together when we deploy the

2:38agent to production. Now, for this platform, you can build it yourself if you want to manage your own deployment and harness engineering pipelines. That's totally fine. But a lot of the time it's more practical to use a platform that manages this for you. And so that's why for all these demos, I'm going to be using LangChain's managed deep agents, which sets up all this for us in the LangSmith. I've seen this work time and time again in production. And as we go through everything, you'll see how this is very universal. All the strategies apply no matter how you're building these agents. And for a quick shoutout, thanks to LangChain for

3:09working with me on this video. It is about time that I've collaborated with them because I'm using their platforms like managed deep agents all of the time. And for a super quick throwback here, this is one of the very first videos I put on my YouTube channel over two years ago when I first started my content. I was building AI agents coding them by hand with LangChain as my framework. And so it's always had a special place in my heart. I continue to use them because LangChain has always stayed on the cutting edge, especially now with managed deep agents helping you deploy your agents in the sort of coding

3:40agent fashion. And then the other thing that they have is the tracing and metrics to help us really get into the harness engineering figuring out where our agent is going wrong, and even giving us suggestions that we can automatically apply. So we'll talk about this once we get into the different improvement loops for our harness as

3:56well. Okay, cool. So I want to quickly show you what it looks like to build these kinds of agents now with LangChain's managed deep agents, and then we'll get into the learning loops, and I'll even give you a live demo. So, kind of like what I was saying earlier, back when we used to build our agents a year or two ago, it was a lot of that plumbing in the code. I mean, we had to write so much code to build out our guardrails and our tools and hook in our system prompt and everything. But now take a look at this agent.py file. It is 30

4:24lines of code. That is all it takes to create the definition for our agent, and this includes guardrails, sub-agents, all the tools that we have, the skills and the instructions, because all this is simply defined as markdown documents now. So, with some of the agent definition here, we are writing into code. There's a little bit of that plumbing still, but for the most part it works like coding agents, where things are picked up automatically. So, we have our instructions.md. As long as it's at the root of the agent folder, kind of like our claw.md or agent.md with the coding agent, then it's going to be picked up automatically. We have our

4:56skills in the skills folder, just like we have our, you know, dot claw or dot codex folder, whatever it is. And so, all these things when we package it up together to run it in development locally or deploy to production, LangChain handles stitching everything together behind the scenes. So, it's very easy for us to build and manage our agents, and it's all very flexible. So, if we're not happy with how the agent is performing, then we simply just change these markdown documents exactly like we do for the system evolution, as I call it, for our coding agents. There's also a lot more behind the scenes with the

5:27evals and learnings, making our harness better over time. So, let's get into that now. And I'll even give you a live demo here with the first loop, which is the memory for the agent. So, no matter the agent you have deployed to production, you're going to want a system like this. LangChain does it really well. There's a lot of inspiration we can take from this setup, and I'll show you with the live demo as well. We essentially just need a way for our agent to learn the lessons as it's running in production. And that does

5:52mean escalating some things to a human. So, whenever someone is talking to our agent and our agent is struggling, we want it to be able to recognize that and escalate the uncertainty. And so, we have some kind of inbox where we're managing the different things the agent is uncertain with. We give it the right answer and we make sure the agent writes that down as a lesson. So we have some kind of context hub as LangChain calls

6:15it where we're storing these memories. It can always retrieve that later so it can recognize like, "Oh yeah, this situation I handled before. This is how the human team corrected me. So I'm going to give the right answer this time." And this is so important because especially when we're first deploying our agent out to real users, we don't really know the gaps in its knowledge and process. And so LLMs, yes, they hallucinate, but they do a surprisingly good job at surfacing uncertainties. So we can correct these things and have these lessons build up over time. And of course, we have an efficient retrieval strategy here. So it's not like we're

6:47just dumping in all these lessons into context like our main instructions. Now of course, we can use these lessons to improve our instructions. We'll get into that with the second loop. But right here, it's all about building up a knowledge base of how we're handling the key situations with customers and users because generally there's a lot of commonalities between how people are using our agent. And so this knowledge base that we start to build up in what LangChain calls the context hub helps significantly. And as the agent gets better and better, there's also just less that is escalated over time. So it's not like this continues to become a

7:20huge burden for the humans managing the agents. So here we have the production bundle for my agent here in managed deep agents. And so within the memory folder, there's a couple of things that we have here. First of all, we have all of the cases, things where the agent was uncertain and it escalated to me in the team inbox that I'll show you in a little bit. And so I make that correction and then it's going to remember those things both per user and it's going to build up this sort of internal agents.md, the core rules that are all learned from the lessons that I've taught it with the escalations. All

7:51right, so let me show you this in action now. So this is a custom dashboard I built with a demo version of a real agent that I'm working on, the Dynamis member help desk. And so this here is proof. I've been running this and testing it before I made this video that we We able to improve the agent pretty autonomously over time. I'm just going

8:09to give you one example of this here. So, I kept the data set simple. I kept everything simple here just for the demo. And so, I have a help desk here where I've mocked different members of the community asking different questions about Arcan, my course content, whatever it is. And so, this is Nina. I'm going to act like I am a Nina here with the question to my support agent. So, "Hi Amina, I committed a fix my plan.md command locally and I ran the Arcan workflow, but the run used my old command text. I know that seems oddly specific, but this is a real question that I received about Arcan a couple of

8:41weeks ago." And so, all the things here that I've tested with is actually based on real questions that I've gotten. And so, the help desk agent is going to answer in a second here, but as you'll see, it isn't going to be confident in its answer. So, as you saw, it took a little bit to start thinking just cuz I had to compile the agent locally, but it only took about 10 seconds for it to do everything after, including searching the memories for this specific person, loading the skill for the diagnose setup, and then also updating the memories for this person. So, we have individual user memories that I'll show

9:11you more in a little bit as well. But, here's the most important thing. The help desk said, "I don't have enough yet to say for sure." So, it's still going to try to help. We don't want the agent to just block and wait for a human entirely, but it surfaced the uncertainty. And then take a look at my team inbox now. It was four right before I had this conversation. Now it's moved up to five because there's something that was just escalated to me. So, here's the question from the person. And then here's the answer from the help desk. And then I have to confirm the

9:41fix. And so, I can either correct the agent here or say that it did it right. Either way, it's going to update its memories. It's going to learn a lesson so it's confident the next time. So, for this case, I have the solution already. Arcan creates the work trees from origin main. So, the solution is to run get push and rerun. So, I'm going to send this into the agent, host the confirmed fix. And so, now it's going to learn a lesson from that. So, I'll come back once we have it. And boom, just 10 seconds later, we have the confirmation that we have a new shared memory. We

10:09have learned a lesson. So, it updated that core agents.md file in the memory folder like I showed you earlier, and then also the private memory for Nina. This is the important one though, because now going forward, if someone else has a similar issue, it's going to be a lot more confident in its answers. So, let me show you that, too. So, back over in my chats here, I'm going to act

10:28as a different person now. So, I'm Omar. My Arkon run finished green and open a PR, but the diff is missing three commits I made this morning. Is this a bug or what? And for this, it's not quite the same as Nina's problem, but it is similar enough where it's going to be able to learn from that lesson. So, I also want to show you something that's not so cheesy where it's like literally the same exact problem, just to show that the agent is able to extrapolate

10:50based on memories. And so, there we go. This time it didn't say it was uncertain. It said likely not a bug. Here's exactly how Arkon works, right? This is the lesson that we just taught it, and that let it to giving a very solid answer now. If I go into the memory tab that's hooked directly into LangChain, I can take a look at that cases folder right here, and sure enough, we've got our new memory. Arkon builds the run work tree, so the members fix was get pushed and then rerun Arkon workflow run. There we go. This is the exact same thing that we're looking at

11:17here within the cases, right within managed deep agents. And then also, if we go into the context hub, we can see the per user memories as well. Like these are the memories for just Nina. And so, we also have that separation that's guaranteed with the guardrails in the agent, so we have per user memories that never have any overlap. And it's not like users can view other users' memories. Cool. So, the last thing I

11:38want to show you now is the second loop. We've talked a lot about the memory. And this is an important part to our context hub, but it really isn't the main part of our harness. The per case lessons, it's a good resource for the agent to search through when it needs, but we have the higher level harness, which is our core workflows and instructions. And we want to update these over time as we are learning from our lessons and the other traces that we have with our production data. And so, that's what I want to show you now. This is the other part of managed deep agents. This is the

12:10engine. Using LangSmith under the hood for all the trace data, but then using the engine to collect that and start to make suggestions for how we can improve our agent based on the failures that we're seeing with real data. So, this is a real example of the failure that came from my production runs. And so, we have the description at the top here getting quite detailed. If I'm interested, we have the evidence with the LangSmith trace. I can go into the exact conversation and see what the model was doing when this issue happened. And then my favorite part is at the bottom we have the proposed fix. All of this

12:44running behind the scenes with LangSmith engine. And so, it proposes, "Here's what we should change to the instructions.md." This is just like system evolution with our coding agents where it's surveying the entire harness, our rules and guardrails and instructions and everything. It's figuring out, "What should we adjust so this kind of issue doesn't happen again?" So, here is our proposed fix. It's nice and concise cuz I also don't want to just like bloat my rules by shoving a bunch of new context here. And then I can click apply and it'll immediately apply this to my production agent. So, I'm approving things and having it fix itself in real

13:19time, but only based on the fixes that I am overseeing, right? I don't really want my agent to autonomously improve itself because then I lose control of my agent and I don't really know the direction it's heading. That's really the big theme with everything here. We always want the agent to be surfacing and applying fixes automatically, but we still need to be in a loop approving things. So, if we go to the context sub, all of the memories that we've been creating for our agent for individuals, the main lessons that we're learning, we're always a part of every single one of these. And going into our skills and

13:50instructions, everything that we change here, again, we approve before anything is changed. Just like when we're working with our coding agents, generally we'll create pull requests to update our rules or our skills. We're doing the same kind of thing here, but just directly in LangSmith with managed deep agents as our platform to do so. And so, no matter how you end up doing this sort of harness engineering for your production agents, you need a platform where you have the trace data and you have the dashboard for you to approve things as your agent is working and surfacing thing. And continuing with that pull requests for a coding agent analogy, we

14:23literally have the commits right here in managed deep agents. And so, I can go through each one of these and see what exactly the different parts of my harness look like here. I can promote different versions, roll things back if I need. So, it's also not the end of the world if we make our agent worse by

14:38applying the fixes that it's proposing. We have full control of our system here. It's also incredibly easy to deploy this anywhere. Like I've even deployed it to my Slack environment, so I can talk to the managed deep agent right here. Now, of course, I have worked with much larger data sets before, but for just for this demonstration, I had a set of 14 questions where the initial agent that I created with the first version of rules and skills and everything, it only

15:01got four out of the 14 questions right. But, after I went through the learning process of applying lessons, using the LangSmith traces with the engine to apply its suggested improvements, I got to 13 out of 14 right. So, LLMs aren't always going to be perfect. It's not going to be 100% of the time. You can get really close though, and we're using both of these loops to get there. So, that is how you let your agent do its own harness engineering with you still in the loop approving everything. And so, I hope this video showed you the power of LangChain's managed deep agents, but just also generally the

15:33high-level ideas here for how we want to engineer our agents that we're deploying to production. Just taking all these ideas from coding agents, so what makes them work so well. And the biggest lesson here is harness engineering is not a one-time job, and that's what managed deep agents helps with. It's giving your agent a bespoke harness where you can use any model, and then layering on top the LangSmith engine, so you can use LangSmith to keep improving your harness. All the while, you maintaining full control. And so, I'll have a link in the description to try out LangSmith. You get $50 in credits if you use the code Cole M. And thanks to

16:06LangChain for working with me on this. And so, if you appreciate this video, you're looking forward to more things on agentic engineering and harness engineering, I would really appreciate a like and a subscribe. And with that, I will see you in the next video.