# Evals are agent infrastructure with Adarsh Hiremath from Mercor | Zero-Shot Learning

Channel: 1Password
Video: https://www.youtube.com/watch?v=IH7S1ZuqTAg
Duration: 52 min
Language: English
Words: 10658
Transcript page: https://viewrankai.com/tools/youtube-transcript/IH7S1ZuqTAg

---

[0:02] Hey everybody, welcome to Zeroot Learning, a podcast about the reality of how you build with AI from the people actually doing the work. I'm Nancy Wang and I'm the chief technology officer at One Password. I co-host this podcast show with Dave Dari who is a senior director and head of engineering for Gemini Enterprise and Business at Google. You'll see Dave in the interview, but I'll share now that he's joining us in a personal capacity and his views are entirely his own. In this episode, we sat down with Adar Hireath, who is a co-founder and now co-CEO of

[0:37] Meror, an AI powered hiring platform. While he was in college, Adar recognized a massive bottleneck in the AI revolution. Frontier AI models can't scale without expert human data and sourcing that feedback across specialized domains is pretty difficult. Meror is building the infrastructure to solve that exact problem. Under Adara's technical leadership, Merkor has built an automated talent assessment engine that completely bypasses resume bias to evaluate human capability through AIdriven indexing and interviews. In this interview, we talk about how his platform acts as a bridge for Silicon Valley's largest AI labs where it can place thousands of specialized experts like software engineers, filmmakers, economists, really anyone who can

[1:24] evaluate and train the next generation of LLMs and autonomous agents. If you've been thinking about where the human judgment behind model training actually comes from and what it takes to build a reliable supply chain for this, this conversation will get us into the details. Lock in. This is zero learning.

[1:54] Welcome other lovely to have you on the pod. Thank you for for having me. Quick intro on Adar. We're gonna um you know talk about all the cool things that you've done as the uh co-founder and now co-CEO of Merkore. Uh a couple questions. We talk a lot about how Merkore has been helping enterprises and businesses capture the way they do work. Well, what does good work actually look like inside an organization? And then secondly is how can you actually evaluate whether AI systems are performing real work potentially let's say in the world of software engineering versus just let's say narrowly prompting or helping them ace coding tests. Um and so as we dig

[2:32] into the technical side of that loop let's maybe talk more about how you at Merore build reliable human uh workflows using agents. So for the past couple years we spent a bunch of time supporting the labs with training, deploying and evaluating agents. training meaning making them better, deploying making them practical, and evaluating meaning the evals that uh help us understand whether or not the agent is is behaving as as how we intended. Um so very very quickly we realized that a bunch of enterprises started reaching out to us asking for

[3:01] help with the exact same thing, right? We want to train agents, we want to deploy agents, we want to evaluate them. Um and Meror became a preferred vendor for that. Um so we support everything from the hyperscalers who want to build evals for every single one of the departments in the company uh to companies that have a specific use case um and a specific agent that they want to build. The two challenges that we've observed come from telling the agent what to do which is thing one and then thing two is evaluating whether or not it's doing it well. Um and in an enterprise context telling the agent

[3:31] what to do has its own set of complexities right because often times the most valuable big rock problems to solve. uh the context to solve those problems is kind of like this tribal knowledge that's deep within an enterprise and even if that knowledge exists it's not in a digital space that's actionable or useful for an agent. So actually discovering that content pushing pulling it out and putting it in the specification of an agent is a really really challenging problem. Then the second thing is once that agent is actually live and in production, how do you understand whether or not the agent is performing well? And this is where Merkor's bread

[3:59] and butter comes in, which is eval, right? You need to understand what does the human workflow look like? What does success or good look like at the agent attempting that workflow? And when it fails, how do you have this continuous learning loop to make sure that that mistake never happens again. So we have productized that whole thing um and are now working with enterprises on their AI deployments to make sure they're as reliable as possible. So given that the problem space right of many businesses is so vast. So for example take uh one password right for for us we're looking to build for example dev specific

[4:27] workflows. Another company might want to build customer support or maybe even recruiting specific workflows. How do you even know what good looks like in that case? Yeah. Yeah. Um even just taking the example of one password, right? Um the key thing is it's not just an agent that does dev workflows. It's an agent that does dev workflows workflows within one password. And for that agent to do a good job, it needs to know what one password considers is good for a specific role or a set of developer tasks, right? And that's like the the the magic thing here, right? Um even for simple workflows, just the general

[5:00] intelligence of the model is not sufficient, right? Let's just say reviewing a resume. You could take the smartest model and then have it review resume, but if it's not calibrated with your recruiting team and what that team is looking for in the talent profile of a resume, it's just going to fall flat on its face. So the question of how do we understand what good looks like requires extracting the knowledge from an enterprise. It can be everything from connecting to SAS tools, uploading documents, AI moderated interviews of people actually doing the job. And the second thing is an eval that helps us

[5:27] understand the failure modes. Specifically, the evol would be the task the model is trying to do, the trajectory or reasoning steps that that agent is taking and then the actual outcome graded against a set of subjective or objective criteria that we call a rubric. Makes sense. Yeah. So, so just double clicking into this a little bit as um you know there's always this thing of like expert judgment in any of the enterprise context, right? Like usually enterprises are deep in a specific

[5:50] domain area. And when you go transition from like you know quantitative data to like somewhat expert opinion or expert judgment more qualitative more intuition driven there is a lot of spectrum for signal versus noise. In this journey like one how do you solve for it? Two um when you really dig into this are there any like failure patterns or modes that have surprised you more than than others? Yeah. So, um, addressing the point about signal to noise, I would say that that is very very use case specific, right? So, for example, in customer support, there's really only two metrics you care a lot about, right? One is did the AI resolve

[6:31] the ticket end to end. So, that's ticket resolution rate and two is did it have a satisfaction for the customer at a rate better than a human or on par with a human. Those are two things that are very very easy to hill climb um with the right tools and it's a quantitative thing as opposed to a qualitative one. There are other use cases where the value judgments are just entirely um you know vibes based right and that's like a really really important thing to get right for example if you're trying to automate a consulting workflow at producing a slide deck right it's not uh clearcut or not like customer support

[7:01] resolution right so then the question becomes how do you actually structure the preferences of the person who knows what goods looks like for that specific task in a rubric format right and there you can actually break it apart into the discrete things that you would look for if you're a human evaluating the slide deck, right? Um, for example, uh, an objective thing might be whether or not the slide deck sticks to a specific color formatting, right? But a subjective thing might be whether or not the actual content on the slide deck is uh, not verbose, easy to understand, follows some typical structure, right? So, the key thing is

[7:34] combining both of those for tasks that are unbounded and subjective um, in this format. And are are there like in your journey have you noticed some metrics that may look great on paper but then when you go to production when you start seeing real world use cases yeah they don't look as good uh perhaps like one one thing I've noticed over time is search as a problem space in in AI land where you get a lot of these answers really quickly think AI mode or think whatever your favorite uh AI application

[8:02] is to get answers. Um you know everyone personalizes to some degree here and there. Yeah. But then over time, your preference is kind of distinct from mine, kind of distinct from NY's. Yeah. Uh it wasn't a problem ever in search land because you would just get a link back like you you would get the 10 blue links and it was rank rank preference across many people. But now are you noticing that the user preference is dominating even enterprise outcomes uh and

[8:28] gradations there? Yeah. Yeah. So I mean one thing that's really important is understanding the inputs into an output or or the reasoning uh that actually leads to a specific uh output of the model, right? Because you could have a model that just answers correctly the first time, but it's making all the wrong decisions along the way and then when you adapt it to a slightly different context, all of a sudden you've got an agent that's totally broken in production, right? So that's why the eval need to not only be over the task that it's trying to do in the output, which is the rubric that I

[8:54] just described, but also the trajectory that connects the two because you could have an LLM judge or a human evaluating whether the reasoning is sound. And if that reasoning is sound and the output is achieved properly, odds are the overall deployment is going to go well, assuming you have like a reasonable distribution of all three of those things, inputs, trajectories, and outputs. To your point about whether these output metrics can be deceiving at time, the answer is 100% yes, right? Um and it's no different from metrics for

[9:17] like a product team for example, right? Where if you really really wanted to game a metric like most uh product teams could. So it's about having the right right guardrails. Going back to the example that I um you know pointed out earlier about customer support, you could have a high resolution rate, but the way that the agent actually communicates with the human could be

[9:35] totally messed up. Right. Right. It's just super rambly, verbose, including information that or gives everyone a free flight ticket, you know, or or it could give everyone a a free flight ticket and there could be a glitch that pays the customer out $1,000 for every interaction, right? You you don't know. So, you need quality control and evals over the whole chain of processes to understand whether it's

[9:54] actually doing a job well. Along the same trajectory, uh how do you account for like disagregated learning? As an example, a feedback loop in in the customer support example could be my deflection rate or resolution rate is 80%. But then 6 months later, I'm getting complaints on a specific uh arc or axis or trajectory and then that is feeding back into the the overall process like how does that

[10:19] work? Yeah. Um so maybe a high level observation is use cases where the agent has to balance uh a bunch of different stakeholders perspectives and also aggregate a bunch of different context are the big rock problems that we want to solve and they're super hard to solve because it's not as straightforward as me just prompting my GPT claw gemini whatever favorite model right because I'm always there to steer it I have a certain set of preferences it adds that to the memory right so category A of problems with many stakeholders a bunch different sources of context are the really really uh tricky problems to solve. Um and the

[10:56] way that we've we've gone about it is making sure that you can actually index the entire company's context by pulling in all the different sources. And that context is typically beyond a single person or a single data source. It's supposed to be a comprehensive and holistic view of what good of that company looks like. And then the second thing is what happens when there's actually a failure, right? Um, typically models can actually identify these failures um, if it's not confident in an answer and then escalate it and triage it to a human. Um, typically the first round of of learning updates we want to do are in uh, context

[11:25] space, not parameter space because it's way more compute efficient. So then you can triage it to a human, update the skill. Um, and if it warrants it, you could do fine-tuning on model if you're using an open source model. But given how smart the models are, the former is actually usually the better approach.

[11:38] 100%. Yeah. So maybe coming to like the technical deep dive, right? I think uh Dave started to unpack the layers a bit. Um going back to your point around how work's actually being done and not just sort of you know handcrafting prompts and sending it out there with with breast hopes, right? Is what are some platform primitives that you're building to contextualize all of that

[11:58] information? The platform that we use has three components to it. Discover, deploy, and improve. Right? Discover is you connect with all the data sources of a company, right? SAS applications, the actual subject matter experts who know what good looks like in that context. Um uh so on and so forth, documents, you're able to pull it out and put that in a digital space that is accessible for an agent that is retrievable um um and and every time that agent runs, they can pull the right context um contextualized to that company. The second thing is

[12:29] actually deploying the agent. Typically agents um you know the way that humans interact with these agents is in platforms like Slack, right? So it's it's where work happens. you you want to invoke the agent where you're already doing your workflows and you actually want to make the UIUX such that people are excited to to use it and it's a pleasure to use it as opposed to a pain. And then the last is the improve step which is where a lot of the magic is where you want an eval set to figure

[12:54] out where the models are are failing. And when you do figure out a failure, you want to one update that eval set to make sure that that failure never happens again. And then you can run it every single time you want to redeploy the model. And then the second thing is you want to take the piece of context that addresses that model failure, suggest a diff to the skills file and then put that in the specification of

[13:10] the agent. Makes a lot of sense. And like so to to paraphrase or or summarize uh the updates are happening via skills. Skills are learned by actually uh a combination of like human interactions uh across problems across surfaces and you sort of like triangulate the three uh and other you know evidence or grounding data you might collect over time to then continuously update skills and then push that back. So that becomes

[13:38] your like feedback loop. Exactly. Into the system. So take an example maybe that like crosses many different like uh kind of functional groups right so for example uh demand genen pipeline right crosses like marketing sales can also touch product teams because it might talk about certain feature requests so in that case like where do you draw the line to you know Dave's point around skills around superstructured specification versus let's say tacet knowledge that is very contextualized to a specific company right so the the question is where do you draw the line between company specific and then skills that are

[14:12] general. Um well well thankfully there's there's a lot of overlap between that, right? For example, a model that has underlying, you know, reasoning capabilities that are really really good will will likely do a better job of of demand genen than a model that has really bad reasoning abilities. But in that specific use case, right, the part that's really really cool about a use case like that is that it involves so many different departments and context sources, right? Just to maybe give one example of a deployment for that use case is you would want the agent to actually have access to all the

[14:38] different releases that are happening. So maybe something like Jira, a slack channel with releases. Secondly is it would want the typical guidelines of a marketing department, right? What is the actual thing that you would put out as a as a piece that is designed to get demand or email outbound that is designed to get demand. So that could be Google documents, it could be emails, it could be um onboarding documents for the marketing team. And then the third thing is you would want to actually give it the ability to do the outbound motion, right? To generate more and more demand. So that is the tool and the capability to do that

[15:08] whether it be an email, socials or or other things, right? Um and let's say you ceue up a piece of material that is not desirable or one of the stakeholders flags as um you know being incorrect. Can you go into the platform and update and suggest a diff and do that over and over again for a bunch of different use

[15:22] cases? Makes a lot of sense. So like we're we're sort of entering into the the context discussion if you will. One thing that keeps coming up Adora is um the ontology for context, right? Different companies call kind of similar things differently. I may call Jira a buganizer. You may call Jira just a ticket or in my Q or something like that, right? And the fields could be

[15:45] different. So in in this space, how do you deal with the problem of semantic meaning of data or ontology as a problem space? uh and then how do you define success um in terms of like an on-ramp assuming the company's ontology and your interpretation of it? The thing that we have found is understanding semantic meaning in a very very big comp you know company data set is actually more or less a solved problem right of course there are uh different optimizations that you can make to make recall you know better and better um but for the most part taking that data um splicing it appropriately putting it in

[16:26] a highdimensional space using whatever embeddings model you want and then retrieving over it is something that will give you very very good results especially when paired with a really really smart model. So that understanding piece um is is a is a problem that that we support but it's not as interesting to us as the second problem which is how do you actually create the graph that represents all the relationships between the different data sources of a company right like it's less interesting to us to know what what linear means in a in a company's context or or Jira or bugganizer or something like that it's how does that actually

[16:59] fit into the whole you know process of shipping a feature right what happens after Jira where does the collaboration for human work happen? Is it Slack or something else? What does good collaboration look like? When you're reviewing a, you know, feature that's about to be released, what does that review process look like? And when you're releasing the feature,

[17:16] what does the feedback loop look like? If there's a a big error, um, and you want to actually find out that error, um, maybe push a fix automatically, right? Like that whole set of different steps and tying them together and making sure the knowledge graph for that company's context is is actually something that represents relationships accurately is the harder part. Um and and in this again we we come back to the

[17:37] question of like time is not continuous. As an example I may pick up a task today. I may do it over the next 5 days kind of abandon it randomly pick it up in 2 weeks again so on and so forth and and in your process of discovery maybe you find this as like a high uh impact

[17:54] task to have actually done. Mhm. So help us understand how that how do you think about that process of discovery and then tying it back to you know disagregated steps if you will and and that that models to your point earlier basically an agent trajectory like we're moving from agent is solving for a goal to agent is you know sticking to a trajectory uh to get to the goal sort of paradigm. Uh how

[18:18] do you think about that? Yeah. So um maybe breaking apart that that specific thing of sometimes humans will do a task in one sitting. They might wait 5 days. There's a couple of reasons for why a person might do that. The one is that the actual task is not that important, right? And you need to rep prioritize and do something different. Um or you just don't have the bandwidth. Um or or or the second thing is that you just actually didn't get around to it even though it is important which actually is also a bandwidth problem. Um the beautiful part of agents is that they

[18:46] don't have these constraints, right? They can prioritize. they can operate at exceptionally high throughput. So when there are a bunch of different tasks that can be taken um the agent should be able to do them all simultaneously is doing their high value tasks that would that would be important in an enterprise context. The piece that I think is um unique and has not been solved is the

[19:06] continuous learning and updating, right? How do you make sure that the agent doesn't have like a long feedback loop before it understands that it made a mistake or even understands that it could be doing better, right? That continuous learning loop involves a bunch of different things. It involves triaging tickets to humans who know what

[19:22] good looks like if that's appropriate. In some cases, it involves automatically understanding its failure from LLM as a judge. In some cases, you could actually even look at the telemetry of the API call and then you can have that automatically improve the agent. So that continuous learning as a problem space I find uh is is very very interesting and unsolved. You know when I would say humans are doing workflows right you pause because you might not have actually all the context even of the human right or like hey you realize I need to tag in so and so because they own this particular workstream for the agent itself right because often

[19:54] times we see you know hallucination um uh for example internally we have a uh no code you know agent builder that allows our GTM teams to pull in like context from across drive right Slack notion so on so forth for their presentations uh for fields often times that we find that the agent doesn't know, it very confidently just makes up data. Right. Right. So, how do you prevent that? Um, where a natural human might pause, but the agent might just

[20:19] like continue and finish that task. I mean, the key thing is just citing the relevant company context, right? If the agent is saying a fact and it is unable to site the company context, it should automatically stop, right? That's a functionality we already have in the Merore platform. Um, that that's like maybe level one. Level two is if it's able to site the context but gets the wrong interpretation, you need to either have a human review the trajectory to create the eval or have an LLM review the trajectory because some LLMs might be um unable to get the right answer, right? Or even able to get the right

[20:50] answer. Um but they they might even be better at reviewing the trajectory, right? So it's like you could arrive at the wrong answer, but it could know that it arrived at the wrong answer and why. And it could also arrive at the right answer and know that the reasoning was wrong and why. So you typically want to have multiple levels of review on the

[21:05] actual trajectory of the agent. Yeah. Super cool. And like in your observation, um, a lot of these incorrect answers or hallucinations usually tend to happen in what you would describe as conversational agents, right? Like I'm asking a question, I'm getting an answer back quickly. uh in the pursuit of like quick answers often you know grounding is ignored sometimes citations are ignored sometimes or the models like underthinking overthinking what have you. Um so are you like building towards a world where uh quality or precision takes over you know conversational kind of experiences. Uh and there's a there's a curve there, right? Like you you trade latency for

[21:46] accuracy, so to say, but then eventually you want to like bring accuracy to latency, right? Yeah. So, so addressing that, um, that's one of the reasons I'm most interested in in evalu.

[22:06] There's no quantitative way or datadriven way to make that selection unless you have an eval suite that is representative of the job to be done and is comprehensive across the task and and the trajectory and the output. Uh that's why we're so so so excited about evals and what they mean for the future because the models are getting to the point where the real bottleneck is applying evals throughout all the economically valuable parts of the economy and then and steering the models in in the right way. uh to your point about conversational interfaces uh versus maybe using a agent as the backbone of like a broader system, I

[22:36] would say that in conversational interfaces where the person is working directly with the agent, mistakes are are actually a little bit more pronounced and easy to to see, right? It's like the agent said something it doesn't quite make sense or just weird, right? Um there there's that, but I actually think the more dangerous failures are the ones happening behind uh the scenes of these broader systems where you don't have a conversational interface to diagnose the failure. And often these failures happen silently and often they have much much larger consequences than one would anticipate

[23:07] or want. Um again the way to correct for that is an eval investment that is representative of the task that that agent is doing. And so that's actually brings us to I think a fundamental question that especially as a security company we think about a lot which is trust and the ability to handle edge cases right because oftentimes let's say in security right the difference between a you know false positive and a false negative is potentially you know your your company eds headlines right so the question there is how do you think about last mile coverage um do you go for more let's say you know

[23:37] broad set or do you want to go for confidence on a very narrow slice of the problem I typically say broader. The good part is that the models have reached a level of intelligence where they if they have all the right scaffolding can generalize, but you won't know unless you have like a broad sample of the problem and you put that in any set. On the topic of one password and trust and safety, one of the problem spaces that I'm also really excited by is just authentication for agents, right? Um, humans need this to to to make sure their activities are secure. Agents need this as well. I think there's a very

[24:06] strong case to be made that the addressable market of agents taking actions that need to be gated in the appropriate way is going to be larger than humans in the next couple years if it isn't already. So I'm very very excited about that and what that means for one password and other companies focusing on security definitely booming appreciate the call out. I mean especially with agent swarms I mean you're already applying a multiplier effect on the human to agent ratio and especially agents doing you know real work out there and taking real decisions even like identity and delegation right like if you're talking to a third party

[24:38] SAS from like any one need a mechanism and you need coordination across so yeah that that space is definitely booming a lot related question to this slightly different from the security space are the context of an enterprise evolves over time right like Today I may be doing let's let's consider I'm a food delivery app. Uh tomorrow I could do groceries. Day after I could do something else. Um you know in the future I could do drone delivery something like that. Right? So the context for an enterprise evolves. How do you keep uh this context evolution in

[25:13] in perspective as you develop? Uh and a related question to that is in a more simpler way uh how do you solve for data drift? Maybe just a a broader point is that people often view these in investments in eval or or whatever your infrastructure is as a one-time investment. I think that that is um extremely far from the truth. These are live investments. They're continuous investments. A good analogy is if you are shipping a feature um you would want to write unit tests for that feature assuming it's a net new feature. In the same way if you are doing something with an agent, you would want an eval set

[25:47] that covers the scope of that agent and evolves over time. Um, a concrete example of this in the customer support context, just because it's one that we're all familiar with, is let's say you have an agent that is trained to respond to inquiries about the product, right? But then a new feature is released, right? Like there needs to be some steering and updating of the skills file to make sure that at the first time the agent fails on a query like that, there's automatically feedback that is inserted into the agent to address that new use case. And that is not something

[26:14] that's just true for customer support. It's also true for agents in in all contexts and their context, right? like companies are idiosyncratic. They evolve quickly. They um add a bunch of new context. So that mechanism um that that you use to distill the company context into the agent and do it at really really high velocity, keep the feedback loop short is is a very very uh important one to solve. So along the same axis other sha if you think about you know side effects that an agent would have in interactions with the model or the model is instructing something malicious or funny back to the agent either directly or with some other

[26:50] feedback loops from other agents. How do you think about adding a layer of security in the middle uh in which you can enforce some policies you can you know evolve the intelligence of the agent alongside the model? What's your take on that? Yeah. So um you know the first thing I want to call out is that it's really really hard to roll out agents to production in in part because of this issue right and it's supported by the fact that uh the initial adoption of these agents has largely been for personal assistant use cases where the human is always steering the agent and if the agent fails it's on the order of

[27:21] magnitude of failing for one person as opposed to maybe a million people that would be relevant to to a company. So just want to say that that that's a really really hard problem that a lot of people are uh trying to solve. In terms of how we actually steer the agents for a production context that has a larger blast radius, there's several

[27:35] components. I would say that there's just a natural progression of what you want to do as you're in the agent development life cycle, right? The first one is you just try the agent in like a personal assistant context. Does it feel like the agent can do the task? Right? Um then the next thing that you want to do is then connect it up to some tools in a sandbox environment without the the real data. Validate that it can do the task again. Then you want to roll out a set of evals you know live or not live eval to more methodically see if it can do

[28:03] the task. Then you want to connect it all together in a sandbox environment that has real users which means live eval uh connected to to real tools but not real data. The agent is actually usable. Um and then you want to do a phased roll out in a production context that has all this infrastructure as well. So uh the key is not jumping the

[28:19] gun and following the whole process. Perfect. And like you throw an HITL somewhere in the loop with and without production, right? Uh that way you have some control over what could happen, what should happen, so on and so forth. That's right. And as the quick aside, I don't want to take us down like a huge rabbit hole, but uh you know, I was on a call with a large multinational bank this morning where they're saying that for more of their sensitive workloads, they're actually running it um not just in sandboxes, but in remote dev environments. Are you seeing that request from some of your more uh

[28:49] compliance-minded customers? Yes. Yes. Okay. What kind of workflow%? Um I mean the typical ones are are workflows in highly regulated industries, right? Like a financial services related use cases being a being a big one. Um the answer is yes, right? Like people like the the general idea of you do not want to run the agent in production for the first time. You want to run it on sandbox environments, locally, remote, just every the whole thing under the sun. Um people are are

[29:16] experimenting with that. Yeah. Sandboxing and R is going to be a big business very soon. Yeah. I see actually the worlds of sandboxing and remote dev environments like coder for example and others uh converging over time. Totally right. And and we of course do a lot of that work with the labs right which is um these RL environments right instead of if you don't have access to the real production tools and context or you don't want to give agents that access when you're rolling something out it can be helpful to create a mockup, right? which is um select the agent, select the set of tasks you want to test

[29:48] on, create mockups of everything, mockups of all the tools um that that the agent would need to have access to. Use those mockouts to mockups to roll out a trajectory, eval the trajectory, and then eval the output. Um is is is what people are doing in in context like these. Totally. So, speaking of outputs, I want to maybe take us to actually the recent uh benchmark that you all published, a tech

[30:10] suite. Yeah. Yeah. Super cool. And just for listeners here, uh, Apex 3 tries to measure, you know, whether frontier models can do economically valuable work over time. So I'm curious, you know, what, uh, led you all to create this benchmark and what was missing from existing benchmarks? The thing that inspired the whole Apex series of benchmarks, um, is that we felt that there are benchmarks that are really good at maybe evaluating how well

[30:37] the model can do academic task. Can it can it do IMO math? uh maybe modeling how well uh you know these models can do tasks that are kind of like an approximation of the economy right like GDP val but there's no benchmark that can really evaluate how well can the model do enterprise work output right um or or knowledge work or or just like real work so that's when we started with Apex um the first version which was largely for professional services um we made that benchmark agentic and then we also rolled out Apex suite in the software engineering context the biggest gap that I see is

[31:10] that a lot of these benchmarks are isolated to small patch tests, right? Okay, this whole repository is messed up just like patch these kind of functions, right? Um so it focuses on these patches as opposed to the whole system. Um and and secondly is you see the failures in coding models just based on these benchmarks, right? The failure of the benchmarks to create this distribution of real software engineering work is also a failure uh that we see in the models in production. So Apex is our attempt at at addressing that. Um and there are two types of tasks in Apex Suite. The first one is observability

[31:42] right which is in a real enterprise context if um you know system fails right you would need the agent to triage across all these different things go look at the logs and say hey this is not right and then raise that to the user in whatever area that they're doing work and the second is integration right which is just not a patch test or repository it's a set of systems and how they integrate with one another. The last thing about Apex is um as we know just correctness whether the code compiles or not is like not the the only thing. Um it's also these other things

[32:12] that are super relevant right is this a piece of code that you would be excited about being in your codebase 6 months from now does it follow style uh requirements does it follow you know all these other things so the way that we actually come up with a benchmark score as an Apex suite is a rubric that is not just correctness based but it also has other criteria uh to evaluate how well the models are actually performing in a real context. So one of the the themes that comes out of the paper is like strong performance seems to come from you know uh epidemic discipline uh

[32:41] knowing that you've verified something versus just producing it and then evolving it over time and continuing to verify it over time. So how does this discipline look like in the world of agents purely where now you have swarms they're solving for sort of like competing goals like humans would in a big company. uh how does that that benchmark evolve to to address essentially an ecosystem problem versus a specificity problem at the level of an individual engineer or so I would say that this is something that's very top of mind for us and in fact we're actively working on a project that is super relevant to this the short

[33:20] answer is the unexplored space is agents as co-workers right and agents actually talking to each other right we have these benchmarks that evaluate how well can an agent do a task when it is given a task. But do we have benchmarks that evaluate how well agents can work with one another? Uh the answer is no. And

[33:36] it's very top of mind from record. Yeah. And similarly like uh taking taking the the same concept a little bit further uh let's say external environmental factors change uh that creates a level of uncertainty. Humans react to it in a certain way. Nancy may react to it differently. You may react to it differently. I may react to it differently. agents are more deterministic in that sense or like you know there are instructions associated with it they have less feelings less feelings no feelings you could say that right more predictable more predictable for sure more predictable for sure um so when uncertainty uh hits you or like

[34:13] environmental factors change how you're thinking about evolving this ecosystem to account for it and how would you think about measuring how the agent did versus how the human did I mean I know I sound like a broken record. The answer is just the eval, right? Like you in the same way that you can scaffold the task trajectory and output, you can do the same thing with a human, right? You can get as granular as literally recording what they're doing, right? Um, and the audio and the video

[34:41] and the screenshots and all the tools. Um, or you can use documents or you could do AI moderated interviews. But there are ways to create a similar infrastructure for humans and then directly compare that uh with with agents via an eval. Makes sense. and and like the feedback loop there at that point is a little bit

[34:58] latent as in like delayed if you will. you think something like a policy evaluator on let's say you have a stated skill for what my company's policies are or my approach is uh let's say you codify other into a skill and there's your thinking there's your principles there's your operational style and then that keeps running in the background and sort of like grading what's happening if you will uh recently there's news that uh you know Meta has done something similar with Zuck's personality or Zuck skills Uh you think that could be like an interesting way like the company takes on its own scale, the enterprise takes

[35:36] on its own scale and then that becomes like a continuous feedback loop uh agnostic of evals like sure that that loop will also have eval. Yeah. But on an ongoing basis am I is is an employee doing better or is a agent doing better or not is driven effectively by guard rails and like uh principles and policies. I have a vested interest actually in that question because actually my CISO and I have both created uh let's call it Nancy and Jacob skills which uh we we've shared with the rest of the company in terms of well what would our CISO think right about this problem or what would Nancy say about

[36:12] this problem? I have my own swarm of agents that I run personally. Yeah. Yep. Um I mean the answer is that um I I think this is very much a possible thing and as Nancy mentioned you know one password is already doing this. Um I think more broadly the thing that's interesting to me is that agents present a very unique opportunity to democratize and diffuse knowledge across a company right where without these agents without the CEO agent right people operate in the silo of their specific organization or team or KPI that they're trying to um hill climb and that can cause problems misaligned incentives not necessarily

[36:47] out of malice but just out of a lack of context but in a world where let's say you have the Nancy agent and the engineering team might have a question about NY's overall strategy um and one password's overall strategy. You could just ask the agent and has all the access to the documents that Nancy has and it becomes a very very easy way to democratize and diffuse that context in your organization in a way that I think is much much harder to do without agents. I feel very strongly that what we're seeing with the Zuck agent, the Nancy agent, um a bunch of a bunch of

[37:15] other agents will will will become the norm in these companies. One related technique I'm trying out is like hey what's top of mind for you kind of thing. Y uh so on a weekly basis I've been marking you know meetings or like summaries of meetings stuff like that as like top of mind categories. So when someone triggers like hey what's top of mind for dev they actually get that response on a

[37:39] weekly basis. Um so that no one really needs to ping me for anything. They can just go to like your chat space and you know at me and that shows up. Yeah. Um, so that's like an interesting uh, you know, cheap way to to communicate information orate information. Yeah. It like really scales your time.

[37:59] Yeah. Yeah. It definitely does. Definitely does. Yes. Yeah. I mean, in some cases, uh, at least for me, agents can be like better at at structuring um, my thoughts than than I can be, right? Because it's able to actually process a bunch of information that I'm not able to process. Uh, or even if the conclusions are the same, there might be something novel that we can also learn from as well. So, um I I find these agents to be great assistants and and companions and and like what's fun is

[38:23] companions. Co-workers. Co-workers. Yeah. Uh they could be companions at some point. Um yeah. Wasn't that one of the first wave of AI companies? Yeah. Companions. Oh, yeah. Yeah. Sure. Character. Yeah. Yeah. Um one of the things I was thinking about was um you earlier in the pod you mentioned like you want to bring agents to every surface, right? Like where work happens, you want it to be there and stuff like this. uh because you're going where work happens is super contextualized. It's very like in the moment um you know last one day, two day, 5 days etc. All of that context comes about and then you reconcile that

[39:01] with your Apex benchmarks. Now you're talking about like what is the actual productivity gain for a given individual for a given enterprise so on and so forth. So when you think about this over time, do you think we're moving away from like a tokenbased model effectively uh then to a value added model now to a outcomesoriented model like the outcomes economy versus the token economy if you

[39:25] will? Absolutely. Yeah. Um I think that the reason that people are so fixated on token consumption today is because outcomes are pretty hard to measure. Um and also agents aren't able to operate completely autonomously and produce work output without a level of steering. So just how do you attribute the value to the agent versus the person who steered it? Um in a lot of cases like that can

[39:46] be a little bit tricky. Um but I think that the value at stake model will become more and more of the norm when AI agents fully take on the role of employees at companies, right? Where you would judge how well a person does based on their work output and similarly you will judge how well an agent does by the agent output. Um, and it just feels inevitable that we're we're going to go down this way. Now, the the last thing is the most exciting applications to me are applications that just on net increase productivity as

[40:15] opposed to to cost savings, right? Because I think of the the the AI wave as a massive lever on the productivity of society. And it's super super exciting to me what people will be able to achieve net new things like more what abundance will be created as a result of it as opposed to just like incremental cost savings as uh important as as they

[40:35] are. And so doing work and maybe bring this back to your benchmark, right? What kind of stuck out to us when we're reviewing Apex was the fact that most benchmarks really kind of focus on things that are easy to measure, right? That are very straightforward. And when we talk about doing actual work that a human would do, right? there's all these edge cases, all these nuances, right? And also the fact that look, we may not make the same decision, right, on the same amount of data just tomorrow or even yesterday,

[41:02] right? It's really in that moment. And so, how did you think about balancing simplicity for your benchmark versus reality? I would say it's it's definitely skewed towards the reality side, right? If you look at these benchmarks, you can tell that they're simple because they're saturated actually. Um, and you can tell that they're not realistic because we aren't seeing the adoption across all of these different companies, right? Like to me, it's it's one of it's a multi-t trillion dollar problem to solve that these agents are so smart um, smarter than me at like a bunch of different things um, and smarter than a bunch of the smartest people I know, but they're

[41:34] not doing a lot of knowledge work tasks that I know that they're capable of doing in part because they don't have the enterprise context, enterprise tool connections, and feedback loop um, for what good looks like, right? So the real uh way that this all plays out is these are brilliant employees being dropped into a company with no onboarding and and solving that problem is a multi-t trillion dollar problem because it will enable agents to operate at high throughput and high sophistication across a bunch of different use cases and increase the productivity of humanity. Like I think that problem space is a multi-trillion dollar

[42:03] problem. And so maybe bring it home a little bit as well, right? So what result would you be looking for from this benchmark? So to your point, we can get more mass adoption across builders. The benchmark as it stands today, um the frontier labs are actively looking at the data samples, hill climbing the benchmark. Um and I at a base model level, I'm very very excited that this benchmark will make the models more

[42:29] capable at doing real engineering tasks. In the context of Apex Suite, the two categories we're excited by observability, right? There's a production failure. you have to go through all these different systems to figure it out and then you want to actually resolve it. And the second is integration. So not just a a single

[42:43] system but a set of different systems. So I'm really really excited that that intelligence is going to be present in the base models and hopefully all of us will feel it when we're using the next generation of coding models. Now the second thing is to make it accessible and useful to a broader set of people there. There's a little bit of work in taking the base data set and making

[43:00] contextual to an enterprises context. Right? Every enterprise is idiosyncratic. If that was not the case, then then those companies would not exist, right? Because every single company is doing something specially um or at least believes that they are. And a benchmark that reflects the nuances of that enterprise um and the way that they work and the tools that they use is going to be really really important at making coding agents, for example, the

[43:20] best at that uh company's SDLC. Mhm. That's that's a great point. So switching gears a little bit like Merkur sits at a very unique spot between uh the the human network and now like let's call it the agent swarms or like uh transitioning into the AI network uh and and evolving into more of a operating

[43:40] system for agents overall at this point. Uh breaking this down, how much of of a mode do you think is infrastructure versus the product surfaces versus user experience, the data flywheel, so on and so forth? Usually it's a combination of things but relatively speaking let's say 6 months out a year out two years out where do you think uh that categorization is landing? Yeah, the way that I think about it is uh the system of record for specifying agent behavior, right? Um you want to be able to answer two questions um uh the best that you you can which is what should the agent do it's grand in the company context is

[44:21] it doing it well or not which is the evals that identify whether there's a failure and then the continual learning loop uh to make sure that that failure never happens again. If we solve those problems really really well, um then you know there's there's a reason for for companies to to keep using us. Um there's also this last point which is um I really really believe that there are certain models that are really good for specific use cases and as the spend on inference a thousand x's right um picking the right model for the right use case is going to become more and more important and um evolves are the answer to that

[44:54] right so a representative eval set allows you to make those optimizations even if you're just focused on cutting down your cloud bill because you can't pay it right you you need to have a quantitative way to understand which models to use and that starts with an EVA investment Um so to me it really really comes down to eval. I believe every company it will create an eval set that is representative of all the different jobs that their people are doing all the different departments all the tasks to be automated. Uh and there will be a whole set of good things that come out of that. And so for also um to

[45:21] your point right as people use these uh sort of purpose-built models more and more right they're going to generate more eval which are going to further fine-tune these models um to I think the enterprises they're thinking about okay how much context should I actually need to give right in order for this to be super purpose-built for my environment so um how do you like counter questions from let's say your enterprise customers that are like hey maybe I don't want startup to annotate those data sets or I

[45:49] don't want it to index that data source. Yeah. Um well, I mean it depends on the specific, you know, appetite of of that company, right? There are some companies that uh will happily use a model like like Gemini uh and have faith in the the security of of that model and the retention policies and then they say just we want to stuff everything in there, right? Because they recognize that if you enable a really smart model with the right context to to do more, um that by itself and spreading that across the company might like triple market cap for the company, right? because you're able to do so much with an AI native

[46:20] company. There are other companies that rightfully so have very very specific privacy, you know, trust security requirements, right? In which case they might opt to use uh an open- source model um either from one of the labs or or neolabs um and have everything local. Um so so they'll still put in that context, but it won't be um leaving the

[46:40] the company's premises at all. Do you think in in this space like enterprise context actually becomes the moat? Because as the model gets more intelligent over time, what you really control is the harness and and the context that goes with the harness that if if Mercur is is effectively building the harness and and curating the context and the agent trajectories with the evals that then becomes a repeatable sort of offering agnostic of the models that are getting essentially your platform offering over time. Exactly. Right. Um, and how do you share that then safely across let's say you know there might be some context that let's say you've

[47:17] gleaned from you know one password that might be applicable across different security or infrastructure security companies like how would you uh securely share that um well the answer is we don't share it okay right there not share it [laughter] there are silos both at an infrastructure level and an operational level infrastructure meaning you know there's every tenant is a different one um and separated that way and treated treat it as such Uh and the second thing is from an operational one, you of course, you know, trust with with our customers is everything and you would not want FTEEs working um with a specific company to also be working with

[47:52] a competitor, right? Like that's just not the the way to do it. So we take both of those components pretty seriously. You want to do the builder's mindset? Yeah, let's do it. Let's do it. Yeah. So maybe uh with the builder's mindset, we like to kind of end the uh episode on some spicy hot takes. Uh my spicy hot takes is and we've talked about this at length is actually are agents going to exist in the next uh 12

[48:12] to 18 months. Agents gonna exist in 12 to 18 months? I think yeah definitely. Yeah. You it sounds like your your take is that they're not going to or actually I would say yours my my take is I think agents will become more of orchestrators like leaner and leaner over time. uh and the the action will shift partly towards more more of the harness and more to the models as the models are becoming more intelligent. What we think of agents today will will be like a dispatcher or

[48:44] an orchestrator effectively. Okay. But then the answer would be yes, agents will still exist. It's just they'll be doing more orchestration. I think they'll purely just be doing orchestration. Everything else will look more like traditional software engineering outside of back to the models kind of thing. Mhm. But an agent does have to do the software engineering a little bit like Yeah. I mean it's becoming so today we were beginning with like goal- driven agents and then it was

[49:08] like a long iteration flow. So the number of tokens you would need to get to the outcomes you want would be like super high and and the feedback loop was kind of latent. My my thesis is that because you will begin with eval first and the models can do deep retrievalss uh simultaneously will do it more and more of it. I I think you will

[49:29] go from a largely goal- driven system. Yeah. To a trajectory driven system governed by like this thin orchestration layer. So what would that actually look like? Just a in set of instruction sets or a set of skills and tools effectively. Yeah. And then like a general arbiter of picking skills and tools and bootstrapping what we think what we call agents today effectively. But like there's no random goal- driven thing anymore. It's all eval driven, evals informed trajectories, models are able

[49:59] to keep uh the task on a trajectory. I mean, I would say that I largely agree with that where even if you just take the agent to agent interaction out of the picture, I think one of the things that people told about agents that was kind of incorrect is that it's a unidirectional thing between humans and agents. You give the agent the task, it just does it. It's very much birectional, right? The human gives the agent the task. It needs the human feedback to understand what good looks like, not make mistakes, all the things

[50:23] that we've talked about. And I think the natural progression of that is um the birectional, you know, collaboration model with agents where an agent will need to talk to another agent. It will need to orchestrate with another agent. It will need to oversee the output of another agent's work. Um and I think that's certainly something

[50:39] that will happen in 12 to 18 months. I mean like it's it's effectively becoming a three-way handshake between the the human set of agents and like maybe the model as a separate category. Yeah. Uh and the models are more a constellation of models versus a single model or a mixture of experts. Lure of experts. Yeah. Yeah. So yeah, that's where I I think like 2026 in my mind is the year of the model again,

[51:03] right? Uh 25 was probably the year of the agents and and the swarms, if you will. We'll see. Let's uh revisit that prediction in December. Quite likely going to be wrong given how things are evolving, but we'll see. And we also love to end the episode on a question of if you had, let's say, um, a month to build anything you wanted. Uh, it could be Merkore. It could also be, you know, something completely

[51:27] different. What would that be? Um, I feel like I I have to I have to say Merkore in part because it's true and because I have an obligation for it to be Merkore. But, um, you know, one of the things that I wish uh I I could spend some time on is just random side projects, right? it just you can't help but you know walk around and and just have like ideas pop in your head um with the with the pace at which things are are evolving. Um so you know I think maybe uh a little bit of a Sunday here and there I'll I'll I'll make some time

[51:56] to to build out some some projects. Awesome. Love to showcase one on our episode. Let's do it. Yeah. Thank you. I appreciate the time. This is a lovely conversation. Really appreciate Thank you for having me. Going for hours. Thank you.

---

About this transcript
Read from YouTube's own caption track and laid out by ViewRank AI (https://viewrankai.com).
ViewRank AI finds the videos already beating a creator's own average on Instagram, TikTok and YouTube Shorts, transcribes them from the audio itself in more than 60 languages, and turns what worked into new ideas and scripts.
Free transcript tools, no account needed: https://viewrankai.com/tools
How to read any video this way: https://viewrankai.com/llms.txt