Intelligence Efficiency, Ben Geist | Compile 26

0:00So today I'm going to be talking about intelligence efficiency, right? And how we can maximize it. I'm going to start off with three graphs.

0:20And I want to talk about where we are today as an industry and where I think we should go. So where are we today, right? ChatGPT came out, LLM boom occurred, intelligence skyrockets over time. We've seen this, right? We've all seen the benchmarks. Intelligence is only increasing. It's increasing at an exponential rate. Right? Additionally, with that,

0:44we've seen tokens skyrocket over time. We've seen token maxing occur in the industry, and we've seen industry as a whole really adopt LLMs into their production processes, causing this to occur. Right? But additionally, what I think we've seen is that these models are getting smarter, but they're not getting more efficiently smart, right? The more money you're paying, we see the state of the art over

1:12here. This is the fable. This is maybe last year's models. The more money you're paying for intelligence is actually having diminishing returns, right? And this is not necessarily what the world was sold, right? We were sold with LLMs that you could buy intelligence at a unit economic price. But what we're actually paying for

1:32is tokens, right? And this is not true, right? We all know this as technical folk. We know that intelligence does not equal tokens, yet people are trying to buy this but are actually purchasing this, right? And there's a fundamental difference there that exists, and I think will actually come to bite the model providers in the ass a little bit, right? We see this logarithmic curve to some extent, while folks want a more linear

2:00curve, right? Some data to back that up. I work at Ramp, like Dwarkesh said, so we see a lot of this in the data as well. Just some stats is that we've seen a 13X monthly token spend since January 2025, and we've also seen some political things happen in the technological space, right? We've seen Uber, we've seen Meta clamping down on this token masking and really making sure that this exponential doesn't necessarily continue as it affects

2:31the bottom line. We also recently did a version of SWE-bench of Ramp, and we saw this more logarithmic curve, where intelligence is actually getting less efficient over time, not more efficient. And some of the research I've been doing personally is how can we make sure that this graph is more linear rather than more logarithmic, right? And what I found is that context seems to really be the key in order to provide for the models, we provide more context, they tend to become more efficient and need less work to complete the same

3:00processes. I want to give some mathematical foundations of this. So the way I think about LLMs, and AI as a whole, and agentic processes, is that they are really entropy reduction machines, right? So entropy is a measure of chaos in a state. And if you think about LLMs, what they do is they take very high entropy states, they take PDFs, they take information from the web, they take information from very scattered fields and a prompt, and they organize it

3:30into what the user is asking for, right? They fundamentally reduce the entropy of state. I think out of this field of thought, we can actually derive equations that show how introducing context into the field makes LLMs and AI as a whole more efficient. So there's two fundamental views of entropy. There's an information theoretic view of entropy, which is a Shannon entropy definition, which says that the entropy of a state after an observation is

3:58fundamentally reduced. I think this is pretty obvious, right? You have some chaotic system, you measure it, you understand a little bit more, and so the chaos of the system, at least our local representation, is reduced, right? Additionally, the thermodynamic view of entropy says that the amount you can reduce entropy in a system is

4:18upper bounded by the amount of work you put in. And combining these two equations, we can see that there's an upper bound on the change of entropy in a system, both from the work you put in and from the information we already have about the system. The way I explain this to people is let's say you're cleaning your room,

4:38right? You have to know where the dirty clothes are to then put in the work and put them in the right place, right? So to reduce a chaotic system to a less chaotic system, you have to have both. And I think modern LLMs really focus on how much work we can put in, right? Chain of thought, reasoning. We see that the more work you put in, you can

4:57reduce the entropy system. We've seen it. We know it's not linear, but it's true. But they're not necessarily focusing on the information piece of it all, right? And I'm going to give three examples, to try to illustrate how providing context, memory information, whatever you want to call it, into the system, particularly in the latent space, can make LLMs more energy

5:21or more intelligent, efficient. Okay? And to give some context, the current methods in the field to provide context are like RAG or memory, but we all know that these have problems, right? They don't scale that well. They're typically used for needle in a haystack tasks or semantic search tasks, and really struggle with multi-hop reasoning queries, right? Where multi-hop reasoning, if you don't know, is taking a bunch of facts, putting them together, and constructing a logical answer from

5:46that. They also both happen in token space, which is fundamentally constrained for the models, rather than a latent space, right? And I'm going to talk about three examples in the latent space that I believe are efficient ways to convey context to the underlying models and increase the intelligence efficiency. So this is going to be in-

6:09Multi-agent systems.

6:14My chalk is getting small. Okay. In the context window,

6:26and as inputs, right? So these are the three fields of research I've been looking at over the last few months, and I want to explain them. So let's start with multi-agent systems. So I'm going to give a model of multi-agent systems and then explain how it can be improved and the results we've seen from that improvement. So multi-agent systems, we can say, pretty abstractly, there's a supervisor agent, right? And then can spawn

6:55worker agents.

6:59Right? I'm pretty sure everyone

7:04knows about this kind of stuff, right? And they all complete a task, but let's say this is time down here.

7:12The problem with this, right, is it's extremely inefficient. The amount of tokens that this uses is really astronomical. And in the current economic state where token maxing is largely unsustainable, we need to fix this somehow, right? Additionally, the supervisor agent, as it's doing work, is generating context. It's exploring the system. We also see that the worker agents are all generating their own context itself,

7:39and this is not shared in the global state, right? A context a worker discovers here might have been discovered here, and now we're putting in extra work and becoming very inefficient, right? So, part of the research I did is, let's say we share context. We maintain a global state of context, which I did as a

8:01global KV cache that was shared among the agents. And we can, when we spawn a new agent, take the task that it's spawned with, filter out the information we need from the KV cache. We did this with a compression algorithm, essentially. And then spawn the worker with this initialized KV cache that essentially contains the context of the system that's important to the worker, right? When we judge this system versus the system without context sharing on a multitude of tasks, we saw, in total, there was a 42% to 57% reduction in worker agent tokens, and a total reduction of 21% to 31%

8:45total tokens with the same accuracy, right? So they become highly more efficient just by producing shared context, introducing information into the global system, right? Additionally, some work I also did is about in-context learning, right? We've all seen DeepSeeks release the sparse attention model, right? Or hopefully, we've all seen that. And we've seen that sparse attention has really taken the industry by storm. It makes sense, right? We need a more efficient way than dense

9:13attention. But the way I want to investigate sparse attention is modeling it as a reranker model, right? Fundamentally, if you think of sparse attention, it's looking at one token saying what information of the past tokens are necessary in order for me to generate the next token, right? And this is very similar to how reranker models work in RAG. And so I was wanting to say, okay, if this could model a reranker system, it would be evidence that more efficient model was partly, at least, because of accurate context retrieval over its own token space, or latent space, I guess. And so what we did is we took the queries and the

9:52keys generated by the Lightning indexer from the DeepSeek model, and we computed document scores over a question-answer dataset that was multi-hop to try to see if it would accurately get the right documents for a question. I'm just going to write the equations of how we did that. Feel free to come up to me after and ask about any of these because I'm going

10:12pretty quick. But essentially, we took the ReLU of the cost of the normed query over the queries. Basically, we have a dataset with a bunch of documents and multiple-choice questions over those documents. The queries are from the questions in the multiple-choice dataset, and we did this with the normed keys from the docs itself. And then we could compute a score for each doc that was just a sum of these cost scores times each head, and then we just took the mean over all tokens in the doc, right? And what we found is that using this pretty simple scoring method from the sparse attention mechanism, we actually match or

10:59beat state-of-the-art current reranker models, showing how this essentially mirrors a reranker and is essentially just context retrieval over the context window. So here, two minutes left, but I just want to reiterate that these are both examples of fundamental information introductions to the system, context retrieval, that make the full system much more efficient, right?

11:23Sparse attention is way more efficient than dense attention. Multi-agent systems, we showed that there was a huge reduction in average tokens used in order to get the same result. And the last thing I'm going to talk about is some work I'm doing with the SNAP Lab at Stanford to reduce the number of tokens we actually need in the input rather than the output,

11:45while maintaining state-of-the-art results. So we have this idea of when doing LLM systems, why do we have to put the entire doc in the context window when doing RAG? Instead, what if we just treated memory or context injection as a modality as we do text, as we do image, as we do video, where

12:08it's a full modality onto itself, and how would that work, right? And so what we did ... is we built a memory module, is what we called it, where we took docs in as well as a query. This is over a multi-hop reasoning dataset. We built an architecture that took these in, produced 16 latent representations,

12:35and then injected them directly into the frozen LLM. Here, we used Qwen 8B. And our theory was, okay, if we can inject 16 latency representations that tend to take the information in the docs from the query that is necessary, how will it perform? Directly into the frozen LLM. What we found is using the trivia QA dataset, we actually beat RAG 50, where we took the 50 most relevant docs to the query and injected them

13:05directly in the context window in the token space by a fair margin. So using this approach, we could inject the 16 latent space representations and get a 63% exact match over the, like I said, the trivia QA dataset, while RAG 50 got a 55% exact match. And fundamentally, this actually used 372 times less tokens representations in the input. So it's doing better. It's more intelligent, efficient, while using 372 less tokens. So it's incredibly more efficient just by introducing

13:42the correct context through this method in the latent space. Okay, so I hope that I've convinced you all that latent space representations of context and context retrieval is incredibly important in the LLM sense. I want to leave you with what I think would be the holy grail of a model that could introduce context accurately.

14:03I think it should be low latency, so it can be used at inference time. I think it should scale over large corpuses, and I think that there should be essentially zero switching costs to the base LLM, so that when the next model came out, you could introduce the most state-of-the-art reasoning model while still maintaining this context

14:21introduction into the global system. These are research topics that I'm exploring so that we can introduce them in the large AI tooling systems that we're using at Ramp, such as Inspect, where we're actually only generating more context than we ever have before. Thank you.