Claude Codes New REVIEW.md System, What is it?

0:00So, Claude Code now writes about 90% of its own code, and things are really speeding up at Anthropic. If you're looking at the web app at the moment, the rate of change is really fast. So, something is working. Between June and September, merged pull requests, that's work being done, more than doubled. But, the key thing here is the code that was being reverted or had previously been going wrong dropped by 40%. So, what the heck is going on at Anthropic? Well, what I've tried to do is piece it together from several different sources over the last couple of days, so you can learn how to move at this pace. And all

0:32the while, I'm doing my best to separate the marketing hype from the actual practical gains that I'm seeing other companies I've worked with apply in this space. So, anyone who's been working with agents realize there's no problem running a hundred different agents on your codebase. The big problem is you get a ton of code showing up, and it has

0:49to get reviewed and checked in some way. Most of the organizations I talk to, their biggest issue is the code review. So, the Claude Code team just published how they review code now, and this is their key line. We didn't remove review, but we did move most of the reading to machines optimized for human time and attention, and invested heavily in making it fast and easy to undo mistakes. AKA, the blast radius and the recoverability from that blast. So, I'm Rob from Switch Dimension. I'm going to walk you through Anthropic's process layer by layer, and then I'm going to show you where Claude Code's review product fits in. And it isn't actually

1:24where you'd assume. And then I'm going to finish with a quick look at Seon. It's their replacement for Git, which fits into this whole process. So, if you're a practicing developer, I would definitely hang around. We'll also talk about this one pull request that Anthropic had that went through 50 rounds of AI review overnight. Is that overkill? Let's take a look. Now, let's bear in mind that Anthropic have a ton of inference to throw at all of these problems. They might not fit exactly how you like to build, but it is giving us indications of where things might be going. And let's also bear in mind that

1:53Anthropic are, of course, selling inference and selling a big dream and vision of where this is all going. So, keep that in the back of your mind as we look through their process. That's not to say there's some gold in there for us developers to take away. So, across Anthropic's mono repo changes landing per day went from something like 2,500

2:11in May to more than 30,000 in September. That is a big leap. Claude Code has its own separate repo and that is under the same kind of pressure. So, you can imagine at that speed manual line by line review isn't going to happen. In their words, it's incredibly difficult if not impossible. So, the question they're asking

2:29themselves is what is code review for? Well, it does two jobs. One is catch problems before they ship and then two is spread understanding of the code base across the team. Because when you're reviewing other people's code historically, you're getting a sense for what's going on and how other people are working. So, the guys went back to first principles on this one and with that the question kind of changed. So, instead of what it used to be, did a human review this code, they realized that was always a proxy for the real question was what could go wrong with this code that's just been submitted? And then with that,

3:02which check is it going to be automated or human is the right one to catch it? And obviously, it's great if we could reduce the bottleneck of humans as much as we can. Now, that really sounds like developers being replaced in the review process. I don't think it is. It's about moving us as developers to a better point in the process. Well, at least I

3:20hope it does or else I'm out of a job. The other thing that's changed is where review happens. It's at a different point now. It used to be one check we did at the end just before a merge. But now, because of agents, it can run all the way from the first prompt to the merge, constant checks as we move. So,

3:37it's more like spell check as you write. So, next thing they did was they set a bold goal. So, most PR's in Claude Code repos should merge with at most one human in the loop. And that one human can be the author. Now, that's a change because usually review happened by somebody else. It was a second pair of eyes. So, they track human approvals by pull requests and also the comments that go with those pull requests and then how long that pull request waits on a person before it gets pushed through. These are the new metrics that are coming into play that might supplement or even

4:08replace traditional Dora metrics. And so, the guardrail that they're applying here is the reversion and the incident rates because if those go up, we haven't saved human time, we've only moved it somewhere else. So, basically, if we're getting agents to do all of the coding and we're getting these major amounts of commits and pull requests and merges, that's fantastic, but are we seeing that the customer is really peeved off at the end? Are features breaking? Are things falling over more and more? Because that means we're just fixing up the problems later on in the process. That's what we

4:38want to avoid. So, they sum up the whole system as six layers of checks with rollbacks underneath them. I've been talking about rollbacks for a while. The great idea here is that you can let agents run loose with their level of constraint, but if something does go wrong, you have a rollback to pull them back before it actually goes live or it has any impact on the customer. So, few problems get through all six layers and then you've got rollbacks to handle the rest if they do get through. So, layer one is verification before anyone looks. It's all that checking that we do, the browser tests, etc. So, most review now

5:12happens before that pull request actually exists. So, Claude gets something it can run, it can run tests, it can build, take screenshots, do some integration testing. So, instead of it just saying, "Yeah, it's done." it actually has to prove it's done and show the output. So, then a sub agent jumps in and reviews the diff with a completely fresh context window. Bar Shirni, we all know him, he's the creator of plug code. I've got several videos on him in the backlog if you want to take a look. He estimates that verification makes the final result two

5:41to three times better. So, when the same finding keeps coming up, they fix it at source. So, Claude at MD or agents.md is a short file that describes the project stack, conventions, and goals, and code owners write review.md, which tells the reviewer what matters in their code. You'll see in a lot of code review tools, they can do a lot of nitpicking, kind of these false positives of things that are probably a bit off, but are not really affecting the code. It's important that you put a cap on those so you're not overwhelmed by these small changes. The other problem with nitpicking is it can actually call out

6:13issues with previous generations of code that were just pulled into the PR that are not actually part of the change that's being made. This is something I've seen in my own production workflows and have limited the number of nitpicking because of this. So, layer two is an AI review on every PR before a person looks. It reads through the diff or the difference against the whole repository and then what the pull request stated intent is. So, this reviewer is checking each finding before a human even sees it. Several agents will hunt for bugs in parallel, then a separate group votes on each one of those findings. Then the reviewer only

6:48posts a finding if enough of them agree it's real. Okay, so layer three is a separate security review. So, on Anthropic's own repos, a pull request can't merge while any finding from the bug review or the security review is still open. So, that's a hard gate there and we'll come back to that rule. Okay, so layer four covers the boring checks that still block a merge. So, this is lint, type checking, testing on every platform, and then of course a build test. The pull request template also asks how the author tested the change and then how to undo it. Okay, quick word from our sponsor because this one

7:23ties directly into what we're building today. Specifically, it's about controlling the blast radius of your agents. So, most of you know OpenRouter as one API that gets you hundreds of different models. You can swap between them, compare them, it's all one bill, and I use it constantly. But they just launched something I'm genuinely excited about. It's called Armor and it's in beta. So, here's the problem. Once your agent can call a tool, it can actually do things, run commands, hit APIs, touch your files, and any hook or regex filter that you set up just won't catch these bad calls reliably. So, armor sits in between. Before your agent runs a tool

7:57call, armor checks it against your policy using Type safe's Jev model. Remember those guys? And then it can block anything that's out of bounds. Open router says it beats Claude code's auto classifier at a fraction of the cost. And that's thanks to the system one model. So, you get safer agents without slowing them down or building all these guardrails yourself. Check out Open Router and Armor at the link in the description down below. Okay, layer five is human approval scaled by risk. And this is where I think it gets interesting. Low-risk changes like docs, copy, or additive changes behind a flag, they don't need any human approval once

8:32the automated and security checks are green. Ordinary product work needs one person. And then high-risk code needs two. You might remember some of my past videos on some of Addy Osmani's work. He was originally from Google and now works on Anthropic. He talks a lot about this level of blast radius and how many approvers should be applied to levels of criticality. Basically, if it's going to make a big mess if it goes wrong, have more people check it. So, there are five rules in this system. The first one is the change that's being made sets the tier, not the author. A classifier reads the change and then picks the tier. A

9:05second classifier checks whether it needs a person anyway, and any kind of changes to a CI workflow file always will. Second rule, only trusted authors, and they're mostly from the owning team, get the lighter paths. Third, an agent's approval never counts as a person's. Automated review can clear a low-risk change. It's never going to fill the human slot. Nice to see we're still required as part of the process, at

9:30least for the next little while anyway. Okay, the fourth rule, whoever prompted the change is on the hook. This is the responsibility level. On an agent-written PR, the person directing the changes counts as the reviewer. So, there is always accountability and someone to point the finger at. So, basically, whether you're running hundreds of agents or not, you as a person for orchestrating this system is

9:52responsible for the pull request. Essentially, if an agent does a bad job, it's your problem. It was either your software factory or your level of review that caused this. The fifth rule, an approval covers one specific commit. If an agent pushes a new commit, someone has to approve again. So, layer six is the merge queue. It reruns every check against the main branch with the changes applied. So, basically, if you've got two PRs that are happening at different times and they pass through together, they can't combine in a way that's going to break the main. This is a big deal when you're running with lots of

10:27different agents running lots of different changes at the same time. After that, new behavior ships behind a flag. First of all, it goes to the internal team at Anthropic to have a play with it and see if anything's broken. Then, it goes to a latest channel where wider group of people and customers, maybe about a thousand or more, test it and you'll quickly figure out if something is broken or not and then you can roll back the change. If all of those changes go okay, then you move it to the stable channel. So, the backbone of these six layers is they rely on fast detection when things go

10:56wrong and fast rollback. Flags will switch off without a deploy. There's weekly on-call rotation and then every incident review asks, "Which layer should we have caught the problem on?" This is kind of like a retrospective, so we can keep improving the workflow and the system. So, as usual in any of Anthropic's writings, they also admit practically where the gaps are. So, they're saying in this case they don't have what they would call a circuit breaker. So, essentially, if you're watching your system and then you're seeing that lots of the code the agents are doing is beginning to be reverted, you want to crack down on your system

11:29pretty quick. That's what they're calling a circuit breaker. This is like a typical factory. If you have a bottling process for Coca-Cola and there's a defect on level two of the entire process, you kind of want the whole system to shut down or slow down so it doesn't affect a huge batch of work later on. In Anthropic's words, the argument is that slowing down lowers your error rate, mostly because you're doing less of everything. They'd rather get faster at noticing and fixing problems than have a ton of problems show up. Now, here's the interesting story I promised. One Claude code CLI pull request went through more than 50

12:01rounds of AI review and fixes overnight before its author stopped it. So, all those levels of review we're talking about just went round and round and round because nobody was watching it overnight. So, when they ran the git blame on what the reviewer kept flagging, using git blame you can basically do a root cause and find out what the issue was. So, about 59% of these issues that kept showing up that they tried to fix over and over was code that had been there since the first pass. It wasn't necessarily finding new bugs. It was finding a few more old ones every single round. Because what we've

12:34done here is created a system that checks and checks and checks. If it's trying to be proactive, if it's trying to do a good job, it's probably going to keep finding those issues. And we get caught into these silly little loops. So, it really is about knowing how much and how little. And that really is the trap with agents. A person can wave off a nitpick to their code or a preferential change in a couple of seconds where an agent, being proactive, tries to fix it, which means another

12:57commit, another review, another round. And this is why software factories are a moving system. And I've talked about this loads. Developers are afraid of where they stand in all this. You are now designing and building and curating a moving and growing system that's only going to get more ambitious as time goes on. We're going to be able to do amazing things with these software factories if you start learning them and start hanging around and get invested. So, in this case, Anthropic's fix was a number of review round caps. After three rounds, the reviewer asked the author to step back and rethink the whole approach. Are we going a little bit

13:30overboard here? On later passes, the reviewer only re-reviews the exact diff, flags the entire class of a bug instead of just that one line, because having the whole class together means we're attacking the whole thing, it's not going to surface later on, and then it never repeats the finding. And here's where agent evals come in, or evaluations. They turned past incidents into an eval. So, how they do this is they took real pull requests that

13:53actually caused incidents in the flow. They had engineers who know that code confirm them, and now run the reviewer against that set every day. Is it improving or is it getting worse? This is their benchmark. So, where does the cloud code review product fit in? It just went generally available, at least at the enterprise level, hopefully coming soon to everyone else, and it's their public version of their layer two, the AI review on every pull request. So, agents search the diff in parallel, and then a verification step filters out false positives. This happens quite a lot if you've used any other kind of

14:26code review tools, you know what I mean. Then it ranks the findings by severity and posts inline comments and a check run. So, Anthropic are claiming with this that most reviews are now about three times faster, and that's from roughly about 20 minutes previously down to about six or 10. You can pick quick, standard, or deep per repository, and

14:44then of course the deeper costs more. So, remember we talked about that layer three rule, the product never approves or blocks a PR, and that matches their own internal rules that the agent's approval never counts as a person. Locally in cloud code or cursor or anything else like that, the code review will give you a lighter pass before you do a PR, and then security review covers the security layer. As a developer, personally what I would do is just study how they're doing this, and then build my own version or my own agent to do exactly the same thing based on how I or my company likes to work. Okay, so one

15:15last quick topic, something that might not affect everyone now but may affect us in the future. Anthropic went and replaced Git on its mono repo with an in-house system called Seon plus a review tool they're calling antique. So, this is a unique problem that Anthropic had. The main branch has almost 100 GB of history and their CI checks this out 25 million times a day. A team of seven people have been building this at Anthropic and they finished the migration in about 5 months, which gives you an idea of the scale of this project. Nobody outside Anthropic has tested this as far as I know, but they

15:50report a footprint of 200 times smaller than their old Git mirroring. So, this is really interesting stuff, but what can a normal team copy right now? So, number one, give your agent checks it can run. Add a review.md. Put an AI reviewer on one repo and tune it for a couple of weeks. Set risk tiers by the change, not by the author. So, how big of a blast radius does this have? Never let an agent's approval replace a human's and measure the amount of reverts you're getting or the amount of

16:20things that are going wrong inevitably. So, this is a subjective one. Tell me in the comments, would you let a low-risk pull request merge without any approval by you as a human if every check in your factory workflow was green? And that really depends on the type of code base that you do work in or what the blast radius is. So, let me know yes or no and then why. And I really want to know from the practicing developers here. I'm not

16:45talking about your vibe coded projects. If you want to learn how to build with agents like this, that's what the switchdimension.com accelerator is for. Go check it out, the link is in the description. If this breakdown was useful, subscribe because there's a lot more like this coming. And if you're into the software development life cycle that Anthropic describes, check out this video from a couple of weeks ago about

17:05intent.