Proving it's safe

“The step-based approach also made it simple to run migrations on hundreds of files concurrently”

“Throughput has scaled exponentially. Human review capacity hasn’t. The gap is widening.”

“If your engineer is able to manage five simple tasks, two manageable tasks, and one complex task at a time, your two engineer code review system now requires them to review ten simple tasks (theirs and others), four manageable tasks, and two complex tasks.”

“Each PR is evaluated for its rollout risk: low, medium, high. 33% of our PRs are low-risk and are auto-approved by the bot. The author of the PR can thus choose to merge the PR, which in our case reduced PR lead time by 20-40% (when compared with all PRs).”

“This year, an automated dependency upgrade passed our checks, but still failed in production, impacting end users. We are responding by strengthening safeguards, expanding rollback capacity, and scheduling automated changes during owning teams’ working hours.”

“the volume of change increased faster than some of our verification controls could adapt.”

“some agents were a bit too “ambitious”, trying to solve problems that weren’t strictly in their prompt, like refactoring code or disabling flaky tests.”

“The background agent produces a PR that passes CI but is functionally incorrect. This is the most serious error, as it erodes trust in our automation. When performing changes over thousands of components, these changes are hard to spot in reviews.”

“Engineers are now tripling, quadrupling the number of PRs that they’re producing, but the PRs are stuck waiting for code reviews, right?”

“it’s going to stop at just creating a draft PR and it’s not going to push it to CI yet.”

“So every pull request at Anthropic is code reviewed by [Claude] code. And that actually catches maybe like 80% of bugs, something like this. And it’s the first round of kind of code review.”

“now you have your backlog in your pull requests. And that’s the bottleneck now because you can’t review all of that. And then there are going to be agents that review all that code and auto merge and auto deploy it, and what you get is the Homer Simpson car.”

“we were blocked on merging the PR we needed someone else to um to review it for us and maybe they were in a different time zone perhaps. So what we started doing was building this agentic code owner thing”

“any sort of scenario where you want that AI agent to kind of validate what it’s done, you’re essentially talking about ephemeral environments, even if it’s not exposed to people”

“And also coordination debts. This is when you get really hairy merge conflicts because agents will touch in the same files or developers even doing duplicated work because they both picked up a thing and tried to finish it in one day.”

“Agents self-report "success" without mandatory verification”

“I think we found that the state-of-the-art in December was you could probably run this for about two weeks. By the end of those two weeks, you’d find that, hey, you want to like change a color of a button. Well, turns out this button is implemented in like 10 different places”

“AI-generated code needs to be reviewed, which means the natural bottleneck on all of this is how fast I can review the results.”

“We invoke a dollar land skill and that coaches code X to push the PR, wait for human and agent reviewers, wait for CI to be green, fix the flakes if there are any, merge upstream if the PR comes into conflict, wait for everything to pass, put it in the merge queue, deal with flakes until it’s in main.”

“with each engineer producing three to five PRs per day, even on a team of three, merge conflicts were super miserable, right? Because these PRs tended to be pretty big, we were working on the same parts of the code base.”

“In a system where agent throughput far exceeds human attention, corrections are cheap, and waiting is expensive.”

“As code throughput increased, our bottleneck became human QA capacity.”

“The honest summary is that I made my own output cheaper partly by making my teammates’ review more expensive.”

“And while developers can now generate code and PRs faster than ever, we still only have a fixed number of senior engineers able to review them.”

“Our team averages 10.8 PRs per engineer per week, 2.5x the 90th percentile according to DX benchmarks. But every one of those PRs still needed a human to look at it, and the review queue became the constraint.”

“like our CI melted we had to fix that”

“As we ship more code with agents, the bottleneck shifted from writing code to reviewing it.”

“The continuous availability of code reviewers is also not guaranteed all the time, so we need to only migrate parts of the codebase where we have bandwidth to land the changes.”

“We needed to slow down some of the migration work to avoid overwhelming the teams that had to do the reviews”

“Completion is fenced too: Beads accepts a receipt only when the work ID, generation, claim token, and Workflow identity still match.”

“Obliterate like about 40% of that. It was just scorched it, right? All the fences. We only kept like 14 out of like 100 plus fences.”

“A redirect only works if every consumer of the config layer honors it, and the ones that don’t fail silently, because a working connection to the wrong database looks exactly like a working connection. We shipped two consumers that quietly ignored ours.”

“Opening a PR is not the finish line. A green CI run is.”

“Across all four, the same patterns showed up: Slack as the universal entry point, pre-warmed ephemeral sandboxes, async fire-and-forget that produces PRs, human review as the merge gate, directory-scoped context files (CLAUDE.md / AGENTS.md), bounded CI retries, and MCP as the tool-integration standard.”

“Green is not coherent. Ten agent PRs can each pass CI alone, pass again as a batch, and still be a mess together: three flavors of the same helper, two designs pulling in opposite directions, a thousand lines no human read. A merge queue keeps main buildable. It does not keep main good.”

“Since CI runs cost tokens, compute, and time, we only have at most two rounds of CI.”

“There were a handful of people probably managing a security review process or a launch process or a deployment process or, you know, sometimes reviews were a little slow and they got backed up. Well, now we just threw gas on the fire”

“What’s going to happen is like there’s going to be more uh code being written which means that code reviews will become the bottleneck and then after code reviews uh integrations and deployment CI/CD will become the bottleneck.”

“A convoy of 5 implementation beads produces 5 separate runs, each walking the full formula: implement, review, check architecture, verify coverage. That means 5 separate reviews that each see only their own fragment”

“We want to get rid of the 20% of our codebase that looks and behaves differently to the rest of our codebase. If we do that, our code becomes extremely predictable. It becomes much easier to write our prompts, and also to review our code.”

“We were so happy to announce that we had 1,000 merged PRs in 3 months. We were over the top. Today, we do this in 10 days, and this number keeps going up. What is the result of this? It’s a bunch of PRs to review. This is becoming the new bottleneck.”

“Retries restart from a moving HEAD, dependent members can miss predecessor source, independent outputs reach review without integration, and approval can detach from the published commit.”

“Ruflo is the coordination ledger and policy decision point. Claude Code executes code, tests, commands, and file changes. A Ruflo coordination call records work; it does not perform the implementation.”

“Zero phases completed. Zero output.”

“But the wire connecting agent_spawn → LLM provider → task execution is missing. This isn't a bug in one tool — it's a structural gap across the entire orchestration layer.”

“calculateRequiredVotes() derives the quorum threshold from state.workers.length, so a single caller could cross any strategy's threshold (raft/bft/quorum alike) by voting under fabricated identities — a Sybil attack on consensus.”

“Amazon has has made this change uh that it now requires a senior engineer to review any AI generated change because they realize the junior engineers will just say looks good to me and it causes an issue.”

“The state-tracking fix from this issue was correctly applied — task_assign now transitions tasks to in_progress. However, agents never execute the assigned work.”

“When 10 of them finished simultaneously, I had 10 results to evaluate without enough context to know which ones I could trust. AI agents don’t have a speed limit, but the person managing them still does.”

“we put a lot of work into the to smart assignments with code inbox so that we try to find the most relevant person to review the code both from a ownership and compliance perspective, but also the history of how that person was working, uh their time zone availability, their calendar availability”

“So first thing that we did was we took a whole bunch of like simple CI jobs that used to run once you create the PR and we moved it to inner loop and we kept all of this behind one single command”

“Reviewers are overloaded with the increasing volume of code from AI-assisted code development, and have limited time to identify subtle bugs, security issues, or consistently enforce best practices.”

“About a month ago, we found ourselves with 60 open PRs for a team of five. They had been accumulating for a few weeks and we all had the sudden realization we were looking at two days of just code review.”