The integration queue

“To fix this, we created a system where a neutral third-party agent intervenes on merge conflicts and resolves them on behalf of all parties. Its only goal is to be impartial and efficient, similar to the way merge queues work in engineering teams.”

“The old run accumulated more than 70,000 conflicts before we paused it, accelerating rather than stabilizing, while the new run logged fewer than a thousand over its full four hours.”

“Two agents working on two unrelated screens will still collide on the last line of that file.”

“Among some teams at OpenAI, we saw the number of landed PRs increase by 500% in the first three weeks.”

“We don’t want thousands of loops being set up across the company without any bounds. You have a managed surface that you go to to set up the loop. So it runs on Sunday when we know we have better uh CI capacity available.”

“it’s going to stop at just creating a draft PR and it’s not going to push it to CI yet.”

“with the pull request it’s even worse now because they come in so quickly, that many of them cannot be merged anyways without manual resolution of the conflict. And there’s a there’s a lack of back pressure mechanism.”

“the act of merging um is starting to look a lot like um high-performance uh database problems where you have serialization and you have a single ledger where every single change needs to go in and you to lock the the database in order to be able to commit.”

“We invoke a dollar land skill and that coaches code X to push the PR, wait for human and agent reviewers, wait for CI to be green, fix the flakes if there are any, merge upstream if the PR comes into conflict, wait for everything to pass, put it in the merge queue, deal with flakes until it’s in main.”

“The merge one is terrible, and it's why Gas Town has a refinery to try to basically have a dedicated role that knows how hard merges are and just redo them, right?”

“with each engineer producing three to five PRs per day, even on a team of three, merge conflicts were super miserable, right? Because these PRs tended to be pretty big, we were working on the same parts of the code base.”

“In a system where agent throughput far exceeds human attention, corrections are cheap, and waiting is expensive.”

“like our CI melted we had to fix that”

“For example, 10 agents making 3 attempts each results in 30 executions, even before considering tool-call fan-out. If each attempt triggers multiple APIs or CI jobs, downstream work increases faster than the number of agent runs.”

“We’re talking about roughly a 10x increase in load on some systems. At most companies, that kind of growth might happen over two or three years. At OpenAI, we see it in about six months. That level of acceleration exposes bottlenecks everywhere”

“Opening a PR is not the finish line. A green CI run is.”

“And we really rapidly became bottlenecked on other things like our merge queue, which is the get in line to get your change accepted by the system that then deploys into production. We had to completely re-architect it because so much more code was being written and so many more pull requests were being submitted that it just completely blew out the expectations of it.”

“Green is not coherent. Ten agent PRs can each pass CI alone, pass again as a batch, and still be a mess together: three flavors of the same helper, two designs pulling in opposite directions, a thousand lines no human read. A merge queue keeps main buildable. It does not keep main good.”

“Since CI runs cost tokens, compute, and time, we only have at most two rounds of CI.”

“And it’s not an AI review problem or anything. It’s really just like a release problem, like managing the merge queues, humans getting access to the right set of data in the repository and things like that.”

“I think merge [queues] works for humans at a certain scale, but the merge [queues] could get quite deep. But then if you sort of 10X that, like, conservatively, I think 10X that, and then if you buy into, like, hype cycles and you 100 or 1,000X that, I think it gets completely untenable”

“There were a handful of people probably managing a security review process or a launch process or a deployment process or, you know, sometimes reviews were a little slow and they got backed up. Well, now we just threw gas on the fire”

“What’s going to happen is like there’s going to be more uh code being written which means that code reviews will become the bottleneck and then after code reviews uh integrations and deployment CI/CD will become the bottleneck.”

“If you were to fan out to a couple of hundred subagents and then tell those subagents to run the build and test of an application, what you’ll get is bad form back pressure. Thus, the instruction above is that only a single subagent should be used for validation”

“Retries restart from a moving HEAD, dependent members can miss predecessor source, independent outputs reach review without integration, and approval can detach from the published commit.”

“The merge tax is superlinear. Each resolved conflict changes the base, invalidating other branches’ assumptions.”

“parallel agents are bottlenecked not by their code quality but by the convergence points in the codebase.”

“as we are ready for the influx of PRs, we, in fact, had to, like, add internal code [red] for our CI system because year over year, the number of CI jobs also doubled or more than doubled.”

“So first thing that we did was we took a whole bunch of like simple CI jobs that used to run once you create the PR and we moved it to inner loop and we kept all of this behind one single command”

“every Sunday, when we don’t have that much CI load, it will generate a new PR for this service using that code cleanup.”

“The baseline can change so much during a swarm that the final workers getting merged are trying to merge against an unrecognizable new head.”

“Recurring merge conflicts are the canary telling you that there is a misalignment between a) the work you do, and b) the type of work your architecture supports.”

“This ensures that at most, the repository is 30 minutes out of date”

“Then integrate serially: merge one, re-run the full suite, merge the next. Branches that pass alone can fail together…”