Nool coordination benchmark: Nool vs git worktrees at 10–45 agents
nool.dev/research/coordination-benchmark · §5.6 Implementation Audit; §5.7 Imperfect Intent Robustness; "What this does not establish"
Nool's own pre-registered benchmark, with an interactive data explorer, comparing it to git when 10–45 agents edit one repo at once. Its harness and raw data are open.
A rare vendor benchmark that undercuts its own headline. Unscheduled git worktrees lost 23 and 40 of 60 tickets at 35 agents. But git given the same file-footprint locks ties Nool when the plan is accurate, so the gain comes from claiming files before work starts, which any team can build. Nool's edge shows only when plans are wrong: a 0 to +6.7 pt gap at 20 agents as footprint noise rises, from n=2 runs per level.
What carries beyond Nool:
- Locking files isn't enough. In one run with a perfect plan, every ticket passed its own tests while the shared suite was broken. Check the whole system's health, not just how many tickets were accepted.
- Catching conflicts at merge time wastes money. Starting every agent blind and resolving at merge wasted 37–41% of spend. Claiming files up front kept waste near $0.
- Checks cost speed. With no checks, Nool beats git worktrees (718.5 vs 635.4 tasks/hr). Full validation only ties git, and custom policy rules trail it by 51.8 tasks/hr.
Watch the overhead figure. The page quotes 11–79× git's commit time, against a bare commit. The repo README puts it at about 6× (11× at 100 files) against git with a realistic pre-commit hook, and says those hooked figures came from a loaded machine and need a re-run. Raw data is in the repo: the findings report, plus results/trackc/fleet_runs.jsonl and results/micro/.
- Coordinate
- Verify
- Integrate
- Plan
- Remember