S2: Agent Parallelism Evidence Review¶

Date: 2026-08-20. Method: eight parallel research passes plus direct primary-source verification. Supersedes in part: the two mischaracterised claims of S1 Multi-Agent Architecture Decision, and nothing else on that page (see Correction to S1).

Why this page exists¶

S1 stated a -39% to -70% multi-agent effect and treated it as a general finding. It is not. The figure is real but scoped to sequential-planning tasks only, and the source paper's actual range is +80.8% to -70.0%. That distortion sat in the design documentation for four months and pushed the architecture toward single-agent defaults, a three-to-four agent coordination cap, and "organisational simulation fidelity" as a stated value proposition.

The lesson generalises beyond the one number: a wrong figure in a design document steers architecture silently and indefinitely. Every claim below therefore carries an explicit verification status, and unverified claims are marked rather than dropped.

Verification statuses used¶

Correction to S1¶

Action: S1's headline statistic needs correcting in place, the Nature publication added, and the reasoning-versus-work-stream distinction promoted from a buried caveat to the framing, because it is how the strongest source frames its own result.

The central distinction¶

The literature separates two regimes that S1 conflated.

Reasoning parallelism: many agents debating, voting on, or self-correcting one problem. Consistently negative. The Ringelmann-effect study (arXiv:2606.02646, 2026-05-31, R-squared above 0.99 across 44 conditions) found thirty densely debating agents produce no more answer diversity than one on MMLU-Hard.

Work-stream parallelism: many agents each doing a different, independent piece of decomposed work. Positively supported, including in software specifically.

This distinction is supported and tested, not merely asserted. It is the framing the Nature paper uses for its own headline result.

Findings by question¶

1. Verification as merge authority¶

Verdict: mechanical verification cannot be the sole merge authority. Empirically false at the scale the thesis needed.

Every shipping system keeps a human or an LLM judge as the final merge authority. None uses deterministic checks alone.

2. Judge reliability and independence¶

Verdict: a judge is a triage filter, not an authority. Structural independence is directionally right but modest, with a hard mathematical ceiling.

Design consequences. A cooperative second reviewer may be worse than none. Disagreement must be forced and evidence-grounded. Independence must be by model family, not merely by agent identity: binding every agent to one provider makes the claim nominal.

3. Coordination topology¶

Verdict: flat peer coordination collapses; hierarchy removes the term that causes it. This is the most strongly evidenced finding in the whole review.

Anthropic's counterweight: they state coding is not a good fit for their multi-agent pattern because planning, implementation and testing of one feature share too much context. Their own production research system is deliberately flat at depth 1.

4. Decomposition and the scale ceiling¶

Verdict: the binding constraint is decomposition quality, not agent supply. The ceiling is roughly 11 to 25 coherent independent units and it is method-independent.

The ceiling is corroborated three ways. Contract-first (15-25 files), graph-partitioned existing repositories (1-11 partitions), and design-then-contract (14-22 files) all land in the same range using different methods. It is not one benchmark's artifact.

5. Do contracts survive implementation?¶

Verdict: no. Renegotiation is not an exception path; it is assumed infrastructure everywhere it has been tried.

No paper reports a quantified renegotiation rate. Every source that touches the question points the same way:

• Contract-Coding's own headline example is its reviewer detecting a semantic mismatch mid-build and retroactively patching the contract, adding a dimension nobody specified up front.
• CodeTeam builds machine-checkable contracts (file ownership, interfaces, dependencies) and enforces them by fiat, then reports zero failure analysis of how often agents push against the constraint.
• Constraint Decay (arXiv:2605.06445, single agent): across 80 greenfield tasks under a fixed, pre-specified API contract, capable configurations lose roughly 30 points of assertion pass rate as structural requirements accumulate; weaker configurations approach zero. Status: UNVERIFIED.
• Co-Coder sidesteps rather than answers, avoiding interface conflicts by construction through hub-file isolation in the partitioner.

Every system shipping a contract-first approach also ships a repair loop, and none report the contract holding without one.

Planning assumption: a repair and renegotiation loop is a required component.

6. Recursive decomposition depth¶

Verdict: genuinely open. No safe depth is published, and the decisive experiment has never been run.

The gap, stated precisely: ARIES measured a real system collapsing at the merge but never added a gate. The Six Sigma model shows gating should fix arbitrary-depth compounding but was never run on a real decomposition benchmark. No paper connects them. Taking a decomposition benchmark, gating every aggregation, and reporting the new deterioration ratio is an experiment nobody has published.

7. Session durability and observability¶

Verdict: a genuine open gap, and five of the six properties are weeks of conventional engineering.

The six properties: persists; live-readable by another party while running; resumable by a different agent or process; forkable; steerable mid-flight; partial output survives death.

No system has all six (UNVERIFIED: a negative over a market, established by survey on 2026-08-20 rather than against any source that enumerates it, so read it as "none found", not as "none exists"). Closest partials: Claude Agent SDK (explicit fork, cross-process resume), Restate (the only shipped generic live journal UI), Letta (different clients attaching to one persistent agent), Zed (the only multi-party live read, via a CRDT-synced buffer), Devin (the strongest same-owner steering, plus Slack-thread following as a real multi-party channel).

Minimum viable order, by value per unit effort: durable and resumable first (event log plus periodic snapshot on existing persistence); live-readable second (near-zero marginal cost once the log exists); turn-boundary steering third; forking fourth; partial-output survival as a corollary. Scope live sandbox state out: it is a research project, and the reference system's own fallback is to reissue the interrupted command on resume.

8. Category status and adoption¶

Verdict: the org-simulation category did not survive as a product thesis. The current packaging cannot spread. The self-hosting moat is durable.

Adoption shape. Every project that spread had: one command on the machine the user is already on; payoff inside 60-120 seconds; a shareable artefact native to a medium the audience already trusts (a terminal cast or a chat screenshot, never a bespoke dashboard); peer-to-peer "I built this for myself" distribution; and single-player value needing no coordination. The failure pattern is multi-step setup before any payoff, autonomy claims the reliability cannot back, demos that do not survive reproduction by a stranger, and top-down adoption with no individual whose daily habit depends on the tool.

BUSL-1.1 is a downstream ceiling on redistribution and procurement, not a day-one friction point. n8n runs a non-OSI licence chosen from the start and reached $40M ARR and a $2.5B valuation, but on a multi-year compounding motion, not a spike.

9. Competitive position on the narrowed thesis¶

Five properties: dependency-graph partitioning; hierarchical fan-out with non-coordinating workers; recursive self-split; structurally independent judge; durable observable sessions.

No shipping product has more than two of the five, and none combines partitioning with judging (UNVERIFIED, on the same footing as the durability negative above: a survey result of 2026-08-20 over the products named below, not an enumeration of the market). It is deliberately not extended to unshipped systems, which a survey cannot reach. Cursor's internal system has neither: its splits are planner-owned slices rather than graph analysis, and the judge was removed. Closest shipped competitors are Devin (fan-out plus unusually strong observability), Factory.ai Missions (typed-role coordinator dispatch), and Warp Factories (closest on the judge axis, closed early access). Claude Code ships recursive subagents as a primitive but not as an orchestration policy. Amp shipped the opposite topology in July 2026: agents that spawn agents and message each other and exchange files, which is the flat shape Cursor measured collapsing.

Co-Coder's implementation is public at github.com/Flitternie/CoCoder with five stars and no commercial deployment. Dependency-graph partitioning is a validated technique lying in the open.

Principal risk: Cursor productising their planner/worker system. They run it internally at scale and have stated the intent. Adding partitioning and reinstating a judge is bounded engineering for them, not research.

What closed¶

1. Organisational simulation as an output-quality mechanism.
2. Governance and agent execution as a standalone product.
3. Mechanical verification as sole merge authority.
4. Flat massive parallelism.
5. A thousand agents, blocked by decomposition rather than by cost or machinery.
6. Contracts precise enough to make merges mechanical.
7. The current packaging as a route to adoption.

What remains open¶

One question: does verification at every merge hold off aggregation collapse as recursion deepens? If yes, the 11-to-25 ceiling is per level and scale is real. If no, it is global and this is a twenty-agent product.

Secondary and unexplored: no under-scoping detector exists in any published system; the classical modularity literature (Parnas, information hiding, coupling and cohesion, Conway's Law inverted) has never been connected to agent parallelism; nobody has measured whether decoupling a codebase first raises its partition count; and nobody has tested whether an agent that has read the code splits better than a planner who has not.

The experiment¶

Build in evals/recursion_depth/, on the pattern of evals/loop_ab/, calling the existing completion-oracle gate directly and bypassing charter intake, plan approval and wave dispatch (nine live rounds have died upstream of the interesting part).

New code required: make _do_decompose recurse and increment current_depth (already declared in DecompositionContext, read in six places, written in none); a flag to disable gating for the control arm; per-level instrumentation.

Output: one chart. Depth 1-6 on the x-axis, fraction of leaf work surviving to a correct merged result on the y-axis. Two lines: one gated, one ungated.

Design consequences for this codebase¶

1. Recursion is the priority. Every research thread independently identified decomposition as the binding constraint. That is issue #2699.
2. A cooperative reviewer may be worse than none. Disagreement must be forced and evidence-grounded.
3. Judge independence must be by model family, not agent identity. Binding every agent to one provider makes the claim nominal.
4. Role-based routing for execution is contraindicated; role separation for verification is supported. Anthropic's guidance is explicit.
5. A repair and renegotiation loop is a required component, not an exception path.
6. Stakes-stratified gating is the supported shape: auto-merge low-stakes small-diff work, escalate the rest, keep humans on sensitive paths.
7. The zero-artifact guard creates reward-hacking pressure. Explicit anti-exploit prompt wording is the cheapest measured countermeasure.
8. Session durability is weeks of work on existing persistence and event-bus infrastructure, excluding sandbox state.
9. Self-hosting is the durable moat because it cannibalises every competitor's monetisation surface.

Method notes and limitations¶

Eight parallel research passes plus direct verification of the two most load-bearing papers. The shared web-search budget was exhausted at 200 queries partway through, so later passes relied on direct fetches of arXiv and primary sources rather than search; grey-literature and blog coverage is consequently thinner than academic coverage.

Two claims could not be independently confirmed and are flagged above: the ARIES aggregation numbers (fetched directly by one pass under arXiv:2502.21208, not locatable by another) and Cursor's "right description of intent" statement.

Claims marked UNVERIFIED come from a single research pass without independent confirmation. Given that this review exists because of a four-month-old mischaracterisation, treat them as provisional and verify before any of them steers a decision.

Sources¶

Peer-reviewed or journal-published

• Kim et al., "Capable language models can outgrow the benefits of collaboration", Nature Machine Intelligence 8(7):1157-1172, 2026-07-24, DOI 10.1038/s42256-026-01268-y; preprint arXiv:2512.08296
• Cemri et al., "Why Do Multi-Agent LLM Systems Fail?" (MAST), NeurIPS 2025, arXiv:2503.13657
• OpenHands Software Agent SDK, MLSys 2026, arXiv:2511.03690

Preprints

arXiv:2606.00953 Co-Coder · arXiv:2604.13100 Contract-Coding · arXiv:2606.22082 CodeTeam · arXiv:2605.06445 Constraint Decay · arXiv:2608.16801 When Agents Coordinate · arXiv:2604.03551 AgenticFlict · arXiv:2605.21384 SpecBench · arXiv:2606.28430 Building to the Test · arXiv:2607.20852 Code Monitor Red Teaming · arXiv:2603.23448 c-CRAB · arXiv:2605.29800 Nine Judges, Two Effective Votes · arXiv:2607.13918 Partially Correlated Verifier Cascades · arXiv:2608.18167 Adversarial Review · arXiv:2604.19049 Refute-or-Promote · arXiv:2604.20200 Chasing the Public Score · arXiv:2606.07379 Do Coding Agents Deceive Us? · arXiv:2606.26300 The Verification Horizon · arXiv:2608.01715 Coding Agents as Test-Suite Auditors · arXiv:2502.21208 ARIES · arXiv:2505.02576 RDD · arXiv:2601.22290 The Six Sigma Agent · arXiv:2605.06639 RAO · arXiv:2603.28990 Drop the Hierarchy and Roles · arXiv:2606.02646 Ringelmann Effect · arXiv:2605.02168 Planner Matters · arXiv:2607.21909 Claim Plane · arXiv:2604.28138 Crab · arXiv:2605.22781 DeltaBox · arXiv:2608.00101 Copilot production-scale study · arXiv:2605.06717 Agentic Coding Needs Proactivity · arXiv:2606.31498 Governance Gaps in Agent Interoperability Protocols · arXiv:2509.03730 The Personality Illusion · arXiv:2606.27443 Personality Composition · arXiv:2607.05659 Personality and Emotion in Software Teams · arXiv:2603.27771 Multi-Agent Risks · arXiv:2603.26993 Reliability Limits · arXiv:2604.02460 Single-Agent Outperforms

Engineering writeups

• cursor.com/blog/scaling-agents (2026-01-14)
• cursor.com/blog/self-driving-codebases (2026-02-05)
• cursor.com/blog/agent-swarm-model-economics
• Anthropic engineering: building a C compiler; when to use multi-agent systems; the multi-agent research system
• claude-code CHANGELOG (subagent depth versioning, v2.1.172-219)
• OpenAI, why SWE-bench Verified was retired (Feb 2026)

Implementations

• github.com/Flitternie/CoCoder (five stars, no commercial deployment)