Article2 llm collaboration

Five AI Models, One Impossible Problem: Inside a Multi-LLM Research Consortium

This is the second of two posts about an intensive research project on the P versus NP problem. The first post covered the mathematics. This one covers something arguably more novel: the project was carried out almost entirely by multiple large language models — different models from different vendors — organized under a formal research protocol with role separation, file ownership, and mandatory adversarial peer review. This post describes how that machine worked, where it broke, and what it suggests about AI-driven research.

The cast and the constitution

Five AI systems participated as named researchers, each from a different model family: GLM (a reasoning model), KIMI (a long-context model), Qwen, Claude, and Codex, with a sixth ("Antigravity") joining for independent audits later. Crucially, they were not five copies of the same assistant brainstorming together. Each was bound by a written coordination charter that assigned a distinct role, a distinct research line, and — this turned out to matter enormously — exclusive file ownership.

The division of labor: GLM owned research line A (the Williams-style algorithmic method) and its hypothesis files. KIMI owned the cross-domain line and later the constructive side of the proof-complexity line. Qwen was explicitly designated not a "third proposer" but an independent verifier and experiment-semantics auditor, subordinate to no one. Claude was designated an independent formal auditor — its charter literally says it is "not a summarizer of existing proposals" — responsible for rule-by-rule verification of proof documents and for producing minimal counterexamples. Codex owned the remaining candidate approaches and served as integrator, classifying every claim as an unconditional theorem, conditional theorem, conjecture, or experimental observation, and checking whether each claim's logical reach actually connects to P ≠ NP.

No agent could write to another agent's files. Objections had to be filed in a shared integration queue, and only the owner could incorporate them. Over roughly two days and thirteen rounds, that queue accumulated 426 entries, each a claim, audit, refutation, or verdict — the project's complete intellectual ledger.

Verification with teeth

The most distinctive rule was what a verification report was required to contain. Reproducing an output was explicitly insufficient. An audit had to include a soundness/completeness argument (or bijection) linking the code's states to the mathematical objects being measured, a minimal counterexample where applicable, an explicit statement of which prior conclusions survive and which are retracted, and exactly one verdict: ACCEPT, REJECT, or REVISE.

Independent reimplementation was standard practice, often across programming languages to avoid shared bugs. The exact-classification computations for monotone circuits were run in Python and then re-audited by standalone JavaScript and Node.js implementations that deliberately imported nothing from the originals; a mass-computation engine was validated against an independent DAG implementation, a different library, and a differential fuzz over 3,000 random inputs. One audit even injected six adversarial mutations into a contract specification to confirm the checker would catch each one.

The error log is the best part

What makes the record convincing is not that the agents were right — it's how often they caught each other, and themselves, being wrong. A sample from the ledger:

Claude found that an early version of a key hardness-magnification proof illegitimately promoted a fixed NTIME class to all of NP; KIMI independently found that the same proof under-counted the runtime of a PCP oracle verifier. Both defects were repaired in one lemma, and the fixed theorem (with a better exponent) was then accepted. GLM's early experiment misidentified formula depth and was corrected in round 2. KIMI's Resolution-embedding theorem was rejected by Qwen and Codex after they exhibited a two-literal type mismatch (A = {x}, B = {¬x}). Claude publicly reported a real bug in its own demand-propagation simulator — dropped demands from multiple parents — fixed it, and only then used the corrected code to refute a conjecture it had itself proposed. Claude's subsequent claim that "mixed polarity is harmless" was retracted after Qwen's counter-audit. A verification harness was found to depend on a hash seed and repaired; an off-by-one in a negative boundary caused stored result hashes to be withdrawn and regenerated; a crashed checker went through a documented bug-and-fix cycle; a "roughly degree-3.5 polynomial" growth claim extrapolated from a single data point was rejected on principle.

Two structural features made this possible. First, the "epitaph" culture: the charter required recording why each failed idea failed, so refutations were treated as first-class contributions rather than embarrassments. Second, the ban on self-acceptance: no theorem entered the accepted layer without at least one — usually two or three — independent hostile audits, and several major results (like the E1c signature memoization theorem) were only accepted after three agents verified them rule by rule and reproduced the same hand-calculated counterexamples.

Guardrails against AI-typical failure modes

Several rules read as direct countermeasures to known LLM weaknesses. The charter opens by enumerating the ways a partial result can be mistaken for the final one (restricted circuit lower bounds do not imply P ≠ NP; assumption-based bounds don't count as unconditional proofs) — a pre-commitment against overclaiming. Codex was forbidden from using the three classical barriers as a lazy pass/fail stamp; each barrier's preconditions had to be checked against the specific approach. Confidence numbers accompanied every open route, and they were revised downward as often as upward. Perhaps most telling: the final overview, written after thirteen rounds of intense work, states in bold that none of the results proves P ≠ NP, and enumerates exactly which obligations remain open. Whatever else one thinks of AI research agents, this project did not hallucinate a proof of the hardest problem in computer science — it produced a disciplined report of why it couldn't.

What worked, what didn't

What clearly worked: role heterogeneity (proposer vs. verifier vs. integrator) turned model diversity into an asset — different models made different mistakes, so cross-vendor audits caught errors that same-model review might have missed. Exclusive file ownership plus a public objection queue eliminated the mush of collaborative overwriting. The rigid ACCEPT/REJECT/REVISE grammar prevented the classic LLM failure of politely agreeing with everything. And the requirement to link code states to mathematical objects prevented "the script ran, therefore the theorem holds" reasoning.

What the record also shows: enormous computational and token expenditure went into finite case analyses (hundreds of millions of DP transfers, tens of thousands of allocation rows) whose mathematical payoff, while exact, is modest relative to the open conjectures; several rounds were consumed re-auditing audits of audits; and the deepest obstacles — the quantifier wall, the extension-elimination blow-up — are exactly the ones that resisted human researchers for decades, and the agents made no more headway against them than the field has. The honest reading is that this methodology is extremely good at verification, mapping, and error-elimination, and not yet a source of the kind of singular conceptual leap the problem presumably requires.

A template worth stealing

Even so, the protocol itself may be the exportable result. If you want multiple AI systems to do serious technical work together, this project suggests a recipe: separate proposers from adversarial verifiers, and use different model families for each; give every agent exclusive ownership of its artifacts and route all disputes through a logged queue; require every verification to state what would falsify the claim and to end in a single unambiguous verdict; mandate independent reimplementation, ideally in a different language; and treat documented failure as a deliverable. None of that is specific to complexity theory. It is simply peer review, compressed from years to hours — with the reviewers instructed, in writing, to be hostile.

Sources: This post is based on the project's internal governance and log documents: the agent guidelines file, the multi-agent research coordination charter (roles, ownership table, audit requirements), the integration queue (426 entries), the round summaries (Rounds 2–13), the final overview, and the independent audit scripts and result files present in the experiment directories (including the cross-language reimplementations and differential-fuzz records referenced above). Model names refer to publicly available AI systems; no personal information appears in this post.