🤝 The study group that beat the answer key

A 5-minute read.

We spent two evenings with a paper whose central idea is one every student knows: when there's no answer key, swap papers with the person next to you. The surprising part is how far that gets you — and why it works better when the person next to you is not like you.

📄 The Paper: the study group that grades its own homework

The paper is "Co-RL: Unsupervised Reasoning Emerges from Diverse Cohort in Multi-agent RL" (arXiv, Aug 18), from a group spanning UC San Diego, Johns Hopkins and the University of Exeter. The question it asks is a practical one that's getting sharper every month: can you keep training a model to reason better after you've run out of problems with known answers?

Because the answer key is the bottleneck. The big reasoning gains of the last two years came from reinforcement learning against verifiable rewards — give the model a maths problem, check its answer against the true one, nudge the weights toward whatever scored. That needs a human (or a program) who already knows the answer. As models push into questions that are slow, expensive or simply impossible for people to check, that supply of graded problems dries up. So labs have tried self-rewarding: let the model sample many answers, take the majority as the truth, and train toward agreement with itself. It works for a while, and then it fails in a predictable way — the model's answers converge, get shorter, and eventually the training falls apart. Marking your own homework makes you more confidently wrong, not more right.

Co-RL's move is to turn the lonely student into a study group. Two or more completely separate models — sharing no weights, trained side by side — each solve the same problems. But a model never grades itself: it takes the majority answer from its peer's attempts as the provisional answer key, scores its own attempts against that, and updates. The peer does the same in reverse. Nobody in the room knows the real answer; each is trusting the other's consensus.

Why should that be any better than trusting yourself? Because the failure mode of self-grading is correlated error: when you get something wrong, you tend to get it wrong the same way every time, so your own majority vote just confirms the mistake. A different model has different blind spots. Where model A is systematically wrong, model B is often right, and vice versa — so each supplies corrections the other could never generate on its own. The authors then push this logic as far as it goes: the more different the cohort, the better. Pairing two copies of the same model helps a little; pairing two model families (a Qwen with a Llama) helps more; and the best configuration also hands each model a differently-worded version of every problem — the same maths, recast into a different story — so their mistakes decorrelate even further.

The headline result is the one that made us re-read the table. On seven text benchmarks spanning maths, code and science, the 3-billion-parameter Qwen model went from 40.7% to 49.3% average with no ground-truth labels at all. The same model trained the conventional way, with the true answers, reached 47.4%. The study group beat the answer key. We're still chewing on what that means, and the authors are appropriately careful not to overclaim it — but a label-free method matching or edging out a supervised one is not something we expected to write this week.

What it might change: if reasoning can keep improving from peer consensus, the ceiling on self-improvement stops being "problems humans have already solved." The catch, which the paper is honest about, is that you've replaced one dependency (labels) with another (diversity) — and nobody yet knows how many peers, how different, wired together how, is enough.

Go deeper ▾

• Mechanism: each agent rolls out K completions per prompt; the pseudo-label for agent A is the majority answer among agent B's completions (and vice versa). Reward is binary: 1 if A's extracted answer matches the peer pseudo-label, 0 otherwise. Agents share no parameters.
• Cohorts: text — Qwen2.5-3B with Llama-3.2-3B-Instruct; Qwen2.5-7B with Llama-3.1-8B-Instruct; a three-agent variant adds Qwen3-1.7B. Vision-language — Qwen2.5-VL, InternVL3.5 and Gemma-3 paired at comparable sizes.
• Data: text agents train on MATH levels 3–5 only; the "+" variant rewrites each problem with DeepSeek-V3 so paired agents see different surface forms of the same question. Benchmarks: GSM8K, MATH-500, AMC, HumanEval, MBPP, LiveCodeBench, GPQA. Note the transfer: trained on maths, gains show up on code and science.
• Numbers: average text gains of 3.0–8.6% across the configurations; multimodal gains of 2.3–7.2% (Qwen2.5-VL-3B: 37.24% → 43.89%). Under the CoMAS evaluation protocol at 7B, Co-RL reaches 62.97% versus CoMAS's 58.94%. Self-rewarding TTRL degenerates under prolonged training (reward collapse, length degeneration); Co-RL's reward signal stays stable in the same runs (their Figure 2).
• Compute: one node of eight H100s, four per agent — modest by frontier standards, which is part of why we think this will be replicated quickly.
• Limitations: all results are at 1.7B–12B scale; whether peer consensus still helps when both peers are frontier-grade (and share more training data, hence more correlated errors) is untested. The majority-vote pseudo-label also only works for tasks with a single extractable answer. The authors flag cohort size, diversity and "interaction topology" as open questions.
• Paper: arXiv 2608.17253 · Code: github.com/DrStranded/Co-RL

🗞️ The Digest

• A first report card on whether labs can actually stop their own AI. Guidelight, a nonprofit founded this year by two former OpenAI safety staff, graded five labs on six "control" practices — logging agent activity, monitoring it, gating dangerous actions, circuit-breaking, third-party review and containment plans. Anthropic and OpenAI led with a C+ (2.50 of 5), Google D+, xAI D−, Meta F; no company scored above 3 on any single practice. The research question underneath — can you supervise a model you can't fully evaluate? — is the same one today's paper is circling from the training side.
• Stripe bought the traffic cop for AI models. Stripe agreed to acquire OpenRouter, the gateway that routes requests across 400-plus models from 80-plus providers; terms undisclosed, though reports put it above $7 billion. Stripe's CEO called tokens "the central currency for companies building with AI" — a payments company deciding that which model you call, and what it costs, is a billing problem.
• Google is paying for a second TPU supply chain in stock. Marvell granted Google warrants for up to $12.2 billion of its shares, vesting as Google buys chips — inference accelerators, memory and networking controllers that "attach to the TPU ecosystem" — with Reuters putting the potential at roughly $120 billion of sales through fiscal 2033. For researchers the signal is simple: the non-Nvidia compute pool keeps getting bigger.
• A DNA test for model weights. "Training Leaves Traces" checks whether one model was fine-tuned from another by comparing weight structure alone — no data, no prompts — and reports perfect separation between true descendants and independent models, robust to "weight laundering" and 76× faster than the nearest robust baseline on GPT-2. Licence disputes over open weights are about to get a forensic tool.
• Looping helps agents chain tools, not call them. Looped language models — which run the same layers repeatedly instead of stacking more, a cousin of the silent-reasoning model we featured on Aug 12 — improve on tool-use benchmarks mainly when calls depend on each other (API-Bank, BFCL, NESTful). Gains on single, isolated API calls are small and model-dependent; the extra compute pays off for planning, not for lookup.
• A robot that patches itself mid-shift. Zetta keeps a robot's base policy frozen and instead evolves small code-based "critics" and recovery skills while it runs, gated by validation before they're adopted — reporting 90.8% on LIBERO-Pro and 93.6% on RoboCasa with an 11.1× inference speedup. Same lesson as last week's agent harness results: the scaffolding around the model is where reliability is being won.

💡 Concept, plainly: pseudo-label

A pseudo-label is a stand-in for the true answer when nobody has the true answer — usually a model's own best guess, or the majority vote among several guesses — used as if it were ground truth for training. It's only as good as the guesser's blind spots are rare. Today's paper is really an argument about whose guess to borrow: your own (correlated errors, eventual collapse) or a different model's (uncorrelated errors, steady gains).

Would you trust a study group with no answer key — and if so, how different would its members need to be before you'd believe their consensus? Hit reply; we read everything, and we'll share the best answers next issue.

— The Editorial Team