🎧 The grading rubric that rewrites itself

A 4-minute read.

We spent the weekend with a paper that asks a question every teacher has quietly wrestled with: do you grade the final answer, or the working? The authors' answer is sneakier than either — you keep rewriting the rubric while the student learns.

📄 The Paper: a grading rubric that rewrites itself

Reinforcement Learning with Evolving Rubrics as Rewards for Audio Reasoning (arXiv, Aug 3) tackles a problem in training audio models — the kind that listen to speech, music, or street noise and answer questions about it. These models are increasingly trained with reinforcement learning, which just means: the model tries an answer, gets a score, and adjusts itself to score higher. The whole game hinges on what that score actually measures.

Today there are two common ways to score. Outcome rewards check only the final answer — so a model can be right for completely wrong reasons and still get full marks. Process rewards grade the reasoning itself, but against fixed, hand-written criteria — one generic mark scheme for every question, which the model eventually games or outgrows.

The paper's system, AudioRubrics, does something more like a good private tutor. For every training clip, a stronger "teacher" model listens to the raw audio and writes a custom rubric — a checklist of what a good answer to this specific clip should notice ("identifies two overlapping speakers", "catches the tempo change"). The student model's attempts are then scored against that checklist. And here's the twist: mid-training, the teacher watches the student's actual attempts and rewrites the rubric — retiring criteria the student has mastered and reweighting toward whatever it currently gets wrong. The mark scheme keeps moving to wherever the student is weakest.

Why we think this matters beyond audio: reward design is arguably the bottleneck for reinforcement learning on fuzzy tasks with no neatly checkable answer — writing, medicine, open-ended agent work. Rubrics-as-rewards were already a growing idea this year; making them evolve during training is the new move, and nothing about it is audio-specific.

Our honest uncertainty: the teacher and judge here is a frontier model (Gemini-3.1-Pro), and the authors themselves report that gains scale with how capable that judge is. So this recipe needs a teacher smarter than the student — fine for training small models, but what happens at the frontier, where no smarter teacher exists? The paper doesn't really answer that, and neither can we.

Go deeper ▾

• Setup: a 7B audio-language student model; rubrics are generated from the raw waveform (not a transcript), then regenerated and reweighted per group of rollouts as static criteria saturate.
• Numbers: 78.0% on MMAU Test-mini, 65.8% on MMAR, 65.9% on MMSU — three benchmark exams for audio understanding and reasoning — including an 8.9% relative gain on MMSU's perception split over the best baseline.
• A quiet but nice result: training converges to stable reasoning lengths without the degenerate "ramble longer for more reward" collapse that plagues RL-trained reasoners.
• Limitations as we read them: the reward depends on a proprietary frontier judge (cost + circularity), and the benchmarks are still largely multiple-choice-style — we'd love to see this on open-ended audio tasks. Correct us if we've misread.
• Links: paper · project page with code, model weights, and the rubric dataset.

🗞️ The Digest

• Alibaba ships Qwen3.8-Max — and says the weights come next. The 2.4-trillion-parameter flagship (95B active per token, 1M-token context) is live via API, with model weights "scheduled for release next week." If that holds, it would be the largest openly weighted frontier model to date.
• DeepSeek's V4-Flash goes official. The July 31 release keeps the April preview's architecture untouched — every gain came from post-training — yet posts 82.7 on Terminal-Bench 2.1 and ships adapted for Codex. Post-training alone moving numbers this much is the story.
• Anthropic clarifies: no open-weights ban. Its position paper argues for chip export controls, a crackdown on industrial-scale distillation, and mandatory safety testing for any sufficiently capable model — open or closed — rather than restricting open weights as such. Expect this framing to anchor the policy fight ahead.
• ByteDance opened the public API for Seedance 2.5, its latest video generation model, this week — reportedly generating 30-second single-shot videos with native audio, though we haven't tested it ourselves yet.

💡 Concept, plainly: reinforcement learning

Reinforcement learning is training by trial and score: the model attempts a task, a reward signal tells it how well it did, and its weights shift toward whatever scores higher. It's how models learn things nobody can demonstrate line-by-line — and it's only ever as good as the scoring. Today's paper is really a paper about making that score smarter.

If you were the teacher model, what would you put on the rubric for a recording of a crowded café? Hit reply and tell us — we read every answer, and the best ones shape what we cover next.

— The Editorial Team