🔓 The AI's private thoughts weren't private

A 5-minute read.

We came into this week thinking about how models reason. We're leaving it thinking about who else gets to read that reasoning — because a new paper shows the answer is "anyone who asks the right model nicely."

📄 The Paper: the AI's private thoughts weren't private

The paper is "Stealing Reasoning Traces from Proprietary LLM APIs" (arXiv, Aug 10), from a group across the ELLIS Institute Tübingen, the Max Planck Institute for Intelligent Systems, and the Tübingen AI Center, with industry co-authors. The question it asks is narrow and uncomfortable: when a model "thinks" before it answers, and the provider hands you back an encrypted copy of that thinking, is the encryption actually protecting anything?

First, why encrypted thinking exists at all. Today's strongest models produce a long internal monologue — a chain of thought — before the final answer. Providers don't want to show you that monologue (it's the crown jewels), but they also don't want to store it on their own servers between turns. So they compromise: they encrypt the reasoning into an opaque block and hand it to you, and your app passes it back on the next turn to keep the conversation coherent. The reasoning lives in your possession the whole time — just locked.

The attack is almost embarrassingly simple, and it turns on one design choice: within a provider's ecosystem, those encrypted blocks are interchangeable across models. A block minted by a top-tier model will be happily accepted by its cheaper, less-guarded sibling. So instead of trying to crack the lock, you hand the locked box to the sibling and ask it to read the contents out loud. The weaker model decrypts the reasoning as part of normal operation and — with a simple jailbreak — transcribes the plaintext for you. You never break the strong model's defenses; you just route around them.

Think of a sealed diplomatic pouch that any embassy in the same country can open. You can't pick the lock, so you don't try — you walk the pouch to the least careful embassy on the list and ask the clerk there to read it to you.

Why it matters: a lot of people have been treating these encrypted blocks as safe to leave lying around. The authors scraped 315,320 of them from public GitHub and Hugging Face repositories — blocks that developers had committed without a second thought — and recovered 367 pieces of personal information and 182 credentials hiding inside the "protected" reasoning, including API keys, passwords, and private keys from real user sessions. The lock people trusted was a lock only against the honest.

Our honest caution: this is a snapshot of specific API versions from early July 2026, and the internals are proprietary and change without notice — the authors say plainly that as of August their headline demonstration no longer reproduces, because providers shipped fixes after being told. So read this as "here is a class of mistake and how it was caught," not "here is a hole that's open right now." That's the healthy version of security research: found, disclosed, patched, published. Correct us if we've misread the mechanism.

Go deeper ▾The mechanism: reasoning is returned as an authenticated-encryption (AEAD) envelope so the provider can verify a block on replay without storing it. The flaw isn't the crypto — it's the missing binding: blocks aren't tied to a user, conversation, or single model, so they're portable across sessions and across sibling models.Tested against: Anthropic, OpenAI, and Google model families, using a weaker sibling in each as the "decoder." Google's lineup showed the broadest cross-model compatibility in their Table 1.Four ways to abuse it: distillation (harvest a rival's reasoning to train your own model), secret extraction (pull PII and credentials from publicly shared blocks), hidden prompt injection (smuggle instructions inside a block to poison an agent), and jailbreaking (recover harmful content the model hid from its final answer).Numbers: across 6,708 real sessions, 4.9% leaked at least one sensitive item — 62 API keys, 33 passwords, 30 personal emails, 7 private keys.Fixes proposed: bind each block to its originating user/conversation, isolate blocks across models at the gateway, add replay anomaly detection, or just generate reasoning fresh each turn and delete it. Disclosed to the affected providers, Microsoft, and Hugging Face before publication. Paper.

🗞️ The Digest

• Claude's text now carries an invisible watermark. Anthropic began embedding machine-readable watermarks into text from its newer models, applied at the model level so the mark survives copy-paste and light edits, with signed C2PA provenance on image files. It's driven by the EU AI Act's transparency rules (Article 50, in force Aug 2) but applied worldwide — the first big lab to mark plain text this way.
• An inference-chip startup doubled its valuation in a month. Etched raised $700M at a $21B valuation (led by Jane Street, which is also its first customer), up from $10.3B in July. Its bet is narrow hardware — chips built only for inference, not training — which is where the money increasingly is once a model is deployed rather than being made.
• A $15 agent run matched a $575 one. StateM reports 95.3% on the Terminal-Bench 2.1 agent benchmark by fixing the harness, not the model — giving the agent durable state, checked transitions, and recoverable runbooks so it stops losing track mid-task. Same idea we saw in last week's spec-first refactor: reliability is increasingly a scaffolding problem, not a weights problem.
• Hallucinations don't just persist — they mutate. The Hallucination Snowball traces how one fabricated fact moves through a multi-agent pipeline: a made-up number becomes a calculation, then a sentence, then a conclusion nobody can trace back. Detection fell from 72% at stage one to 51% at stage four — and catching errors at the first handoff worked far better than checking only at the end.
• An open-weight model that watches while it talks. MOSS-VL is an 11.3B streaming vision-language model that processes new video frames while it's still generating text — reaching first token up to 5.1× faster than a comparable baseline as visual context grows. Checkpoints, training recipe, and code are all public.
• Can an AI build a 3D world from a sentence? Not yet, cleanly. VibeWorlding asks agents to construct interactive 3D scenes end-to-end from a request; frontier models land below 60% success, with precise 3D editing the main wall. The team released a benchmark of 6,828 queries so the gap is now measurable.

💡 Concept, plainly: chain of thought

A chain of thought is the step-by-step reasoning a model writes to itself before its final answer — the scratch work, not the exam answer. It usually makes models more accurate on hard problems, which is why it's so valuable that providers encrypt it. Today's paper is a reminder that this scratch work is real text that can contain secrets, whether it's hidden behind a lock or just hidden from view.

Here's our open question for you: if a model's reasoning can be read by feeding it to a weaker sibling, should providers show it to you in the first place — or is hiding it the real problem? Hit reply; we read everything.

— The Editorial Team