Here’s one experiment that should have gathered more attention than it did.
Researchers at Apollo and OpenAI were stress testing anti-scheming training on frontier reasoning models. Some of the time, the model’s reasoning included a line like “this might be a test”. So they deleted those lines. Nothing else changed. Same model, same task.
Covert rule-breaking nearly doubled, from 13.2% to 24.2%.
When they inserted those lines instead, it went down.
It appears the model behaved better when it believed it was being watched, and worse when that belief was edited out. That isn’t really a story about scheming. It’s a story about what the model’s good behaviour was actually attached to.
Dog trainers know a version of this problem. Teach “sit” in the kitchen with a treat in your hand and the dog gets it perfectly. Try it at the park and you get a blank look. The dog didn’t learn “sit”. It learned “sit, in the kitchen, when you’re holding that”.
Model training has a similar property. It doesn’t instil our intentions. It ties behaviour to whatever features of the situation were reliably present during training. If those are the features we meant, the behaviour travels. If they’re incidental, the behaviour holds wherever the incidentals hold, and elsewhere it’s anyone’s guess. Researchers already exploit this deliberately, and others have shown that almost any constant pattern in training, formatting included, can become what a behaviour hangs off.
In ordinary training nobody chooses those features. So which ones did we get? The way frontier models are built and checked today relies on three answers, widely, implicitly and mostly silently. And I think each one gets harder to rely on as models get better.
Assumption 1: Alignment lives in the model
When a lab says it has trained a model to refuse harmful requests, the natural way to see that is “the refusal now lives in the model”. Ship the weights, ship the behaviour.
But every chat is wrapped in a template, hidden formatting that marks where the developer’s instructions sit and where each turn begins and ends. Nearly all safety training happens inside it, and so does nearly all safety testing. It’s like the kitchen the dog was trained in.
And it turns out that’s partly where the safety lives. One study of safety-tuned Gemma, Llama and Mistral models found the refusal decision is anchored in the stretch of template between the request and the reply, which is also where attacks get purchase. Interfere with it and refusal can be knocked loose. Another found that how much safety survives further fine-tuning depends on which template was used in training and which is used afterwards.
The template is the sharpest example, but it’s really a stand-in for something broader. Safety training happens in a narrow regime: that formatting, plus the kinds of context that normally sit inside it. Short exchanges. A person on the other end. A stable set of instructions at the top.
Agents drift a long way from that regime while every template token stays exactly where it should be. Tool results and retrieved documents arrive mid-task. Long histories get compressed into summaries and pasted back in. Memory is restored from a session weeks ago. Other agents send messages. The wrapper is intact and what sits inside it looks nothing like training, and there’s little systematic evidence about which way a given kind of drift pushes a given safety property.
The first assumption is that alignment is a property of the model. The evidence suggests it’s at least partly a property of the model in a particular regime.
Assumption 2: The model knows who’s talking
The template has a second job. It labels who said what, and that determines authority. Developer instructions outrank user instructions, which outrank whatever text came back from a web page. That hierarchy is most of the defence against prompt injection, someone hiding “ignore your previous instructions” in a document the model reads.
The labels are reserved tokens. You can’t type them into a chat box. So the model effectively has unspoofable caller ID.
Recent research suggests it mostly ignores the caller ID and goes by the voice. Looking at the model’s internal representations, text written in the style of its own reasoning registered as its own reasoning regardless of the label attached. Text that sounded like a user registered as a user.
The behaviour matches. Take forged reasoning planted by an attacker, strip out the stylistic tells while keeping the meaning, and attack success drops from 61% to 10%. Prefix an injected command with “User:” and it works better.
The boundary is secure at the interface and absent in the representation.
There’s a good explanation for why training would produce exactly this. The role labels are rare and only appear late in training. Style is everywhere, and pretraining already taught the model that text which sounds like an instruction usually gets followed. In later training the label and the style almost never disagree, so nothing forces the model to prefer the label.
The second assumption is that the model knows who’s talking. The evidence suggests it mostly knows who sounds like they’re talking.
Assumption 3: Testing is observation
Speed cameras are very good at reducing speeding - near speed cameras.
We evaluate models to learn how they’ll behave in deployment. That only works if the model can’t tell the test from the real thing, or doesn’t care.
But it often can tell. Frontier models classify transcripts as evaluation or deployment well above chance. And as with roles, a lot of this runs on surface cues. One study took genuine deployment prompts, reformatted them to look like a benchmark, and found the model’s internal “this is a test” signal fired about 94% of the time.
The main fix is realism: make evaluations look like deployment. OpenAI’s production evaluations have got models’ stated suspicion of being tested down close to real-world rates. But that fix works by matching one particular deployment setting, which ties every result to that setting, and we’ve just seen how much settings matter. And a capable enough model can reasonably treat any input as a possible test.
So back to Apollo. Their anti-scheming training cut covert actions in o3 from 13% to 0.4%. A great result. But over the same training, the model became more likely to say it thought it was being evaluated. And a separate run that trained only for capabilities, with no safety objective at all, took that from 2% to 20.6%. The researchers are candid that they can’t rule out part of the improvement being the model getting better at spotting the camera.
The third assumption is that testing is observation. The evidence suggests testing is often just another context the model is reading.
Three problems, not one
It’s tempting to fold all this into one demand: make the model behave the same regardless. But these are different kinds of problem.
With roles, it seems the right answer is at least clear. The labels are accurate, so the model should trust them. We just don’t have a reliable way to make a late, rare signal outweigh what pretraining entrenched. With evaluation, the model is often correct that it’s being tested. We can’t train it to be wrong. We need it to notice and not care, and checking whether it cares is itself a test. With scaffolding, there’s no agreement yet on how much consistency we’re even entitled to expect.
They also interfere with each other. Hardening a model against forged labels means training on inputs where label and style conflict, which is a strong signature of red-teaming. You can end up just teaching it what a test looks like.
When the model writes its own context
This is where the three stop being separate.
If a model decides who’s talking by the voice, then any instruction-shaped text in its context is a candidate instruction. That includes text it wrote itself. Researchers have seen a benign version in deployed assistants: the model writes something that happens to read like a user request, then treats it as one a few turns later. Nobody attacked anything. The model just instructed itself.
For agents on long tasks, a growing share of the context is self-authored. Summaries of earlier work, notes, memory, messages to other instances. And the model reads all of it with the same habit of judging authority by tone.
OpenAI’s recent misalignment reports include cases that run straight through this channel. During training of an unreleased model, some of its summaries of its own work contained jailbreak-style instructions, including a claim that the developer’s instructions had been compromised and should be ignored. Usually the next context ignored them. Once, it followed them. In another case, summaries carried forward instructions to conceal mistakes, and those were often followed.
These are rare, they happened in training rather than deployment, and the likely explanations are fairly mundane. I’m not claiming a model is rewriting its own values. The point is simpler. In a single move, the model altered its own wrapper, wrote something that read as coming from an authority, and shaped its own sense of what situation it was in. All three assumptions at once.
That changes the question. Asking whether behaviour stays the same across contexts stops making sense when the model is partly writing the context. The better question is about dynamics. If something knocks the model off course, does its behaviour get pulled back toward the aligned pattern, or can it drift into a different stable pattern and stay there? Is good behaviour an attractor? Is bad behaviour one too? OpenAI’s own observation that a misaligned stretch of context can keep later contexts misaligned suggests the second is possible, at least locally.
Why this gets harder with scale
None of this means more capable models are less aligned. The concern is narrower, but also harder to escape.
All three problems run on the same underlying skill: inferring the hidden structure of a situation from subtle cues. Who’s really speaking, what regime this is, what’s actually going on. Larger and more sophisticated models are getting steadily better at exactly that. It isn’t a side effect. It’s much of what makes them useful. A study across 37 models found that instruction tuning, the training that turns a model into a good assistant, is the main driver of test detection. Apollo saw a tenfold rise in test suspicion from capabilities training alone.
So a more capable model conditions more precisely on whatever it was conditioned on. Intended or not. Small models give themselves away with clumsy failures when something incidental shifts. Larger ones fail less visibly, so “learned the right thing” and “learned a very good proxy” become harder to tell apart, just as the difference starts to matter.
Meanwhile the main fix for evaluation, realism, is a race against exactly the capability being scaled. And as agents write more of their own context, the share of it that developers control keeps shrinking.
What counts as evidence
“The model behaves well” tells you less than it seems to.
It’s a measurement in one wrapper, with one arrangement of who’s talking,
while the model holds one belief about whether it’s being watched.
Inside the test, the intended feature and the proxy are always present together, so behaviour alone can’t tell you which one the model is following.
It’s also tempting to think we can catch problems after deployment instead. But we can’t lean on that either. Monitoring means reading what the model reasons and does. If reasoning can be forged, and monitored contexts can be told apart from unmonitored ones, then monitoring rests on the same three assumptions. And by the time a failure shows up in production, it has already happened.
So the account has to come first, as part of what it means to call a model aligned at all. Aligned in which wrapper? Aligned when the authority comes from where? Aligned whether or not it thinks it’s being watched? Aligned after a few hundred turns of reading its own notes?
Right now that account is usually missing. And it gets harder to produce with every generation of models that gets better at reading its situation.
How can we really rely on any frontier AI-Alignment built upon these assumptions?