The model detects a corrupted context but does not report an error
Sh0ny
11 августа 20262 min read
In short
The study reveals an unfortunate disconnect between what can be detected within a language model and its ability to predict whether an answer is correct. For monitoring purposes, this means that the model’s confidence alone is not enough, but the internal signal cannot be applied without taking into account the specific model and the type of error.
The problem of monitoring AI turned out to be more complex than it seems: a model may “know” that the context is corrupted, but this does little to help determine whether it will make a mistake in its final answer.
The researchers tested this on multi-step arithmetic chains. Linear probes detected context corruption almost perfectly; however, the signal they detected did not allow for reliably distinguishing a correct result from an incorrect one. Moreover, this observation held true across different model families, including reasoning models.
There is an even more troubling detail. When the models were forced to output confidence in a structured format, the answers effectively boiled down to two values, and their error rates remained indistinguishable. Even maintaining the signal over several steps did not help predict the outcome—the preliminary hypothesis that “persistence beats peak” was not confirmed.
The practical takeaway is not that internal probes are useless. They can supplement a model’s standard verbal confidence, but they should not be treated as a universal “stop-error” button. In the tests, the branch-and-pick strategy yielded positive results for various models and did not disrupt correct chains on Llama-3.1-8B: it corrected four and did not disrupt any. In contrast, reprompt and replace-prior disrupted correct chains at roughly the same rate as they rescued incorrect ones.
The limitation here is fundamental: no universal intervention has been found. The effect depends heavily on the model and the type of error, and the mere fact that a corrupted context is detected does not necessarily mean the model will produce an incorrect response. Therefore, a working monitoring architecture must choose its response based on two parameters: which model is running and exactly what error has been detected.
If you were to implement such monitoring today, what would you entrust with the authority to halt the response: the model’s confidence, an internal probe signal, or a combination of both? Source: cs.AI updates on arXiv.org