A confident LLM error can be stable — and that is more dangerous

A language model's nastiest error is not the one it doubts. A study on arXiv shows that some wrong answers stay confident and stable even after small changes to the request.

That changes the familiar logic of self-checking. If a model has revised its answer several times and kept the same confidence, that is not yet proof it has found the truth. It may simply have locked into a wrong state.

The authors tested this on a set of binary factual questions from different domains. One test looked at changes in confidence and the number of overconfident errors under a forced choice of answer. Another measured how much the model's internal states change.

The picture that emerged is curious. Abstention-aware self-critique — self-criticism with the option of declining to answer — did help reduce losses in the domains where the audit found confidence problems. But direct baseline methods with known labels rated that effect more highly. In other words, self-checking is useful, yet it is no substitute for proper calibration on labelled data.

There is a more unexpected result too: across three open-weight models, self-critical prompting consistently reduced the sensitivity of hidden states at different layers. That looks less like a change in the wording of the final answer and more like a local stabilisation of the model's operation under the prompt.

But that is also where the main risk lies. Errors the audit flagged as overconfident did not prove clearly more sensitive to small perturbations than confidently correct answers. So "shaking" the model with an extra question may not be enough: some wrong answers are stable precisely because the model is miscalibrated, not because its reasoning is fragile.

The work's limitations matter too: the study rests on a binary factual audit, considers three open-weight models and does not claim that local stabilisation equals good calibration. So in an applied system I would not treat re-checking as an independent guarantee of quality. Important answers need external verification, the option of declining to answer, and metrics validated on labelled examples.

If your system already uses self-critique, does it measure whether there are fewer errors, or only how confidently the model repeats its own answer?

Source: cs.AI updates on arXiv.org