A neutral test fails to detect how dialogue reinforces an LLM's bias

The most important finding of this study is inconvenient for conventional LLM testing: a biased user context can amplify the model’s bias. Moreover, this does not manifest in all systems or for every query, so the average result from neutral tests can easily paint an overly optimistic picture.

The authors tested eight state-of-the-art instruction-tuned LLMs in multi-turn dialogues. In six out of eight models, biased responses occurred more frequently after a biased user reply than in a zero-shot scenario—where the model immediately receives the target prompt without any conversation history.

It’s not just the 6 out of 8 figure that’s important here. The study distinguishes between two effects: the mere fact of interacting with a user who expresses bias, and the meaning of a specific remark. This reveals that a model can pick up the direction of reasoning from the context, even if the next prompt itself appears relatively neutral.

But there is also a paradox. When bias is expressed too explicitly, some models, on the contrary, begin to suppress it. Likely, balancing mechanisms kick in: the system recognizes an explicit risk signal and tries not to express bias openly. As a result, the hidden influence of a conversation can be more dangerous than a blatant provocation, which the model has already learned to detect.

The practical takeaway is that we need to test more than just answers to individual questions. It’s worth adding dialogue chains to the tests: first, the user sets the context or demonstrates a controversial line of reasoning, then asks the model to make a decision. Otherwise, we’re measuring the model’s accuracy under controlled conditions, rather than its behavior in a real-world conversation.

The limitations are also significant. The available description does not name eight models, does not provide effect sizes for each of them, and does not analyze specific categories of bias. Therefore, the paper cannot be interpreted as proof that any LLM necessarily adopts the user’s views: the result depends on the model, the wording, and the context. The authors mention a benchmark of 24,300 user prompts evaluated by a panel, distributed across all 81 cells of the “user bias × human bias” interaction matrix, but the abstract does not provide details about these cells.

If you’re testing an AI system for real-world conversations, which is more important right now: catching explicitly biased responses or examining how the model gradually changes under the influence of the conversation history? Source: cs.CL updates on arXiv.org