High temperatures don't save LLMs from giving identical responses

LLMs can produce different formulations while remaining essentially consistent with one another. The authors refer to this as the “artificial collective intelligence” effect: even at high temperatures, answers to open-ended questions often converge on a narrow set of ideas.

The problem here is not merely aesthetic. If a model almost always chooses the same type of reasoning, it is less suited for brainstorming, searching for unexpected hypotheses, and generating alternative solutions. Simply increasing the temperature, according to the study, does not eliminate this homogeneity.

The proposed method consists of two steps. First, the model must itself select an unusual, individual “meta-person” who sets the starting point for the response. Then, Filtered Temperature Scaling is applied: first, Top-p filtering discards options that are too unlikely, and then an extreme temperature of 4.0 or higher is applied to the remaining candidates.

This is an important combination. A high temperature without filtering can disrupt the coherence of the text, while filtering without sufficiently strong scaling leaves the model within its familiar probability range. In the proposed scheme, grammatical and probabilistic acceptability are preserved first, and then the selection space is expanded.

In an experiment on the INFINITY-CHAT dataset, we tested open-weight models with fewer than approximately 20B parameters. The average pairwise cosine similarity of the responses decreased from approximately 0.85 to 0.65, and for most questions, the metric fell below the 0.7 threshold.

However, this result must be interpreted with caution. A decrease in semantic similarity means that the responses became less similar, not that they became more accurate, insightful, or useful. Furthermore, the available description lacks details on how factual correctness, robustness against bad agents, and the cost of the additional generation step were measured.

The practical conclusion so far is this: if you need more than one “averaged” version of a response, it’s worth managing diversity at the level of the model’s initial role, rather than simply adjusting the temperature. At the same time, production systems will require a separate check to ensure that the achieved diversity hasn’t resulted in the model confidently generating unnecessary noise.

Source: cs.AI updates on arXiv.org