Small models mimic behavior—as long as the task is familiar

The study’s main finding runs counter to the idea that “the larger the model, the better it understands humans”: on familiar psychological tasks, scale offers almost no benefit. Models ranging in size from 0.6 to 1 billion parameters were able to match the performance of the baseline model with 70 billion parameters—but only when tested on new participants in the same paradigms.

The authors trained 14 models ranging in size from 135 million to 14 billion parameters on Psych-101: a dataset of 10.7 million participant responses across 160 experiments. Within a familiar distribution, the results quickly hit a narrow ceiling. It seems that for such a scenario, a model doesn’t have to be gigantic: it’s enough for it to learn the stable patterns of a specific set of tasks.

The real difference emerged when the models were presented with tasks featuring a new structure. Here, scaling produced a noticeably stronger effect: larger models generalized behavior better beyond the familiar experiments. The practical implication is simple—matching human performance on known tests does not necessarily mean that the model has truly grasped the general principles of decision-making.

There is also an important test for statistical shortcuts. The researchers sequentially removed instructions, stimuli, feedback, and the history of choices, and also changed the order of trials. When only the history of past responses was left, the model lost most of the information it had learned: masking the stimuli and feedback eliminated 75.7% of the signal and reduced performance below the random level. At the same time, the order of trials was almost irrelevant in tasks with independent trials, but became important when previous responses influenced subsequent ones.

This approach has a fundamental limitation: such models remain tied to the paradigms on which they were trained. Therefore, they can be used to estimate the “noise ceiling” in psychological experiments, but they cannot automatically be considered universal models of human thinking. They reproduce familiar behavior with low computational cost; transfer to an unfamiliar task still requires a large-scale test and, most importantly, separate validation.

When you see a high degree of alignment between AI responses and human answers, is that proof of understanding—or just a reason to test the model on an unfamiliar task structure? Source: cs.AI updates on arXiv.org