For language models, "warm-up" depends on architecture, not scale

In language models, researchers found not a universal scale effect, but rather an architectural dependency: some models exhibit a sharp transition in signal processing, while others scale their performance almost linearly. This could change how we explore the “working space” in neural networks: we’ll have to look not only at the model’s size but also at how it computes.

The authors introduce the Ignition Index—a measure of the sharpness of the transition. They feed signals of varying strength into the model and track how the accuracy of a linear probe changes across layers. They then tune the sigmoid function: a high slope coefficient indicates an almost abrupt “ignition,” while a low one indicates a gradual accumulation of information.

It is important not to confuse this metric with a measure of consciousness. It tests a specific dynamic property related to Global Workspace Theory in linguistic representations. And yet the result is interesting: in a study of 11 models across five architectural families, feedforward transformers achieved an index that was, on average, 89% higher than that of SSM models. Mamba’s profiles turned out to be nearly linear—without a noticeable global jump.

The recurrent Huginn-3.5B revealed another nuance: the “ignition” was 2.12 times stronger along the iteration axis than along the model depth. In other words, for some architectures, it is not the number of layers that matters most, but rather the number of computation passes. For Pythia-410M, the transition occurred at training step 256, even before the induction heads were formed. It is possible that important mechanisms do not emerge simultaneously or in the order that is commonly assumed.

The study has significant limitations. The relationship between the index and model scale and signal strength was not confirmed; therefore, the metric does not yet explain why the sharp transition occurs or to what extent it generalizes to other tasks. The study also does not prove that the models possess consciousness or a fully-fledged global workspace—it demonstrates measurable probe behavior in a selected set of models. The experiment code is published in the Ignition Index repository.

If you had to choose between scaling up a model and changing its computational architecture, which do you think is more likely to result in a qualitative leap in reasoning? Source: cs.AI updates on arXiv.org