Not all LLM layers are the same: GLIDE compresses the KV cache where it is safe to do so

When the LLM context grows to tens of thousands of tokens, the main bottleneck is not computation but memory: the KV cache swells, and the I/O load during decoding chokes the throughput. The standard solution—replacing softmax attention with linear recurrence—works, but it degrades quality. GLIDE proposes a more nuanced approach: rather than making cuts everywhere, it makes them selectively.

The key idea is based on an observation the authors call “layer-wise heterogeneity.” The early layers of the model are highly sensitive to the removal of softmax: remove it, and the model loses expressiveness, which is difficult to compensate for in later layers. Deep layers, on the other hand, exhibit redundancy and can easily withstand aggressive replacement with a linear alternative. This is not a hypothesis but an empirical fact that changes the very formulation of the problem.

GLIDE exploits this disparity. Each layer receives an adaptive balance between linear recursion and a variable-sized softmax window. Early layers retain a wide softmax window, while deep layers switch to linear aggregation. Unlike uniform hybrids, which compress everything equally, GLIDE compresses the softmax trace non-uniformly across the model’s depth, preserving expressiveness where it is critical.

The practical result: end-to-end latency on long contexts is reduced without compromising quality. For those running LLMs on long prompts in production, this is a direct path to reducing inference costs—not by pruning the model, but by intelligently distributing the load across layers.

The main question that remains open is: to what extent does this layer-wise profile transfer across architectures? If the sensitivity of layers to softmax is universal, GLIDE becomes a standard optimization technique. If not, each new model family will require separate profiling.

Source: cs.AI updates on arXiv.org