MLLM was accelerated by a factor of 3.09, leaving 5.56% of visual tokens
Sh0ny
10 августа 20262 min read
In short
Researchers trained a multimodal model to predict in advance which parts of an image are needed for a response and to discard the rest even before the language layers are activated. On LLaVA-NeXT-7B, this preserved 97.5% of the original quality, but the results cannot yet be automatically generalized to all models and tasks.
The key point here isn’t that MLLM has once again introduced a pruning mechanism. What’s more important is the timing: the system discards unnecessary visual tokens even before the first language layer, rather than after several computationally expensive passes, when part of the computational resources have already been expended.
In the conventional approach, the model examines the attention in a pre-selected middle layer and uses it to decide which image fragments to keep. But the same layer isn’t equally useful for different questions. For one image, the relevant signal appears earlier; for another, later.
The MAP method solves both problems at once. During training, it compares the attention for the source and control questions, selects the layer best suited for a specific example, and passes its behavior to a lightweight predictor. When the model is running, this predictor evaluates the importance of visual tokens based on multimodal features—without constructing attention maps.
Across ten benchmarks for LLaVA-NeXT-7B, MAP retained only 5.56% of the visual tokens while maintaining 97.5% of the model’s quality without pruning. The end-to-end speedup was 3.09x. The practical significance is clear: the efficiency gains do not come from faster analysis of a large number of tokens, but from the fact that most of them never even reach the computationally intensive layers.
However, there are important limitations. All figures cited refer to LLaVA-NeXT-7B and ten benchmarks—the source provides no data on other architectures, real-world user queries, or the cost of training the predictor. Furthermore, the method must still correctly assess the importance of tokens: an error made at an early stage cannot be corrected by subsequent layers. Therefore, this is not universal proof that any MLLM can be seamlessly accelerated by a factor of 3.09, but rather a well-defined result for a specific model and testing regime.
If you had to choose, which would be more important for such an acceleration: a 2.5% loss in quality or the risk of missing a small but critical fragment of an image?
Source: cs.AI updates on arXiv.org