The model sees the broken route but does not know whether to fix it
Sh0ny
13 августа 20262 min read
In short
In MoE models, quantising the KV cache changes expert selection for some share of tokens, but the switch alone does not tell you the answer got worse. That is a serious limit for smart inference repair: a local signal finds the problem but cannot tell a harmful change from a useful one.
The nastiest part of the experiment is not that 4-bit quantisation of the KV cache sometimes changes a token's route. It is that after detecting such a switch we still do not know whether it needs fixing.
In a Mixture-of-Experts the router chooses which experts will process a token. A small numerical perturbation can throw a token across a decision boundary — even when the router itself runs in BF16. On OLMoE-1B-7B with 4-bit KV, roughly a third of the measured damage turned out to be tied precisely to a change of route: the RMF estimate in the pilot came to about 0.31.
But detecting a switch proved markedly easier than assessing its consequences. The router's margin to the decision boundary predicted the fact of a flip with an AUC of 0.772. Determining from local observable statistics whether the flip had been harmful or beneficial, however, failed: the result was at the level of random guessing.
The practical conclusion is simple: a strategy of "roll back every suspicious route" may repair not only errors but perfectly good improvements. That means selective repair cannot be built on router confidence alone or on other local signals available during inference. You need either a more informative source of feedback or a willingness to pay for superfluous corrections.
The result's limitations are substantial. The work offers no mitigation; it describes the causal apparatus and the limit of detection for a particular family of features. The main numbers come from a pilot on OLMoE-1B-7B; the transferability of sign-inseparability between models is shown, but the benefit of a clean-reference remedy depends on the architecture. An experiment with a genuine int4 KV kernel gave a compatible estimate, yet the 95% confidence interval [-0.111; 0.394] includes zero — this is not an independent full replication. Finally, a strict test of the claimed barrier fell just short of the threshold, so what is at stake is not proven impossibility in general but a limit for the local features tested.
If you are building a system for automatic inference repair, which is more dangerous for you: missing a harmful route flip, or rolling back a beneficial one? Source: cs.AI updates on arXiv.org