MoE models can be compressed, but not uniformly across layers
Sh0ny
17 августа 20262 min read
In short
A study of Qwen3.6-35B-A3B reveals an unexpected skew: late MoE layers tolerate aggressive expert pruning, while early and middle ones lose quality fast. That is a useful guide for compressing models, though not yet a universal recipe.
A Mixture-of-Experts model can be compressed considerably harder than it seems, provided you do not touch it uniformly. The study's main practical conclusion: the experts' margin of safety is distributed by depth, and later layers tolerate the pruning of low-importance experts far better.
The authors tested Qwen3.6-35B-A3B: 40 MoE layers, 256 experts in each, with top-8 routing. Experts were masked by the magnitude of their weights and the result was evaluated on XLCoST — a benchmark for translating code between languages.
Uniformly masking 30% of experts left only 150 of 300 results in the Good+Similar categories. But strategies concentrated on the later layers preserved 249–255 such results while disabling between 640 and 1,145 experts.
In a separate check on 500 queries the best compromise proved to be an even narrower policy: masking 50% of experts in layers 35–39. It preserved 419 Good+Similar results while disabling 640 of 10,240 experts. In other words, the question is not how many experts to remove but where exactly to do it.
The authors also tried reducing the number of active experts per token from eight to six. On a sample of 100 queries that gave a noticeable cut in execution time with no loss of Good+Similar, but combined with aggressive masking the approach does not yet add up to a dependable general strategy.
The limitations are substantial: the study was run on one model and one benchmark, and the conclusions rest on series of checks over 100, 300 and 500 queries. The observed reduction in time is not accompanied by an exact figure in the source description. Besides, the work so far provides a basis for future physical removal of weights, activation-based scoring of experts and fine-tuning, rather than a ready universal compression tool.
Even so the direction looks sound: for MoE it is more sensible first to build a map of sensitivity by layer and only then to cut the model. The standard "remove the same percentage everywhere" is not merely suboptimal here — it can throw away quality exactly where the model is most vulnerable.
If you had to compress a MoE model for production, would you first economise on the later layers or demand a check on every one of your own task types? Source: cs.AI updates on arXiv.org