MoE without a tokenizer: byte entropy determines who gets to process the patch

In a tokenizer-free LLM, the model can process not predefined tokens, but dynamic groups of bytes—patches. The problem is that each such patch then receives the same amount of computation, even though its meaning and size can vary greatly.

EntropyMoE proposes using an existing signal—the patch’s entropy—for routing to Top-K experts. Simply put, the model first groups the bytes and then, based on that same characteristic, decides which experts to send the resulting fragment to.

An important detail: routing takes into account not only entropy but also the patch length. And the contribution to the load is determined by the coverage of the bytes. This makes more sense than treating every fragment as an equal unit of work: a long patch can indeed represent a larger volume of input.

According to the authors, EntropyMoE achieved the lowest held-out bits-per-byte among comparable dense and sparse models, while maintaining comparable accuracy on subsequent tasks. But the main result here isn’t the promise that “MoE has become faster,” but rather this: patch entropy can serve as a coordinate for conditional computations even without classical tokenization.

There is also a limitation that’s easy to overlook in a summary. The abstract lacks specific data on actual speedup, inference cost, model size, or hardware performance gains. Therefore, for now, it is more accurate to speak of a more efficient distribution of computations and prediction quality rather than a proven practical reduction in costs.

If you had to choose between more complex routing and simple, uniform processing of all fragments, which would be more important to you: better quality with the same architectural complexity, or proven time and cost savings in production? Source: cs.AI updates on arXiv.org