How to add compute at generation time without making prompt processing heavier
Sh0ny
14 августа 20262 min read
In short
Scaling an LLM the usual way makes both prompt parsing and answer generation more expensive at once. The Dual-Flow Transformer proposes separating those costs, but the price of the idea shows up as a new trade-off between the speed of the two phases and prediction quality.
Large language models have an awkward feature: processing the prompt and generating the answer load the hardware differently, yet they usually get the same portion of compute. So adding layers or width improves the model at the cost of both phases at once — even when the user only cares about the quality of the continuation.
The Dual-Flow Transformer separates these jobs. The main flow processes the prompt completely and creates an ordinary KV cache. The auxiliary flow does not run while the prompt is parsed and switches on only from the last position — from there it adds computation specifically for predicting the continuation.
These are not two independent models. The flows share the main attention, MLP and output matrices, and use a common cache. They keep separate token embeddings and a light connection between the flows. In theory that allows already-loaded weights and KV cache keys and values to be reused during batched processing.
The most practical consequence shows up in MoE models. The number of experts in the main and auxiliary flows can be tuned independently: you can control the cost of prompt processing, the cost of generation and the final quality separately. In other words, a model can be made relatively light at the input while getting more computation for each subsequent token.
By the reported experiments, on token-matched comparisons Dual-Flow showed lower validation loss across various architectures and data configurations. But that does not yet equal a faster real service: the available description carries no measurements of latency, throughput or cost on specific hardware. The authors themselves show a trade-off between prompt processing, decode-time computation and quality, and the work is published as arXiv v1.
The idea looks important not because it adds one more architecture but because it changes the scaling question: not "how do we enlarge the model as a whole" but "in which phase does it actually need extra computation". If you had to choose between a faster first answer and a stronger continuation, which matters more for your tasks — a short prefill, or the quality of each subsequent token?
Source: cs.AI updates on arXiv.org