A year of real LLM traffic: why average metrics mislead

The problem with testing LLM serving is not necessarily a shortage of data. More often we look at it from too far away: how many requests in total, which models are popular, what the average load is.

Such a view is convenient, but it hides the behaviour on which caching and load balancing depend. The same total traffic volume can load a system differently — depending on which users address which models and how those links change over time.

The authors analysed a year-long production trace from Chutes. It contains requests from many users to many models: not only popular ones but models from the long tail. The analysis was carried out at several levels — from the overall picture and temporal dynamics down to individual models and users.

The practical conclusion is simple: a benchmark for LLM serving assembled on short or synthetic traffic may be testing a different system from the one you intend to operate. A fairer assessment needs scenarios where changes in load and the structure of user–model interactions are visible, not just averaged indicators.

There is an important limitation as well. This is a study of a single production trace — Chutes — albeit over a whole year and with broad coverage of models and users. The work gives no universal portrait of the whole market, and detailed results depend on what data will be available in the full trace. The authors promise to release it alongside the paper so other researchers can use real workloads rather than sampled or synthetic ones.

If you test infrastructure for LLMs, what influences your decisions more right now: the average load, or the rare user scenarios it hides? Source: cs.AI updates on arXiv.org