Down with 85% latency: Generative recommendations are speeding up

Generative recommendations are an attempt to reformulate classic ranking as an autoregressive generation problem: the model learns not to rank items, but to “complete” the identifier of the next product. It sounds elegant, but in practice it runs into two obstacles: tokenization, which fails to recognize cross-domain relationships, and beam search, which slows down inference to the point where it’s unusable in real time. The GenCDSR arXiv preprint tackles both problems at once—and the reduction figures look promising, though with some caveats.

The authors’ main criticism of existing generative recommendation methods is that they tokenize items within a single domain and fail to account for the fact that a user, say, watches movies and reads books, and their interests across these domains are correlated. GenCDSR proposes cross-domain hybrid tokenization: a multi-tower architecture with hierarchical shared-specific and fine-grained codebooks that simultaneously capture common patterns and domain-specific differences. Essentially, this is an attempt to separate “what domains have in common” from “what is unique” at the codebook level, rather than at the feature level.

The second part is decoding. Beam search in generative recommendations is slow, and this hinders production deployment. The authors propose a serial-parallel strategy: they use a hierarchical structure of semantic identifiers to partially parallelize generation. The architectural details are not disclosed in the abstract, but the idea is to avoid sacrificing consistency for the sake of speed.

Results on three public datasets show an average accuracy gain of 1.5% and an 85.1% reduction in inference latency compared to SOTA baselines. The first figure seems modest—a 1.5% improvement in recommendations is often within the margin of error. The second result is impressive: an 85% reduction in latency is the difference between “works on paper” and “can be deployed in a real-time pipeline.” The code and datasets are open-source, allowing you to verify the claims using your own data.

The key trade-off here isn’t accuracy versus speed—the gain in accuracy is too small to justify changing the stack. The real value lies in the fact that the generative approach is becoming deployable. Whereas GR used to be a research toy with unpredictable latency, an architectural pattern is now emerging where latency is predictable and controllable. One question remains open: will the 85% accuracy hold up on more challenging domains and when scaling codebooks? Three datasets aren’t enough to be certain, but they do provide a reason to investigate.

Source: cs.AI updates on arXiv.org