LLMs Evaluate Each Other: What the RII Index Actually Measures

Traditional benchmarks for LLMs miss the mark when a question has no single correct answer. If multiple answers are correct, the correctness metric fails to distinguish clear and useful text from confusing text. An article on arXiv proposes a solution: let models evaluate each other blindly, and the agreement among them will serve as a proxy metric for quality. The idea is elegant, but it hides a fundamental trade-off that the authors honestly acknowledge—one that marketing would surely conceal. Five state-of-the-art models generate responses to the same prompts across domains ranging from programming to mathematics and security. Each model then blindly ranks the anonymized responses of its competitors through structured voting. The results are aggregated into the Relative Intelligence Index (RII)—a measure of how often other models prefer the responses of a particular model. The authors observe consistent patterns: some models consistently receive higher rankings from their competitors. This is where the paper’s main contradiction begins. The RII does not measure objective quality or alignment with human evaluations. It measures the extent to which the style and structure of one model’s response align with what other models consider “good.” This is inter-model conformity, not truth. If all five models systematically prefer a certain response format—say, a detailed argument with clear steps—RII will reward that very format, even if a concise and accurate answer would be more useful to a human. The authors explicitly state: the results reflect “inter-model agreement in preferences, rather than objective correctness or human judgment.” They cite previous work showing a partial correlation between aggregated model preferences and human evaluations, but this is a weak basis. Partial correlation means that in some areas, models and humans agree, while in others they disagree, and you don’t know in advance which area you’re in. The practical takeaway for engineers who want to use this framework: RII can be useful as a low-cost, scalable signal for comparative evaluation in open-ended tasks. But relying on it when selecting a model for production means optimizing for the preferences of other LLMs rather than for the needs of your users. The real question the article raises—but doesn’t answer—is: Is it possible to build a metric that combines blind model voting with targeted human checks in areas of divergence? Until such a hybrid approach exists, LLM self-assessment remains an interesting research tool, but not a substitute for human-in-the-loop evaluation.

Source: cs.CL updates on arXiv.org