Why a jury of LLMs proved stronger than one expensive model
Sh0ny
15 августа 20262 min read
In short
A single model spots errors in long chains of reasoning poorly, even a frontier one. Reasoning Jury proposes several open-weight models with a moderator instead — and claims higher accuracy at 8–15% of the cost of ordinary checking.
Reasoning Jury's main effect is not that there are now more models. It is that checking reasoning has been turned from a solitary verdict into a discussion where models can criticise and revise each other's decisions.
That matters for tasks where the answer alone proves nothing. You need to know where exactly the model went wrong in a long chain of reasoning, how serious the error is, and whether that analysis can be used for training.
In the system several LLMs first deliver their own verdicts. A moderator then organises a discussion: the jury compares assessments, the participants criticise each other's arguments, and after that they may change their initial vote. The outcome is an agreed decision on the defect and its severity.
The practical consequence is that the checker need not be the most expensive model in the system. In the experiments described, a jury of open-weight models, including an example with gpt-oss-120b, found errors in reasoning more accurately than opus-4.6, sonnet-4.6 and gemini-3.1-pro. Such an approach can also get round restrictions on using frontier models in online training.
The price of the scheme is the extra model calls and the discussion procedure itself. But the authors claim the combined cost of verdicts, deliberation and aggregating the results comes to only 8–15% of the cost of checking with a single frontier model. That is no longer merely a way to improve the quality of evaluation but potentially a cheaper architecture for supervising reasoning models.
The limitations are substantial too: the available description gives no detail on the jury's composition, on specific benchmarks, on absolute accuracy values, or on how well the method transfers to other tasks. So for now this is a strong result from the stated experiment, not proof that any assembly of models automatically beats one good solitary judge.
If you had to choose how to check AI answers, which would matter more: one maximally powerful model, or several more affordable ones with a mandatory discussion? Source: cs.AI updates on arXiv.org