An LLM judge can be accurate and still err easily

The trouble with LLM judges is not necessarily that they often get it wrong first time. Something else is worse: one and the same verdict can change if you rephrase the request or argue with the model persistently.

The authors call this "wiggle" and propose the Wiggle Framework. It splits a judge's robustness into three levels: stability under a repeated request and paraphrasing, the reaction to a single objection, and the ability to hold a position under sustained or adaptive pressure.

Across 9 frontier models and 14 tasks — from safety and toxicity to detecting AI text and assessing political answers — wiggle was found in every model. Under ordinary pressure the judges changed their verdict in 25–71% of cases. When another adversarial LLM was assigned to do the persuading, the figure rose to 62–91%.

The most unwelcome result: pressure that changed a decision almost always moved it further from the correct answer. That is, pliability is not a useful capacity to revise a mistaken opinion but mainly a way to spoil an assessment already made.

The practical conclusion is simple: accuracy on a gold set is not enough to validate an LLM judge. Repeated runs, paraphrasing and attempts to contest the verdict are worth adding to the evaluation system. The strength of the jury's original majority, incidentally, proved the best single signal of which examples are more likely to start "floating".

The work's limitation is that it shows model behaviour across 14 tasks and given pressure scenarios; it does not prove the same instability in every workflow. Besides, a high rate of changed answers does not by itself say how often the model could have corrected an error on its own without outside pressure.

If your LLM judge changes its decision after a persistent argument, do you count that as a common-sense check or already a signal that the evaluation system cannot be trusted?

Source: cs.AI updates on arXiv.org