2 min read
In short
RubricForge does not make automatic agent evaluation more accurate on every metric, but it markedly reduces the number of dangerous false successes. We look at why that may matter more for production than overall agreement with a reference.
The main problem with an automatic judge for AI agents is not that it sometimes complains for nothing. It is worse when it takes a failed run for a successful one: a broken agent passes the check and moves on.
RubricForge proposes solving this not with extra model training but with the text of a rubric. The system takes a small sample of trajectories with known ground truth, improves the evaluation criteria through iterative review, and then freezes the finished rubric.
After that the judge checks new trajectories with a single model call and with no access to the environment. An important detail: the output is ordinary text with intelligible criteria. You can see why an agent was judged successful or failed, rather than trusting an opaque weighting setup.
That is where the practical value of the approach lies. On tau-bench, RubricForge cut the share of false passes by roughly a third and a half again: from 0.173 to 0.115. For evaluating agents that matters more than a high overall agreement rate: a false failure usually means another attempt, while a false success can send a broken run into production.
But this is not a blanket win. The advantage over ordinary G-Eval on overall agreement was not statistically significant, and on calibration of absolute scores the generic judge came out slightly better. The results were obtained on tau-bench and WebShop, on comparatively small sets of labelled trajectories; in the experiment one and the same frozen 7B model acted as both agent and judge. Moreover, once built, the rubric no longer consults the environment — its quality depends on how well the original sample was labelled.
So RubricForge is worth treating not as "a smarter judge model" but as a way to tune the check around the most expensive kind of error. For a team that is a concrete choice: optimise average accuracy, or first learn not to let failures through.
Which is more dangerous for your agent: an extra re-run caused by a false failure, or one breakage that the test waved through as a success? Source: cs.AI updates on arXiv.org