2 min read
In short
A lengthy web search can turn a single minor error into a confident but incorrect answer. SearchAuditor improves the process of finding and correcting such errors, but the results show that automated auditing is still far from being a reliable form of quality control.
The main problem with a lengthy search isn’t just finding the right page, but also figuring out exactly where things went wrong. An error at the beginning of a long chain of reasoning can imperceptibly ruin dozens of subsequent steps, yet the agent will still generate a polished response in the end.
This is exactly what SearchAuditBench tests—a set of 1,243 failed search agent trajectories. On average, each trajectory contains 73.1 messages and 65,100 tokens. For each one, experts identified a critical step, the cause of the failure, and a correction option with evaluation criteria.
Based on this, the authors propose SearchAuditor: a system that examines an error from multiple perspectives, ties conclusions to evidence, and attempts not only to find the cause but also to fix the run.
The result is particularly interesting when compared to other approaches. Even the strongest baseline approach using a state-of-the-art model like GPT-5.5 passed the end-to-end test in only 26.6% of cases. SearchAuditor raised this figure to 32.3%—a noticeable improvement, but not yet at a level where the audit can be considered an automated quality control measure.
The limitations here are more significant than the presentation suggests. We’re talking about 1,243 failed trajectories collected from eight open-weight models across five benchmarks, so the result does not represent the entire market for search agents. Furthermore, even the best approach succeeds in only about one-third of cases. The authors report that, after corrections, the agents recover better and can resume a failed run, but this does not resolve the underlying problem: the system still often struggles to correctly identify exactly what went wrong.
The practical conclusion is simple: for long-running tasks, you need not only an agent that knows how to search, but also a separate layer to verify its intermediate solutions. However, for now, such a layer tends to mitigate damage rather than guarantee the correct answer.
If your search agent were wrong two out of three times, would you add an automatic audit or would you prefer to reduce the task to a few verifiable steps? Source: cs.AI updates on arXiv.org