AgentLens: A Benchmark for Evaluating the Behavior of Code Agents
Sh0ny
9 июля 20261 min read
In short
Researchers have introduced AgentLens—a benchmark for interactive code agents that evaluates not only the final result but also the entire workflow. The tool combines formal verification with LLM reviews and is available as open-source software.
Most existing benchmarks for code agents reduce evaluation to a binary outcome—whether the task was completed or not. However, real users experience the agent’s entire workflow: how it follows instructions, uses tools, checks its own actions, recovers from errors, and interacts with humans. The AgentLens project proposes evaluating this entire trajectory.
Evaluation Approach
The benchmark combines several methods:
- formal verification—used where there is an objective criterion for success;
- LLM trajectory reviews — textual feedback on the progress of task execution;
- side-by-side comparison — parallel analysis of different runs.
Each run provides a readable explanation of why a specific score was obtained. According to the authors, this makes AgentLens useful not only for ranking models, but also for diagnosing behavior, comparing successive versions of one’s own agent, and tracking regressions in the nightly pipeline.
The benchmark’s source code is publicly available in the agent-lens-bench repository.
Source: cs.AI updates on arXiv.org