A Benchmark for AI Agents Using Real CVEs: Why It Matters
Sh0ny
23 июля 20262 min read
Updated 25 июля 2026In short
The cve-bench project proposes evaluating AI agents on the task of fixing real vulnerabilities in Python. I’ll explain why benchmarks based on synthetic tasks mislead everyone and what will change when the metrics become more accurate.
An open-source project called cve-bench has appeared on GitHub—a benchmark for evaluating AI agents on the task of fixing real vulnerabilities in Python code. The author is asking the community for suggestions on which CVEs to add to the task set. At first glance, this seems like just another tool for measuring LLM progress. But behind it lies a much more interesting question: how can we actually tell if an agent is capable of fixing security issues, rather than just guessing the answer based on a template?
The problem with synthetic benchmarks is well known to those who work with agents in production. A model that excels on HumanEval or SWE-bench is often helpless when faced with real-world legacy code, where a vulnerability is spread across five files and the correct fix breaks three tests. Synthetic tasks are contaminated—their solutions are already present in the training dataset, and metrics improve not because the model has become smarter, but because it has learned the distribution. CVE tasks disrupt this focus: each vulnerability has a specific historical context, a specific patch, and specific regression tests. It’s impossible to predict these in advance.
The project currently looks like an early framework: a Docker environment, a harness for running agents, a task template, and a folder with results. This is an honest approach—instead of providing ready-made numbers, the author offers a tool and invites the community to increase the complexity. If the benchmark reaches a critical mass of real CVEs, it will provide the first relatively reliable way to distinguish agents that truly understand the code from those that merely paraphrase the documentation.
The main question isn’t which CVEs to add, but who will take on the task of evaluating the quality of the patches themselves. Fixing a vulnerability without breaking compatibility or introducing a new one is a challenge where people regularly make mistakes. If the benchmark only checks whether “the tests pass,” it will quickly become just another target for gaming. If it adds checks for semantics and regressions, it will become the standard.