An AI agent doesn't understand code: why model confidence isn't proof

The main problem with AI agents in large codebases isn’t the speed or quality of generating a single function. The problem is that the model confidently infers relationships between components without any evidence. Changing a single method can break a backend route, a frontend component, or a background task—and the agent doesn’t even suspect it.

The CodeSlicer offers an approach that I believe is fundamentally sound: don’t treat an AI’s assumption as a confirmed relationship. The tool works as a local CLI, an MCP server, and a visual analyzer that builds a project’s influence graph—from functions and classes to DI providers, HTTP endpoints, frontend components, and tests.

The key difference from conventional code graphs is the preservation of the evidence chain and provenance for each edge. The system doesn’t just say, “These two functions are related.” It shows why it considers them related, with what level of confidence, and on what basis. This allows the agent to work with code where some dependencies are confirmed by static analysis, some by runtime observations, and others are marked as “unknown regions” requiring manual verification.

The pipeline includes inventory, extraction, semantic resolution, support packs, mutation testing, and runtime observation. Mutation testing here is not an academic add-on, but a means of verification: if breaking an edge doesn’t break anything, then the connection was false.

The numbers for the Python scenarios are impressive: 21 test scenarios, 29 mutation scenarios, 20 required semantic edges—and 0 false positives, 0 false negatives. For TypeScript and the frontend-backend bridge, 12 scenarios, 15 mutations, and 4 cross-language chains were tested with an endpoint precision of 1.0. It sounds almost too good to be true, but the mutation testing methodology gives us reason to trust the results—at least for the labeled scenarios.

Integration with AI agents is done via CLI, MCP, and skills. This means that the agent does not receive raw code to parse, but rather a pre-structured graph with metadata on the reliability of each connection. Architecturally, this shifts the distribution of responsibility: the model is responsible for generation, while the tool is responsible for verifying the impact.

The main question that remains open is scalability to real production codebases. Annotated scenarios represent a controlled environment. But the very principle of distinguishing between “assumption ≠ confirmed edge” is precisely the trade-off that many agent-based frameworks lack—frameworks that are quick to pass off a probabilistic guess as fact.

Source: All Articles in a Row / Artificial Intelligence / Habr