The main obstacle for AI agents may not lie in the model, but in the data

An AI agent may make confident mistakes not because it lacks intelligence, but because it does not understand exactly what a particular field in the data means. Descriptions such as “customer status” or “type of inquiry” often lack the rules, exceptions, and local context that are obvious to employees within the company.

It is precisely this gap that the ISEE (Interactive Semantic Enrichment System) addresses. Its core premise is more important than the system itself: the quality of an agent’s performance is limited not only by the model but also by the quality of the semantics available to that model.

Semantics Missing from Documentation

In application data, a significant portion of the meaning resides in users’ minds and is rarely published or formally documented. For humans, this isn’t always a problem: they know what a custom field means and which values to consider equivalent. For an agent, however, this lack of explicit information leads to errors in analysis, search, and tasks such as entity linking.

ISEE doesn’t simply automatically rewrite a field’s description. The system first evaluates its quality on its own scale, then gathers domain knowledge from users and works with them to enrich the description. This is a key difference from the “let the LLM figure it out on its own” approach: unknown context is not generated but is extracted through interaction with humans.

The authors tested the system in several ways: through user research, automated user simulation, quantitative evaluation, and case studies. According to their data, ISEE reduces cognitive load, improves the quality of descriptions, and enhances the results of subsequent tasks. However, the available description of the work does not include numerical results, so the scale of the improvements cannot be assessed.

The practical takeaway for teams building agents on top of corporate data is quite straightforward: before adding another model or complicating the prompt chain, it’s worth checking how unambiguously the fields themselves are described. If the meaning has to be reconstructed from verbal agreements, the agent needs not only access to the table but also a process for gathering this context.

ISEE appears to be useful precisely as such a process—with quality measurement and expert involvement. But this does not negate the main limitation: the system helps capture users’ knowledge, not replace it. Where no one can explain the rules of a field, there will be nothing to improve the description with.

Source: cs.AI updates on arXiv.org