Vector Search Breaks the Agent's Memory: Why Semantic Proximity ≠ Identity

When a multimodal agent works with a long video, it needs memory—otherwise, it won’t be able to link the person in the first frame to the same person half an hour later. The standard approach: compress segments, store them in a vector database, and search by similarity. The problem is that this approach systematically misrepresents the identity of entities.

Vector similarity captures semantically similar fragments, but does not guarantee that they are the same object or character. Aggressive compression and frame-by-frame slicing only make matters worse: subtle signs of identity are erased even before recording. The result is mixed-up entities, a cascade of errors, and hallucinations that cannot be traced back to a specific memory glitch.

The ViSAGE framework offers three mechanisms, each of which targets a specific weak point.

Cross-modal binding anchors an entity’s identity to different modalities over long time intervals—rather than relying on pixel-by-pixel or textual similarity at a single moment.

Bidirectional memory updating is the most interesting feature. When evidence of identity arrives with a delay (which is common in long videos), ViSAGE doesn’t just update the current context—it retroactively rewrites historical records. The past is corrected in light of new data, and future inferences are based on this consolidated picture.

Multi-agent cross-validation adds a constraint on evidence alignment: if a piece of evidence does not confirm identity, the agent does not make a guess but refuses to respond. For production use, this is more important than any level of accuracy—the ability to say “I don’t know” instead of producing a confident hallucination.

In experiments, ViSAGE outperforms the strongest baseline by 5.9% in accuracy. But the number is secondary. The main point is to demonstrate that vector search, as a universal memory mechanism for agents, has a structural flaw: it is optimized for similarity rather than identity. For short-term tasks, this goes unnoticed; for long-term horizons, it is critical.

The practical takeaway for those building agents with long-term memory is this: if entities can recur or evolve over time, pure RAG is insufficient. You need either retrospective correction of records or an explicit identity model with the ability to “abstain.” Otherwise, the agent will consistently confuse objects—and you’ll spend weeks debugging hallucinations caused not by the model, but by the memory architecture.

Source: cs.AI updates on arXiv.org