The memory of LLM agents loses preferences after significant events

LLM agents currently struggle with the very task for which they need long-term memory in the first place: recognizing that a user’s preferences have changed. In the FinPerMA benchmark, even simply retrieving relevant records after significant events sometimes outperforms specialized memory systems.

This is an important distinction. It is easier to remember that a user owns a certain asset than to deduce from a new event how the user’s profile and recommendations should now be adjusted. In financial advising, such an error is more dangerous than a typical failure in recall: an advisor may remember the correct facts, but applying them may no longer be appropriate.

What Exactly Does FinPerMA Test?

The authors built a test using frozen longitudinal investor trajectories. These include personal data, events, and a “Post-Shock” checkpoint—the moment when it is necessary to verify whether the agent has incorporated a material event into the user’s persistent model.

The generation process combines theory-based deterministic rules, controlled LLM storytelling, and automated quality checks. This makes the test less reliant on random, weakly constrained trajectories, which are often used in similar evaluations.

The experiment involved 2,994 questions and 276 participants. The authors tested seven state-of-the-art models and up to seven memory configurations. No configuration with full context exceeded approximately 0.47 overall accuracy, and on multiple-choice questions, the result remained around 39%.

The Main Shortcoming Is Not in the Facts

Attribution analysis shows that summarization often preserves factual details but discards preference cues. For a typical chatbot, this might appear to be successful story compression. For a personal agent, however, it amounts to the loss of the most critical layer of context.

Simple retrieval can therefore outperform specially designed memory systems. After “shocking” events, the gap widens: it’s not enough for the agent to find an old record; it needs to understand which parts of the user model are now outdated and which are not.

The practical takeaway for developers is quite down-to-earth: an agent’s memory cannot be evaluated solely based on how well it answers questions about the past. We need scenarios where the user model must be updated after an event, and the result must be tested for stability after some time has passed.

FinPerMA does not prove that a specific memory architecture is always inferior to retrieval. But it clearly highlights a weakness in current approaches: they optimize context retention, even though personalization also requires controlled revision of inferences. Until this is resolved, an agent’s “long-term memory” remains more of an archive than a user model.

Source: cs.AI updates on arXiv.org