An Architecture for LLM Agents That Captures Failures, Not Just Successes

LLM agents write code and conduct research, but their memory is reset after each session. RAG based on raw sources isn’t helpful: it doesn’t retain context. As a result, teams endlessly repeat the same mistakes because dead-end branches and discarded hypotheses aren’t preserved—neither in publications nor in repositories.

The llm-wiki-memory-template architecture offers a solution: an LLM-powered, linked wiki between raw sources and the agent. The key idea is “append-only” by default. The wiki stores not only what worked but also the failures. This solves the problem of lost negative results, which publications and code sharing structurally fail to address.

The template is designed as a substrate for collaboration in three dimensions: “human-human,” “human-AI,” “AI-AI.” Each layer is supported by a separate architectural element.

The authors tested the approach on three deployed use cases and one design. The most illustrative example is a project by two of the authors. A retrospective audit revised the reported accuracy of two previous experiments: from the reported 20 out of 20 to 14 and 12 based on evidence, and then to 18 and 18 after correction. The error path was preserved in the artifact.

The authors’ main thesis is that preserving failure paths, agent honesty, and appropriation are end-to-end socio-technical properties of the artifact. These are not merely technical mechanisms.

For practitioners, the conclusion is clear: if your agents do not remember what they have already tried and why it did not work, you end up paying for the same mistakes over and over. This pattern is worth studying for those building pipelines with long-term memory for agents.

Source: cs.AI updates on arXiv.org