75% vs. 8.3%: Why Agents Fail at Coherent Visualizations

LLMs are excellent at generating individual charts, but coherent multi-view visualizations—where views exchange data and respond to each other’s interactions—remain a stumbling block for them. The reason isn’t the quality of the model, but rather that an error in one component silently breaks the others, and the end-to-end agent can’t catch it in time. A paper Crystalis shows that structured decomposition solves the problem radically: 75% end-to-end success versus 8.3% for an agent using the same model. This ninefold gap isn’t due to some magic in the prompt, but rather a consequence of architectural choices. Crystalis does not attempt to extract analytical quality from the LLM, as this depends on domain knowledge and user expertise. Instead, the authors pose a more fundamental question: can the model at least structurally correctly assemble a CMV, and what abstractions are needed to do so? The answer is a query-centric model, where visualization is broken down into structured queries over a dependency graph. The graph covers three types of components (Data, Visualization, Interaction) and three levels of abstraction (requirement, specification, executable object). Two mechanisms operate on top of this structure. Progressive nucleation assembles each query vertically—from the requirement to the executable object—in the order of dependencies. Semantic annealing checks for horizontal consistency between queries at each level through layered logical checks. Essentially, it’s layer-by-layer assembly with checkpoints, rather than a single large-scale generation. The benchmark consists of 12 tasks and five frontier models. Result: up to 75% end-to-end success. The agent-based coding baseline on the same model achieved 8.3%. A user study involving 12 practitioners confirms that decomposition and iterative refinement are convenient to work with. The conclusion for practitioners is clear: when system components are tightly coupled and an error in one component cascades to break the rest, an end-to-end agent is not the best choice. Structured decomposition with an explicit dependency graph and layer-by-layer validation works an order of magnitude better. One question remains open: does this approach scale beyond visualizations—to arbitrary multi-component systems where coupling between parts is high?

Source: cs.AI updates on arXiv.org