You can't generate a good report from a table with just one prompt

Generating a report from a table usually follows a pipeline-like process: the model generates text, creates charts, and draws conclusions. This approach is convenient for a prototype, but it fails to address a simple question: what should you do if a visually appealing graph contradicts the numbers in the table, while the text confidently explains an erroneous conclusion?

The preprint MCTS-Report suggests not writing the report in a single pass, but rather assembling it using Monte Carlo Tree Search. The system treats report generation as a sequence of actions: planning chapters, defining visualization tasks, building charts, organizing insights, and editing the narrative. After each step, it has a current state of the report, on which subsequent decisions depend.

This is a significant shift in architecture. Here, the LLM doesn’t simply output a final answer but participates in exploring the space of possibilities. The reasoning path is stored in the tree nodes, so subsequent actions receive the context of the report already created, rather than just the initial table.

But the search itself does not solve the problem automatically. Its quality is determined by how the system evaluates intermediate results. The authors use a composite reward function: they verify numerical facts via SQL, separately evaluate the quality of the graph, the consistency of the graph with the text, and the completeness of the structure. Additionally, penalties are introduced for duplicate graphs and precondition checks to filter out invalid actions.

This is where the most practical idea of the work lies: for analytical reports, it is not enough to evaluate the “clarity” of the text. Independent quality signals are needed to catch different classes of errors. SQL can verify a number, but it cannot tell whether the chosen visualization type is appropriate. Evaluating a chart does not guarantee that the accompanying paragraph actually describes it. Separating the criteria allows us to seek a compromise rather than masking all problems with a single final rating.

To test this, the authors compiled MMRBench—a set of real-world tables from six domains with reference report structures and verifiable key conclusions. In the experiments, MCTS-Report achieved an overall score of 77.9 and outperformed the claimed strong baseline methods in terms of structural completeness, numerical accuracy, alignment of graphs with text, and novelty of insights.

This figure should be viewed with a degree of caution: the available description lacks details on the composition of the baseline methods, the variation in results, and the contribution of individual system components. Therefore, 77.9 is an argument in favor of this approach, but not proof that any analytical report should now be built using MCTS.

The practical conclusion is different. If you’re designing an agent to work with data, it’s more useful to start not with the prompt “make a nice report,” but with a tree of permissible actions and checks after each one. Such an agent is more complex and likely more expensive than a linear pipeline, but its errors can be diagnosed: an incorrect number, a poor graph, a duplicate visualization, or a mismatch between the image and the text—these are different types of failures that require different checks.

Source: cs.AI updates on arXiv.org