GraphRAG cites more sources, but it doesn't always provide better answers

GraphRAG has a problem that persists regardless of the embedding model or corpus used: it systematically overloads responses with citations. However, the consequences of this behavior depend on the data—for one type of document, response accuracy drops sharply, while for another, the extra references hardly interfere at all.

The authors tested three combinations simultaneously: two embedders—ranging from the local e5-small to Azure text-embedding-3-small—two corpora, and two LLM evaluators—GPT-5.4 and GPT-4.1. In total, there were 4,440 runs in the main matrix, 600 cross-corpus runs, and 1,200 paired assessments of how well the answers matched their sources.

The results for citations proved stable: GraphRAG returned between 11 and 15 source identifiers per response. At the same time, citation precision remained at 0.12–0.23, and recall at 0.68–0.87. In other words, the system often found the right material but included too many references in its response, including those with weak relevance.

This is where an important caveat comes into play. In the DO-178C requirements corpus, where relationships between entities are typized, GraphRAG’s faithfulness dropped from 74% to 40% when moving between hops. On the Wikipedia paragraph chains from MuSiQue, the opposite occurred: the metric increased from 42% to 58%, because the extra paragraphs were still thematically relevant.

Hence the practical conclusion: there is no clear winner among RAG methods. On two-step questions in the DO-178C corpus, the conventional approach performed better, while in MuSiQue, GraphRAG performed better; this preference held true when using both embedders. In other words, robustness to changes in the embedder does not necessarily imply robustness of the method in general—the corpus itself may be the decisive factor.

There is also a separate issue with evaluation. The same LLM evaluator could change its verdict when the retrieval state changed: the self-kappa for GPT-5.4 across embedders was only 0.137, and the decision changed in 41% of cases. The authors also trained a router that classified the number of hops based on dense embeddings with a macro-F1 score of 0.86, but this does not eliminate the need to verify the evaluation methodology itself.

The limitations here are significant: the findings were obtained using two corpora, two embedders, and a pair of evaluators, rather than across all possible RAG systems and datasets. Therefore, the paper does not prove that GraphRAG is bad, but rather that claims of its superiority without testing across different corpora and evaluators are too fragile.

If you’re choosing RAG for your product, are you willing to consider the system superior based on a single successful test on a single dataset? Source: cs.CL updates on arXiv.org