Why the "Correct" RAG Fails on Real-World Data
Sh0ny
27 июля 20261 min read
In short
BM25 and the re-ranker are industrial defaults that can actually lower the metric in practice. A case study showing that the real work wasn’t the arrows on the diagram, but rather taking measurements and cleaning the housing.
The industrial RAG recipe sounds simple: a vector database, an embedder, a re-ranker, hybrid search with BM25—and the system is ready for production. Twenty lines of code, a nice diagram. But on real data, almost every “correct” step on this list can actually make things worse.
BM25 caused the metric to drop. The re-ranker caused the metric to drop—twice, in two different configurations. The only thing that actually helped turned out to be not the code, but several days of tinkering with the data and running tests.
This doesn’t mean that RAG doesn’t work. RAG works. The problem lies elsewhere: the real work begins where copying default recipes ends. An evaluation harness, corpus cleaning, and honest metrics show which elements of industry benchmarks work for you and which ones hurt you. You can only verify this using your own data—“that’s just how it’s done” isn’t a valid argument here.
If you’re building a RAG system, the first question isn’t “which embedder should I use?” but “how will I measure whether things have improved?”