LLM for Wastewater Treatment Plants: 99.5% Accuracy Isn't the Most Practical Option

In an industrial application, a model may perform well in a laboratory test but still prove impractical for a real-world enterprise. In this study, the best result—99.5%—was achieved through direct access to a running simulator, but another approach appears to offer the most practical compromise.

The goal was not for the model to simply restate general information about wastewater treatment. It needed to explain why N2O levels are rising, or predict what would happen after reducing aeration by 20%—taking into account the interrelationships between variables and the rate at which the effect spreads.

The researchers left Qwen2.5-32B-Instruct frozen and compared three methods for grounding its responses in the CCSS-IX simulator:

• live simulator prompts—99.5% on the causal benchmark consisting of 198 questions;
• structured parameter injection—79%;
• a DRR retriever, which extracts numerical parameters for reasoning—75.8%.

For comparison, the best retrieval-augmented baseline scored 48%. At first glance, the conclusion is obvious: connect the simulator directly and get a nearly perfect result. But this approach requires an accessible and properly configured simulator. If one isn’t readily available, a high score becomes a demonstration of potential rather than a transferable solution.

DRR scored lower on the original test, but after transfer to a biologically distinct installation, it maintained 88%. The static parameter table did not survive the transfer. This is an important distinction: industrial systems vary from site to site, and versatility can be more valuable here than a few extra percentage points on familiar data.

Even more interesting are the results for the 60 counterfactual questions—that is, questions about the consequences of an intervention. Only Method 3 handled queries of this type; it outperformed Method 2 by 16.3 percentage points, with a 95% confidence interval ranging from 7.1 to 26.4 points. In categories related to time scale and operating mode, DRR achieved 100%.

The authors tested the idea outside the context of wastewater treatment: on ARC with the OpenBookQA fact corpus, the selective retriever achieved 79% accuracy, compared to 76% for Llama-3.1-8B without restrictions and 74% with full data injection. This does not prove the method’s universality, but it shows that the performance gain is not necessarily tied solely to a specific simulator.

There are also significant limitations. A live simulator requires infrastructure, a static table cannot be transferred between installations, and DRR is a separate model with 110 million parameters that takes approximately 17 seconds to train for each site. Furthermore, the results were obtained on benchmarks consisting of 198 and 60 questions, not in the described field test involving real operators.

My conclusion is simple: for industrial AI, you shouldn’t choose a solution based solely on maximum accuracy. If a model needs to answer “what-if” questions and work across different sites, a slightly less accurate but transferable parameter extraction layer may prove more useful than direct access to a perfect simulator.

In your task, would you choose maximum accuracy at a single site or a less accurate model that can be transferred to others? Source: cs.CL updates on arXiv.org