A cheap LLM passed all the tests—and failed at the one thing you didn't think to check

The developer of a bilingual news bot found an LLM that is 8 times cheaper than the current one: inclusionai/ling-2.6-flash at $0.010 per million input tokens, compared to $0.098 for deepseek-v4-flash. The quality on the tests is the same, but the speed is 4 times faster. He rolled it out to production. An hour later, a headline appeared in the feed: “How Not to Let Your Smartphone Overheat: Tips for Hot Weather”—with Chinese characters right in the middle of the Russian text, and a Chinese comma instead of a regular one. The model is Chinese, and on long generations, it reverts to its native language more often than we’d like: 57% of long outputs contain leaked CJK characters. For short outputs, the rate is 17%. For deepseek-v4-flash, it’s 0 out of 12. The longer the model generates, the higher the chance that it will forget which language it started with. This isn’t a bug visible in benchmarks. It’s a blind spot in the tests. The author checked the accuracy of categorization and the completeness of the JSON. It didn’t occur to him to verify that the response to a Russian prompt would be in Russian. The free models on OpenRouter are a different story. Out of 17 models priced at $0, only three actually work. One is optimized for code, the second takes 11–21 seconds to respond and sometimes returns an empty response, and the third doesn’t support response_format—OpenRouter silently routes requests past it. With a batch of 60 news items and a 600-second timeout, a model that takes 20 seconds per call is guaranteed to fail the run. Free doesn’t mean it works. The solution for CJK character leaks turned out to be pragmatic: a regular expression detects CJK characters and full-width punctuation; when triggered, the same request is sent to the expensive cleanup model. The cheap model handles most of the news, while the expensive one cleans up after it. In a test run during the first 10 minutes, the detector triggered 25 times, 23 of which were on long analyses. The savings also turned out to be more modest than promised: $8.6 per month versus $14.8 on DeepSeek alone, rather than an eightfold reduction. The reason is that 57% of the long parses incur double costs: first from the cheap model, then from the expensive one. Three financial surprises that aren’t obvious before deployment. The bill doesn’t depend on the number of users: 30 active users account for 0.3% of the costs; the rest is eaten up by the news feed, which runs regardless of the audience. If the balance is negative, OpenRouter stops working even with free models—it reserves credit for the response for everyone. Without an explicit max_tokens setting, the entire model context is reserved: a request for 65,536 tokens, even though the response takes up about a thousand. A single line of "max_tokens": 2048 solves the problem. The main takeaway here isn’t about money. A cheap model can be excellent by every metric you’ve thought to measure, but it breaks in ways you didn’t think to check. If you’re switching to a Chinese LLM to save money—add a response language check to your tests before deployment, not after users see Chinese characters in their news feeds.

Source: All Articles / Machine Learning / Habr