LLMs vs. Domain-Specific Models: Claude Is More Accurate, DeepSeek Is More Stable, but There’s No Real Alternative

A recent paper on arXiv examines whether LLMs can replace specialized corpora in terminology translation—and reaches an honest conclusion: not yet. But along the way to this conclusion, they uncovered insights that are important not only to translators but to anyone who uses LLMs for professional tasks.

The researchers selected four models—GPT-4o, GPT-5.2, Claude Sonnet 4.5, and DeepSeek—and ran 160 terms through them (80 each in two domains: Earth sciences and natural language processing), translating from English to French. They tested two prompting strategies: “terminological” mode (the model searches for an equivalent term) and “translation” mode (the model translates within context).

The results varied significantly. Claude Sonnet 4.5 performed best under the most favorable configuration. DeepSeek did not take the lead in peak accuracy but proved to be the most stable model—with less variation across domains and strategies. GPT-4o and GPT-5.2 fell somewhere in the middle.

This is a familiar trade-off: the model that performs best under ideal conditions isn’t necessarily the best in production. If you need reproducibility and predictability, DeepSeek is the better choice. If you’re willing to fine-tune the prompt for a specific task and maximize peak performance, go with Claude.

The most practically valuable conclusion of the study isn’t about which model is “better,” but rather how much we can trust a model’s self-assessment. An analysis of confidence estimates showed that a model’s confidence is only a partial indicator of terminological accuracy. A model can be confident and still be wrong. This is nothing new for those who work with LLMs regularly, but it has now been confirmed through measurement on a narrow task where there is an objective criterion for correctness.

The researchers’ conclusion: LLMs are useful as support tools, but they do not replace specialized corpora. Corpora require time, technical skills, and access to data—but they provide something that LLMs do not yet offer: verifiable accuracy. For teams integrating LLMs into their workflows, this means one simple thing: don’t eliminate verifiable sources until the model learns to honestly flag its own errors.

Source: cs.AI updates on arXiv.org