Why 70% of Political News Became "Neutral" for RoBERTa

Just because a model classifies 70% of political articles as neutral doesn’t mean they lack a stance. Perhaps the problem lies in the measurement method itself: a binary “positive–negative” scale fails to capture ideology, rhetoric, and nuances of emphasis.

This is precisely what the study’s authors refer to as “neutral collapse”—the reduction of substantive political text to an analytically useless category. They compared RoBERTa’s tone analysis with an LLM platform that considers several dimensions at once: the direction and intensity of political bias, sensationalism, emotional presentation, and framing.

A telling result: among the articles that RoBERTa labeled as neutral, 23% had a probability of negative tone higher than 0.30. In other words, the “neutral” label sometimes masks a noticeable signal that the model is unable to classify as either positive or negative.

The practical conclusion here is not that LLMs automatically understand news better. Rather, a single tone is insufficient for political analysis: what matters is not only the emotional sign of the text, but also whom it holds accountable, what threats it emphasizes, and the frame in which it places the event. For researchers, this means a shift from a single label to a set of interpretable characteristics.

However, the approach still has limitations. The study used a corpus of only 50 political articles from 17 international media outlets, so the results cannot be automatically generalized to all languages, genres, and editorial systems. Furthermore, the LLM’s multidimensional output is not an objective measure of political bias: it, too, depends on the task formulation, evaluation criteria, and the model’s ability to distinguish context. The authors demonstrate the shortcomings of traditional SA but do not prove that any LLM framework solves the problem.

If you’re analyzing news using a single tone scale, which is more important to you: a convenient, unified metric, or the risk of losing the political meaning of the text? Source: cs.CL updates on arXiv.org