Double-blind peer review does not hide the author from the LLM

An anonymous research article is no longer necessarily anonymous to a language model. If an LLM can identify the author based on the title and abstract, double-blind peer review loses its main advantage—protection against bias related to the researcher’s status and affiliation.

The most troubling aspect here is not that the model recognizes writing style. The study’s authors excluded stylistic and bibliographic cues, but the LLM still retained information about how the researcher frames the problem and which research directions they choose. This results in a kind of semantic fingerprint of the author.

In the experiment, the models were given only the titles and abstracts of papers published after their training had been completed. They were then asked to select an author from among five candidates—experts in the relevant field. According to the authors, the LLM was more effective than humans at narrowing down the list of possible authors, with its confidence concentrated on a small number of candidates.

This raises a practical issue. It is not enough for editorial boards to simply remove the author’s name, affiliation, and reference list: the choice of topic and the way the problem is framed can give away the author’s identity. Consequently, anonymity becomes not a property of the manuscript itself, but a race between concealment methods and semantic analysis.

The study’s limitations are significant. It relies solely on titles and abstracts, examines publications after the model has been trained, and tests the selection against five predefined candidates. It cannot yet be directly concluded from this that any LLM will be able to reliably de-anonymize any manuscript in a real-world editorial system. But the implication is quite troubling: if a scientific paper is recognizable by the very logic of how the problem is framed, removing the author’s name no longer solves the problem.

Can peer review remain truly impartial if anonymity must be protected not from people, but from semantic analysis?

Source: cs.CL updates on arXiv.org