The gap between expert and automated classifications in MeSH—an artifact of the evaluation design

In systematic reviews, classifiers help filter out thousands of abstracts to retain only the relevant ones. The input is often supplemented with MeSH terms—either from experts (with a delay of several weeks) or from automated tools (immediately). Until now, these methods have not been directly compared as features, and no one has asked whether the results depend on how the classifier is evaluated.

The authors used the Cohen et al. (2006) benchmark across three topics—Statins, Opioids, and ADHD—and ran a bag-of-words logistic regression (seven runs) and BiomedBERT (five seeds). They then began to vary the evaluation design and observed how the gap between expert and automated MeSH classifications changed for the “Statins” topic.

The results speak for themselves. With a canonical 5-fold split on the full corpus, the “expert minus automated” gap is +0.096 WSS@95%. Once the corpus size was scaled down to smaller topics (n = 803), the gap narrowed to +0.033, and the 95% bootstrap confidence interval included zero. With 10-fold cross-validation on the full dataset, the gap is +0.021, and the interval barely crosses zero. BiomedBERT yields +0.020—within the margin of error relative to the 10-fold bag-of-words result.

In other words, the “significant advantage of expert MeSH” is largely an artifact of the specific evaluation scheme. A reasonable design adjustment eliminates 80% of the gap.

Another nuance: 15.1% of the “Statins” inputs exceed BiomedBERT’s 512-token limit when expert MeSH terms are added to them. This truncation may explain why the gap is smaller for the Transformer—but it cannot be separated from the volume of training data in this experiment.

Power analysis shows that the Statins size effect could not have been detected with the variance seen in Opioids or ADHD. Null results on these topics are not informative but are limited by the design. This is an important point: “no difference” and “insufficient power to detect a difference” are different things, and benchmarks routinely confuse them.

A practical takeaway for those building screening pipelines: when using transformers or 10-fold bag-of-words, the gap between expert and automatic MeSH on the tested topics is approximately 0.02 WSS@95%, with confidence intervals crossing zero on at least one boundary. It may not be worth chasing expert annotation for the sake of the classifier if automatic annotation is readily available.

A broader lesson: benchmark conclusions about feature sources can change significantly with reasonable modifications to the evaluation design. If your experiment shows a “significant” effect, check whether it holds up under an alternative cross-validation scheme and whether your sample size is adequate.

Source: cs.CL updates on arXiv.org