ML reviewers should be motivated by credits, not by appeals

Good reviews in machine learning will not appear from yet another polite instruction to reviewers. The authors of a position paper argue that conferences need real incentives: a contribution to reviewing should turn into a resource that can be spent.

The idea resembles an internal currency for academic conferences. A researcher earns OpenReview Points for good reviewing practice and then spends them at one or several major conferences.

The points could be spent on various benefits: free registration, say, or a request for additional reviewing resources for one's own paper. That changes the logic of the system itself: reviewing stops being a plea to "help the community" and becomes a condition of access to some of its opportunities.

The second element of the proposal matters too — not only credits but executable, sufficiently precise procedural rules. A currency alone will not repair a bad system: if it is unclear what counts as a good review, points easily turn into a reward for quantity rather than quality. Rules are needed that limit abuse and account for genuine contribution.

That said, what we have is neither a working service nor the result of an experiment but a position paper. The authors discuss two main questions — how to limit the flow of submissions and how to reward good reviews while discouraging bad ones — and propose two alternative designs. But the abstract does not show how quality will actually be measured, who will settle disputed cases, or how resistant the system will be to gaming.

The practical conclusion is simple: if conferences want stronger reviews, they will have to pay for them — not necessarily in money, but at least in intelligible and checkable benefits. What incentive would make you write a genuinely thorough review — a bonus in access, or the risk of losing points?

Source: cs.AI updates on arXiv.org