How to Keep Bad Papers Out of Conferences (with Minimum Reviewer Effort)

Abstract

Reviewing conference submissions is both labour-intensive and diffuse. A lack of focus leads to reviewers spending much of their scarce time on papers which will not be accepted, which can prevent them from identifying several classes of problems with papers that will be. We identify opportunities for automation in the review process and propose protocols which allow human reviewers to better focus their limited time and attention, making it easier to select only the best “genetic” material to incorporate into their conference’s “DNA.” Some of the protocols that we propose are difficult to “game” without uneconomic investment on the part of the attacker, and successfully attacking others requires attackers to provide a positive social benefit to the wider research community.