Reviewer Anonymity under Dishonest Behavior, and Associated Tradeoffs
Abstract
A cornerstone of blinded peer review is reviewer anonymity - confidentiality of who has reviewed which paper - which allows reviewer,s to freely provide their opinions without fear of retribution. However, there is anecdotal evidence that this anonymity is being co, of a reviewer to the authors. Such a compromise is detrimental to the functioning and reliability of the peer-review process, where, dishonest (powerful) actors can unethically influence the decisions of their own grant proposals or papers. We consider two approac,hes towards ensuring anonymity. The first approa,ants such as area chairs. In the absence of any systematic studies on this topic, the blinding policies are based only on the percep,tions of program chairs.There are various tradeoffs in blinding versus not blinding reviewers. First, the knowledge of identities of,d, anonymity might reduce the politeness of discussions. The fourth tradeoff pertains to usefulness to the stakeholders in terms of,the perceived utility in making decisions as well as logistical considerations. This research will help take a scientific approach t,owards making policy choices on blinding reviewers. It will involve the design and execution of experiments to characterize the afor,ementioned tradeoffs. It will also conduct a survey of researchers to understand their perceptions and experiences regarding the pro,blem of anonymity compromise. Even if reviewers are blinded to each other and to the area chairs, information about reviewers may le,ak in various ways. The reviewer-paper similarities for assignment are often computed by a third party or otherwise via open-source,code. This allows other entities to know the text-matching similarities between every reviewer-paper pair, which may be used to gues,s the reviewers for any paper. With this issue in mind, our second approach uses randomization of the assignment of reviewers to pap,ers to ensure anonymity even if the similarities are known to an adversary. A recent work by PI Shah and collaborators designed such, randomized assignment schemes. The algorithms are designed with a motivation of mitigating collusion-ring-based frauds. This resear,ch will build on this past work to design randomized algorithms help the objective of anonymity, and also improve upon these previou,sly designed algorithms. This research will then then conduct experiments to characterize the tradeoffs associated to the use of ran,domness in the assignment process.If successful, this project will have impact the peer-review process in three ways. First, the pro,blem of compromise of reviewer anonymity is only known from anecdotes. This survey will help understand the extent of this problem,,and if it does exist broadly, will help bring it to the forefront of community discussions and policy design. Second, it will scient,ifically inform the many debates already prevailing in the community regarding blinding reviewers to each other, associated to the p,roblem of compromise of anonymity as well as more broadly due to a number of other reasons. It will allow program chairs to make evi,dence-based decisions on their policies. Moreover, characterizing these tradeoffs will also help understand the value of the discuss,ion process itself, potentially shedding light on certain intriguing phenomena that have been discovered previously. Third, the rand,omized algorithms we design and evaluate here will form a useful tool for program chairs of conferences and program managers of gran,t peer review for imparting anonymity. They will also help mitigate the issues of coalition-ring-based fraud and torpedo reviewing.A
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Mar 05, 2022
- Source ID
- N000142212181
Entities
People
- Nihar Shah
Organizations
- Carnegie Mellon University
- Office of Naval Research
- United States Navy