Before scientific literature is accepted, it has to undergo the peer review process. For an explainer of how that process works, please read here.
Typically, the peer review process is a "single-blind" one, meaning that the reviewers are aware of the identity of the authors of the paper and the institution where they work. However, the authors of the paper are not privy to who reviewed their paper.
Peer review could also, theoretically, be "double-blind." In that case, neither the authors nor the reviewers would know the identity of the other.
The question is, does it matter whether reviews are done single or double blind? More specifically, does knowing the identity of the authors influence the opinion of the reviewers?
A team at Google sought the answer to that question when they were asked to co-chair the program of 10th Association for Computing Machinery International Conference on Web Search and Data Mining (WSDM) 2017. At the time, they were asked to consider a double-blind review process. When they looked for the research that was available on the topic, they found that there was none. So, they decided to study it themselves.
Because research in the computer science field frequently appears in peer-reviewed conferences before journals, the authors chose to test the question of single vs. double blind with the submissions to the conference (only about 15-20% are chosen.)
The experiment was set up to test the hypothesis that there is a difference between single and double blind reviews. A committee was randomly split into two groups - a single-blind program committee (SBPC) and a double-blind program committee (DBPC). The SBPC sees author names and affiliations, while the DBPC does not. Each paper submission received four reviews - two from the SBPC and two from the DBPC. The reviews were then analyzed and compared.
In their analysis, the authors looked at the following factors -
- Is the paper from an academic institution?
- Is at least one of the paper’s authors is female?
- Is the paper from the United States?
- Is there a famous author included? (a famous author has at least 3 accepted papers at earlier WSDM conferences and at least 100 papers according to a commonly used computer science bibliography.)
- Is the author from the same country as the reviewer?
- Is it from a top university? (The top 50 global computer science universities per www.topuniversities.com)
- Is it from a top company? (Google, Microsoft, Yahoo!, and Facebook)
The authors of the study found three main differences between single-blind and double-blind reviewing.
First, single-blind reviewers were more selective in their choices of the papers to select (see figure below). Second, single-blind reviewers chose more papers from top universities and top companies, compared with their double-blind counterparts. Third, single-blind reviewers are more likely than double-blind reviewers to submit a positive review for papers with a famous author and for papers from a top university or a top company.
Cumulative distribution function of number of bids for single- and double-blind reviewers.
To bottom line is that, as could have been predicted, when the reviewers are privy to who did the work and where, it influences their decision. Moreover, they are more likely to select paper from top institutions and famous authors than reviewers who are blinded.
So, where does this leave us? Should science switch to double blind reviewing? This type of study should be the start of a very important conversation.
If two papers are submitted that are of equal quality. One from a new scientist at a little known school and one from a big-wig with a big name behind them why shouldn't they have an equal chance at being selected?
If you are a top researcher or work at a top institution, your answer is likely to differ from the rest of the scientific community. But, science is a world where the rich get richer, with more papers leading to more grant money and vice versa, while the new investigators need to claw their way to the top for publications, funding and reputation. But, in an ideal world, the science would speak for itself.
By continuing a single blind system, we are allowing for many more variables to be brought into the process. And, that seems, well, totally unscientific.
Source: Andrew Tomkins "Reviewer bias in single- versus double-blind peer review" PNAS vol. 114 no. 48. 12708–12713
