Peer Review Integrity: Analysis of Bohannon's Sham Paper Experiment
The integrity of scientific research relies heavily on peer review—the process by which experts in a specific field evaluate a manuscript for accuracy and validity before it is published. To test the effectiveness of this system, researcher Bohannon conducted a provocative experiment by submitting a deliberately flawed "sham paper" to various journals to see how many would accept it despite its scientific failings.
Key Facts
- Out of 304 journals, 157 accepted the flawed paper.
- Only 36 journals provided review comments that identified scientific problems.
- 86 journals published the paper despite performing a legitimate peer review.
- 82% of publishers on Beall's list accepted the sham paper.
- India was identified as the largest base for fee-charging open-access publishing in the study.
Acceptance versus Rejection Rates
When the experiment concluded, the results revealed a concerning lack of rigor across many publications. Of the 304 journals contacted, 157 accepted the paper, 98 rejected it, 20 were still in the review process, and 29 did not respond. To isolate the failure of the review process itself, Bohannon focused on the 106 journals that clearly performed some form of review.
The findings were stark: only 36 of these journals recognized the scientific problems within the paper. Even more troubling, 16 of those 36 journals accepted the paper despite the poor reviews. Ultimately, 86 journals that claimed to perform peer review published the fraudulent work.
[no image]
The Role of Prestigious Institutions
The failure of quality control was not limited to obscure publications. Many journals that accepted the sham paper were associated with prestigious publishing companies and institutions, including Elsevier, SAGE, and Wolters Kluwer (via its subsidiary Medknow, specifically the Journal of Natural Pharmaceuticals), as well as several universities.
Examples of Rigorous Review
Conversely, some journals maintained high standards. PLOS (PLOS ONE), MDPI (Cancer), and Hindawi (Chemotherapy Research and Practice, ISRN Oncology) rejected the paper. PLOS ONE was noted for providing the most rigorous review of all, being the only journal to identify ethical lapses, such as the lack of documentation regarding the treatment of animals used to create cancer cell lines.
Evaluating Industry Watchlists: DOAJ and Beall's List
The experiment provided a real-world test for tools used to identify predatory publishers—entities that prioritize profit (via publication fees) over academic rigor.
Among publishers listed on Beall's list, 82% accepted the sham paper. While Jeffrey Beall argued this validates his list's ability to spot poor quality control, science communicator Phil Davis noted that the 18% of Beall-listed publishers who rejected the paper suggests that nearly one in five may be falsely accused.
Publishers listed in the Directory of Open Access Journals (DOAJ) fared better, with a 45% acceptance rate. In response to these findings, the DOAJ announced the implementation of new criteria for inclusion in their directory.
[no image]
The Global Landscape of Journal Fraud
By tracing IP addresses, WHOIS registrations, and bank invoices for publication fees, a global map of the publishing industry emerged. The data highlighted a significant concentration of fee-charging open-access publishing in specific regions.
India emerged as the largest hub, with 64 publishers accepting the flawed paper and only 15 rejecting it. The United States followed as the next largest base, with 29 acceptances and 26 rejections. In Africa, Nigeria had the highest acceptance rate, with 100% of the contacted publishers accepting the paper.
| Category/Region | Acceptance Rate / Count | Notes |
|---|---|---|
| Overall Journals | 157 / 304 | Total accepted vs total submitted |
| Beall's List Publishers | 82% | High correlation with poor quality control |
| DOAJ Publishers | 45% | Led to updated inclusion criteria |
| India | 64 Accepted / 15 Rejected | Largest base for fee-charging OA publishing |
| United States | 29 Accepted / 26 Rejected | Second largest base |
| Nigeria | 100% | Highest acceptance rate in Africa |
Frequently Asked Questions
What was the primary goal of Bohannon's experiment?
The goal was to test the effectiveness of the peer review process by submitting a scientifically flawed "sham paper" to various journals to see if they could detect the errors and reject the work.
How many journals failed to catch the scientific errors?
Out of 106 journals that performed a discernible review, only 36 identified scientific problems, and 86 journals ultimately published the flawed paper.
Which journal provided the most thorough review?
PLOS ONE provided the most rigorous review and was the only journal to identify the ethical issues regarding the treatment of animals in the study.
What is the difference between Beall's list and DOAJ in this study?
Publishers on Beall's list had a much higher acceptance rate (82%) compared to those in the DOAJ (45%), suggesting Beall's list is more effective at identifying predatory publishers, though it may include some false positives.
Which countries had the most fee-charging open-access publishers?
India was the largest base, followed by the United States. In Nigeria, every publisher contacted in the study accepted the sham paper.