Twice-Told Critiques: Claim-Level Homogenization of Peer Review in the LLM Era
Abstract
Critique overlap among peer reviews is rising, and reviews identified as LLM-generated are less distinct from each other, indicating decreasing diversity of critiques.
Peer review relies on several readers finding different problems. We measure critique overlap in conference and journal reviews. At ICLR, measured overlap rose from 12.0% in 2026 to 21.3% in 2027, in reviews from the same stage of the review process, with year-on-year rises at NeurIPS, ICML and the F1000 and PLOS journals. A third ICLR review of a paper added 5.46 new problems to the other two in 2027, against 6.12 in 2026. We apply MELD, a detector of LLM-generated text, to all reviews. Within a paper, two reviews that MELD flags overlap more than two unflagged reviews: by 3.0 points at ICLR in 2024-2025, by 4.5 points in 2026-2027, and by 5.4 points across five journal sources. These results provide evidence that LLM-flagged reviews of one paper are less distinct from each other than unflagged reviews, and suggest that the diversity of critiques among reviews of a single paper is falling.
Community
This is an analytical paper about side effects of LLM use in reviewing papers in major conferences and journals. One of the main findings is that reviewers of the same paper now repeat each other’s critiques more often. This overlap rose sharply at ICLR 2027, with smaller rises at other venues. Another finding is that two LLM-flagged reviews of one paper share more critiques than two unflagged ones, at conferences and in journals.
Repo: https://github.com/IlyaGusev/twicetold
Dataset: https://huggingface.co/datasets/IlyaGusev/twicetold
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- Same Papers, Different Verdicts: Tracing Version Drift in LLM Peer Review (2026)
- Copies Are Not Confirmations: How News Duplication Fools LLM Judges and Continual Learners (2026)
- Defensive Writing in GPT Models (2026)
- Learning Checklists to Reduce Lottery Effects in Scientific Peer Review (2026)
- What Do LLM Judges Actually Verify? Matched Controls and Record Gaps for Document Currency (2026)
- AI Reviews Are Biased Towards the Appendix (2026)
- Review Rarely Beats the Stronger Model: Separating Verification from Capability Transfer in LLM Pipelines (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper