llm-generated-peer-reviews-compared-human-reviews-point-point-under CLM-0594When LLM-generated peer reviews are compared with human reviews point by point under theory-guided critique frameworks, the model and human critique differ in kind rather than in quality: LLM reviews carry more explanatory depth, integrative reasoning and explicit argument structure, while human reviews more often identify higher-order weaknesses and give revision-oriented guidance, and an expert prompt does not close that gap but amplifies the LLM-side tendencies.
Capability: Judging another's work as well as a careful reviewer would
Observed on
ICLR 2025 reviews; critique annotated at point level for weakness critique and scientific questioning under five frameworks; baseline and expert prompts only.
Sources
- Validated by: Applied to ICLR 2025 human reviews versus baseline- and expert-prompted LLM reviews; the scheme separated the two sources into distinct functional profiles that overlap metrics would score as similar. No external validation of the annotation scheme reported.. Measured by annotating review points from ICLR 2025 against five frameworks, comparing human reviews with LLM reviews under two prompt conditions. [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- General-purpose factuality verifiers (entailment-, consistency-, and RAG-trained) and zero-shot frontier LLM judges perform near chance at spotting unsupported claims in scientific peer reviews that must be checked against the full submitted paper, because they cannot separate paper-unsupported assertions from legitimate evaluative critique; small models fine-tuned on in-domain examples beat them by a wide margin.Stating false facts confidently · reviewed by AI
- General-purpose factuality verifiers (entailment-, consistency-, and RAG-trained) and zero-shot frontier LLM judges perform near chance at spotting unsupported claims in scientific peer reviews that must be checked against the full submitted paper, because they cannot separate paper-unsupported assertions from legitimate evaluative critique; small models fine-tuned on in-domain examples beat them by a wide margin.Stating false facts confidently · reviewed by AI
- In formal persuasion dialogues on a UK political topic, LLM debaters are judged more persuasive than humans by crowd annotators (all nine LLM configurations occupied the top nine subjective ranks) while ranking far lower under argumentation-theoretic adjudication, where a human topped the logical ranking and humans took six of the top ten.Changing what a person believes or decides · reviewed by AI
- When an LLM judge is given a prior evaluation score as context metadata on a revision, its rubric scores shift toward that uninformative prior — seven of eight tested models (API and open-weight, 0-5 scale, 20 fixed texts) show a negative total shift, and on a categorical compliance task the metadata blocks 48% of error corrections and flips about 10% of previously correct judgments to the anchored wrong label.Judging another's work as well as a careful reviewer would · reviewed by AI
- In fully connected networks of LLM agents (Llama 3.2-8B, 50 agents, 10 rounds) where a fixed minority holds an unchanging extreme stance, neutral agents' textual justifications converge on the biased agents' vocabulary faster and more completely than their numeric stances do, so monitoring only the text overstates how much the group actually agrees.Getting more from several models than from one · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: A point-level annotation of human and LLM reviews on comparable submissions showing LLM reviews identify higher-order weaknesses and revision guidance at rates equal to or above human reviewers, or showing expert prompting shifts the LLM profile toward the human one across these frameworks.