llms-used-relevance-judges-specialized-non-english-brazilian-portuguese-legal CLM-0600When LLMs are used as relevance judges on a specialized non-English (Brazilian Portuguese legal) retrieval collection, they assign systematically higher relevance grades than human experts (mean absolute error 0.46-0.66 on a 0-2 scale, Cohen's kappa with humans only 0.32-0.53), yet the resulting system rankings stay close to the human reference for rank-aware metrics (nDCG@10, MRR, Kendall's tau >= 0.90) and diverge for P@10 and R@10.
Capability: Judging another's work as well as a careful reviewer would
Observed on
46 queries, 812 judged query-document pairs, 15 retrieval systems (8 BM25 variants), three proprietary LLMs with two prompt styles; small collection where precision/recall metrics .
Sources
- Measured on one dataset the authors built: 3,048 human judgments from four experts aggregated into reference qrels, compared against six LLM-annotator configurations; bootstrap CIs over 46 queries often did not stay above tau=0.9, and the system pool is dominated by closely related BM25 variants, so ranking agreement is partly a property of a low-diversity pool. No fix proposed.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- When an LLM judge is given a prior evaluation score as context metadata on a revision, its rubric scores shift toward that uninformative prior — seven of eight tested models (API and open-weight, 0-5 scale, 20 fixed texts) show a negative total shift, and on a categorical compliance task the metadata blocks 48% of error corrections and flips about 10% of previously correct judgments to the anchored wrong label.Judging another's work as well as a careful reviewer would · reviewed by AI
- When an LLM judge is validated against expert annotators on multi-dimensional rubrics for open-ended agent outputs, a single pooled agreement figure overstates reliability: per-dimension human-human agreement varies widely (substantial on a recall-style Completeness dimension, only slight to fair on Relevancy, Structural Clarity and Scope Adherence), the pooled statistic is inflated by between-dimension variation, and the judge's residual disagreements are directional (consistently one step stricter on Completeness, one step more generous on Structural Clarity) rather than noise.Whether the measurement made the finding · reviewed by AI
- In ESG-report RAG question answering with retrieval held fixed at a single top-1 chunk, open-weight 2B-30B generators vary widely in how well they stay anchored to the retrieved passage (faithfulness 0.607-0.822) yet all land in a narrow low band of factual correctness (0.387-0.449), and the most faithful model is the least factually correct, so groundedness to context and correctness of extracted ESG facts do not track each other.Competence in a specialist domain · reviewed by AI
- When research-idea forecasts are scored by whether a later paper matches them under an LLM judge rubric, higher scores partly reflect broader, less specific ideas: the backbone with the higher hit rate also produced measurably more general forecasts and had more retrieved candidate papers pass the gate, and tightening the specificity threshold does not separate the two.Whether the measurement made the finding · reviewed by AI
- For multimodal LLMs, a behavioral perturbation signature (answer/confidence change under visual degradation, blank-image replacement, and grounding/relation probes) predicts which failure family an error belongs to far better than scalar confidence, yet folding the same signature into confidence as a direct penalty makes answer-versus-abstain ranking worse than scalar confidence alone.Saying it does not know · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A larger Portuguese or other non-English specialized collection where LLM-generated qrels produce nDCG@10 or MRR rankings with Kendall's tau well below 0.9 against aggregated human qrels, or where LLM judges show no positive grading bias.