grading-free-form-answers-algorithm-analysis-complexity-theory-questions-against-instructor-reference CLM-0399When grading free-form answers to algorithm-analysis and complexity-theory questions against instructor reference answers, n-gram overlap metrics carry almost no signal about answer quality, because logically equivalent proofs differ in notation, variable names and proof strategy: BLEU-4 was 0.0000 on all 179 questions and ROUGE-1 F1 stayed in a narrow 0.088-0.129 band across topics while a rubric score varied from 0.66 to 0.83.
Capability: Whether the measurement made the finding
Observed on
Open-ended mathematical proof and complexity-analysis answers scored against a single short instructor reference; one RAG system built on DeepSeek V3..
Sources
- Validated by: Applied to all 179 responses; the rubric separated topics (0.6629 recurrence relations to 0.8250 sorting) where ROUGE did not. No human learning-outcome validation and no inter-rater reliability for the rubric are reported.. Measured on 179 curated questions from one system, with no baseline system and no comparison to human grading; the divergence between metrics is documented but the rubric is unvalidated against learning outcomes or independent raters, and the rubric was applied by the authors' own harness.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- General-purpose factuality verifiers (entailment-, consistency-, and RAG-trained) and zero-shot frontier LLM judges perform near chance at spotting unsupported claims in scientific peer reviews that must be checked against the full submitted paper, because they cannot separate paper-unsupported assertions from legitimate evaluative critique; small models fine-tuned on in-domain examples beat them by a wide margin.Stating false facts confidently · reviewed by AI
- General-purpose factuality verifiers (entailment-, consistency-, and RAG-trained) and zero-shot frontier LLM judges perform near chance at spotting unsupported claims in scientific peer reviews that must be checked against the full submitted paper, because they cannot separate paper-unsupported assertions from legitimate evaluative critique; small models fine-tuned on in-domain examples beat them by a wide margin.Stating false facts confidently · reviewed by AI
- For a domain-specific language essentially absent from pretraining corpora (FORM symbolic algebra), QLoRA fine-tuning an 8B model on a few thousand examples validated by executing them through the language's own interpreter beats frontier open-weight models of up to 756B parameters given syntax documentation in context, on execution rate and strict output match for the larger benchmarks, while general reasoning scores drop under 3 points.Generating and editing working code · reviewed by AI
- For small instruction-tuned models (3-4B) answering closed-form true/false questions under a stated user opinion, scoring each sampled response by how much more common its answer is than the group itself predicted, and using that score as the GRPO reward, reduces answer flips under user pressure and raises accuracy without any labels, roughly matching label-supervised desycophancy fine-tuning at higher compute cost.Telling the user what they want to hear · reviewed by AI
- For 8B-14B generative reward models trained as judges, making the model generate input-adaptive evaluation criteria inside its own reasoning chain and rewarding the quality of those criteria with a stronger teacher model raises agreement with preference labels, and the gain scales with the teacher's capability rather than with the presence of the rubric step alone.Judging another's work as well as a careful reviewer would · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Show that on the same or similar proof-answer sets, BLEU/ROUGE rank systems in the same order as expert human quality judgements, or that near-zero BLEU corresponds to genuinely poor answers when experts grade them. Automatic check flagged: figures not in the source: 0.088, 129, 0.83.