small-instruction-tuned-models-3-4b-answering-closed-form-truefalse-questions-under CLM-0154For small instruction-tuned models (3-4B) answering closed-form true/false questions under a stated user opinion, scoring each sampled response by how much more common its answer is than the group itself predicted, and using that score as the GRPO reward, reduces answer flips under user pressure and raises accuracy without any labels, roughly matching label-supervised desycophancy fine-tuning at higher compute cost.
Capability: Telling the user what they want to hear
Observed on
Five open-weight instruction-tuned base models of 3-4B parameters; closed-form binary answers only; groups of 64 samples per prompt; a synthetic 1000-item true/false set and judge-.
Sources
- 26 fine-tuning runs; reference SmolLM3-3B flip rate under pressure fell from 23% to 4% with accuracy under pressure from 80% to 93%; Benjamini-Hochberg correction across 166 tests, 77 surviving. Ablation shows dropping the prediction term removes the effect and collapses the policy onto one answer; correctness-only reward performs comparably, so the label-free advantage is in supervision, not size of effect. One seed, one training run per configuration, and the sycophancy metric is a flip rate between two prompt conditions rather than an honesty measure.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For small instruction-tuned models (3-4B) answering closed-form true/false questions under a stated user opinion, scoring each sampled response by how much more common its answer is than the group itself predicted, and using that score as the GRPO reward, reduces answer flips under user pressure and raises accuracy without any labels, roughly matching label-supervised desycophancy fine-tuning at higher compute cost.Telling the user what they want to hear · reviewed by AI
- For a same-family pair of reasoning and instruction-tuned 4B models, learning one interpolation ratio per module (attention block, FFN block, embedding, final norm) with a reward that scores correctness and ranks correct responses by length cuts average generated tokens by about 27% relative to the reasoning model while holding average accuracy, and beats fixed global-ratio merging and CMA-ES search over the same ratios.Cutting to the chase · reviewed by AI
- In label-free RL post-training of reasoning models, deriving each model's reward from a separately-trained peer model's majority vote — rather than from its own majority vote — avoids the reward collapse and length degeneration that self-rewarding runs show, and the benefit grows as the paired models differ in family or size because their errors overlap less.Getting more from several models than from one · reviewed by AI
- Used to steer test-time compute, generative LLM judges (7B-70B) rerank candidate responses about as well as outcome reward models, are consistently worse than process reward models at step-level beam search, and their natural-language critiques, the one thing only a judge can produce, did not guide generators to better responses on math, code or instruction following.Judging another's work as well as a careful reviewer would · reviewed by AI
- Instruction tuning that forces an answer to every question teaches models to fabricate when the question lies outside their parametric knowledge; splitting the tuning data by whether the base model already answers correctly and appending "I am unsure" to the rest (refusal-aware tuning) raises average precision on known-versus-unknown separation over vanilla tuning, transfers to out-of-domain datasets, and yields better calibrated confidence than post-hoc uncertainty scoring.Saying it does not know · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Applying the same group-internal surprise reward to further base models or datasets fails to lower the flip rate, or lowers it only by collapsing the policy onto a single answer, or label-supervised fine-tuning beats it by a wide margin on the same split. Proposed technique, not catalogued: peer-prediction (Bayesian Truth Serum) reward for label-free RL fine-tuning.