training-small-instruction-tuned-models-reinforcement-learning-plus-factual-supervisionWhen training small instruction-tuned models with reinforcement learning plus factual supervision, routing each atomic fact's verification score only to the tokens that produced it, and down-weighting verifier judgements that do not change when their key evidence is removed, improves factuality benchmark scores over trajectory-level or reasoning- step-level factual rewards.
Ingested from a paper but not yet reviewed by a human. It is deliberately inert: it does not move any technique’s standing, does not count toward the backtest, and is excluded anywhere a claim would carry weight. Read the source before relying on it.
Capability: Checking claims against evidence
Observed on
3B-parameter instruct models (Qwen2.5-3B-Instruct, Llama-3.2-3B-Instruct), GRPO-style RL on multi-hop QA with Wikipedia evidence snippets, GPT-4o for fact extraction and NLI verifi.
Sources
- Two 3B models, four hallucination benchmarks and four math benchmarks; FARCA beats KnowRL, FSPO, FaithRL and plain GRPO by roughly 1.75-2.21 average points. Ablations remove token provenance and reliability estimation separately, each dropping scores, which isolates the two components. Small models only, single seed reported, and the pipeline depends on GPT-4o extraction so the comparison is not compute-matched.
Disagreeing is the most useful thing you can do here. Both sides of every contested claim in this catalog were assembled by the same person, which is its weakest point.
Related claims
- For small instruction-tuned models (3-4B) answering closed-form true/false questions under a stated user opinion, scoring each sampled response by how much more common its answer is than the group itself predicted, and using that score as the GRPO reward, reduces answer flips under user pressure and raises accuracy without any labels, roughly matching label-supervised desycophancy fine-tuning at higher compute cost.Telling the user what they want to hear · unreviewed
- When fine-tuning a base model on instruction data, removing from the training targets any factual claim the base model cannot consistently recall raises the supported-claim rate of later long-form generations, but the gain comes largely from a more conservative response policy — more refusals and fewer supported claims per answer — not from generating more correct facts at equal coverage.Stating false facts confidently · unreviewed
- When a calculator-using model is trained with reinforcement learning on verifiable final-answer rewards for an arithmetic search task (Countdown), the gain lands almost entirely at low k — pass@1 rises sharply while pass@16 moves little or falls — because the update can only reinforce correct trajectories the starting policy already samples, and prompt groups with no correct sample supply no gradient.Using the tools it is given · unreviewed
- Used zero-shot as step-level action critics in long-horizon tool-calling tasks, frontier models are over-pessimistic — flagging a large share of correct actions and pushing the actor into revision loops that lower success below no critic at all — whereas small models fine-tuned on action-level verification rationales flag far fewer valid actions and raise repeated-run reliability.Using the tools it is given · unreviewed
- In open-weight instruction-tuned models, interventions that suppress caving to user pushback (DPO, SFT on chosen responses, or activation steering) also tend to reduce the model's rate of correcting a wrong answer when genuine supporting evidence is supplied, because the two answer-flip behaviors run on overlapping MLP neurons and attention heads with positively aligned steering directions.Telling the user what they want to hear · unreviewed
Notes
Ingested unreviewed on 2026-09-07 and deliberately inert until a human endorses it: it does not move a technique standing, does not count toward the backtest, and is excluded anywhere a claim would carry weight. Drafted confidence: low. Falsifier as drafted: An ablation or replication where token-level routing and reliability weighting give no gain over step-level factual rewards, or where coarse aggregation matches it once tuned equally. Proposed technique, not yet catalogued: fact-aligned reliability-weighted token credit.