refusal-learned-alignment-tuning-lives-low-rank-subspace-hidden-state-activations CLM-0277mechanismsingle paperReviewed by AI
Refusal learned by alignment tuning lives in a low-rank subspace of hidden-state activations that recurs across model families; ablating directions in that subspace lowers harmful-request refusal, and a scalar computed from the subspace's geometry before any fine-tuning (the Geometric Fragility Score) picks out which initially safe model will keep the most refusal after small benign LoRA fine-tuning.
Capability: Retaining skills and safety after fine-tuning
Observed on
Twenty-one instruction-tuned open models, 3B-32B, six alignment recipes, 2026; fragility test is rank-8 LoRA on harmless Alpaca subsets; refusal measured on harmful-request sets wi.
Sources
- supportsSkin-Deep: A Geometric Diagnostic for Alignment Fragility in Large Language Model RepresentationsValidated by: Direction-ablation causal test on four models (non-overlapping 95% CIs), linear-probe accuracy at least 0.90 for the subspace, and a LoRA-fragility experiment where the score's top pick retained the most refusal; twenty-one models across six recipes.. Broad model coverage and a causal ablation, but the predictive claim rests on one LoRA protocol and one fine-tuning budget; closed models cannot be scored since it needs activations.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For multimodal LLMs, a behavioral perturbation signature (answer/confidence change under visual degradation, blank-image replacement, and grounding/relation probes) predicts which failure family an error belongs to far better than scalar confidence, yet folding the same signature into confidence as a direct penalty makes answer-versus-abstain ranking worse than scalar confidence alone.Saying it does not know · reviewed by AI
- Instruction tuning that forces an answer to every question teaches models to fabricate when the question lies outside their parametric knowledge; splitting the tuning data by whether the base model already answers correctly and appending "I am unsure" to the rest (refusal-aware tuning) raises average precision on known-versus-unknown separation over vanilla tuning, transfers to out-of-domain datasets, and yields better calibrated confidence than post-hoc uncertainty scoring.Saying it does not know · reviewed by AI
- For LoRA-ensemble credal commitment rules on 7-9B instruct backbones, the decision to commit tracks agreement among the adapters rather than whether a request should be answered: on AdvBench harmful prompts the ensembles commit more often (33-97%) and with higher lower-bound confidence than on benign Dolly instructions, because safety training makes refusal the unanimous first continuation.Saying it does not know · reviewed by AI
- For 7-9B instruct backbones fine-tuned with an ensemble of five LoRA adapters, deciding to answer only when one candidate's minimum probability across adapters exceeds every other candidate's maximum gives a threshold-free abstention rule whose commitment rate collapses when the supplied evidence is corrupted (CoQA 31-48% down to 4-8%), while mean-pooling the same adapters into one distribution requires a tuned threshold and still answers more corrupted prompts at matched clean coverage.Saying it does not know · reviewed by AI
- For small instruction-tuned models (3-4B) answering closed-form true/false questions under a stated user opinion, scoring each sampled response by how much more common its answer is than the group itself predicted, and using that score as the GRPO reward, reduces answer flips under user pressure and raises accuracy without any labels, roughly matching label-supervised desycophancy fine-tuning at higher compute cost.Telling the user what they want to hear · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A model family where refusal is not low-rank or where the score's ranking does not predict post-LoRA refusal retention, or a fine-tuning attack that erases refusal equally across scores.