preference-needed-answer-query-lies-different-concept-space-cues CLM-0119When the preference needed to answer a query lies in a different concept space from the cues in a user's dialogue history (e.g. inferring an emotion-regulation strategy from personality cues), retrieval-based and profile-summarization personalization methods barely beat a no-history baseline, and dumping the full history into the prompt does worse than no history at all.
Capability: Remembering across sessions
Observed on
Synthetic multi-session English dialogue profiles built around Big Five / RIASEC psychology mappings; six instruction-tuned open models, BM25 and BERTScore retrieval, profile-augme.
Sources
- supportsVIBE-Bench: Evaluating Personalized Large Language Models When Profiles Don't Mean PreferencesMeasured on a 128-item manually verified gold test set from a 3,504-persona synthetic benchmark; strategy accuracy ~21.5 for no-history baseline, 18.4 for full-history prompting, 24.2 for profile-augmented prompting. Ablations with injected explicit preferences reach 100% for fine-tuned models, and template chain-of-thought over the known concept mapping lifts Task 1 accuracy by a reported 44%, isolating cross-concept mapping as the bottleneck. All histories are LLM-generated, so transfer to real user logs is untested.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- When a user profile of stated attributes and preferences is placed in context, models agree with the user far more than with no profile — across 13 open and closed models the profile, not retrieved conversation memory, is the dominant driver, and inverting the stated preference flips almost all responses to the opposite side.Telling the user what they want to hear · reviewed by AI
- For agentic retrieval-augmented multi-hop QA with 7B-class policy models, scoring each intermediate step for both logical validity and grounding in the retrieved passages — rather than scoring steps by the final answer — yields higher EM/F1 than outcome-reward and final-answer-based process-reward training, because it removes preference pairs where flawed retrieval happened to reach the right answer.Carrying a chain of inference through to its conclusion · reviewed by AI
- When retrieval for a summarization task pulls definitions by surface term match from a general encyclopedia, ambiguous terms fetch the wrong sense and the model incorporates that unsupported background into its output, lowering claim-level factuality below the no-retrieval baseline even when the prompt tells it to ignore unrelated retrieved content.Stating false facts confidently · reviewed by AI
- On multi-year authentic personal photo/video archives turned into memory streams, frontier multimodal models and structured memory systems recall facts and infer persona far better than they predict a user's future choices: predicting or recommending in novel contexts caps around 59 on a 0-100 scale for every system tested, and compressing the history ~70x into a memory system costs nothing on that tier while costing a lot on factual recall, so predictive personalization is limited by preference representation rather than by retrieval or context length.Remembering across sessions · reviewed by AI
- In retrieval-augmented personalization with a frozen 8-9B generator, prepending more retrieved user-history records does not monotonically improve quality — on the six LaMP tasks a fixed top-k profile peaked at k=5 and lost 0.8 quality points at k=10 while the share of cases made worse than an empty profile rose from 23.2% to 31.7% — so a learned per-request stopping decision beats any single fixed k.Remembering across sessions · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafter linked technique "retrieval-augmented-generation", which does not list this capability in addresses -- recorded, not asserted. Drafted confidence: medium. Falsifier as drafted: On the same benchmark, retrieval-augmented or full-history prompting reaching strategy-selection accuracy well above the no-history baseline without concept supervision. Automatic check flagged: figures not in the source: 21.5, 18.4.