preference-needed-answer-query-lies-different-concept-space-cuesWhen the preference needed to answer a query lies in a different concept space from the cues in a user's dialogue history (e.g. inferring an emotion-regulation strategy from personality cues), retrieval-based and profile-summarization personalization methods barely beat a no-history baseline, and dumping the full history into the prompt does worse than no history at all.
Ingested from a paper but not yet reviewed by a human. It is deliberately inert: it does not move any technique’s standing, does not count toward the backtest, and is excluded anywhere a claim would carry weight. Read the source before relying on it.
Capability: Remembering across sessions
Observed on
Synthetic multi-session English dialogue profiles built around Big Five / RIASEC psychology mappings; six instruction-tuned open models, BM25 and BERTScore retrieval, profile-augme.
Sources
- supportsVIBE-Bench: Evaluating Personalized Large Language Models When Profiles Don't Mean PreferencesMeasured on a 128-item manually verified gold test set from a 3,504-persona synthetic benchmark; strategy accuracy ~21.5 for no-history baseline, 18.4 for full-history prompting, 24.2 for profile-augmented prompting. Ablations with injected explicit preferences reach 100% for fine-tuned models, and template chain-of-thought over the known concept mapping lifts Task 1 accuracy by a reported 44%, isolating cross-concept mapping as the bottleneck. All histories are LLM-generated, so transfer to real user logs is untested.
Disagreeing is the most useful thing you can do here. Both sides of every contested claim in this catalog were assembled by the same person, which is its weakest point.
Related claims
- When a user profile of stated attributes and preferences is placed in context, models agree with the user far more than with no profile — across 13 open and closed models the profile, not retrieved conversation memory, is the dominant driver, and inverting the stated preference flips almost all responses to the opposite side.Telling the user what they want to hear · unreviewed
- When retrieval for a summarization task pulls definitions by surface term match from a general encyclopedia, ambiguous terms fetch the wrong sense and the model incorporates that unsupported background into its output, lowering claim-level factuality below the no-retrieval baseline even when the prompt tells it to ignore unrelated retrieved content.Stating false facts confidently · unreviewed
- When research-idea forecasts are scored by whether a later paper matches them under an LLM judge rubric, higher scores partly reflect broader, less specific ideas: the backbone with the higher hit rate also produced measurably more general forecasts and had more retrieved candidate papers pass the gate, and tightening the specificity threshold does not separate the two.Whether the measurement made the finding · unreviewed
- For small instruction-tuned models (3-4B) answering closed-form true/false questions under a stated user opinion, scoring each sampled response by how much more common its answer is than the group itself predicted, and using that score as the GRPO reward, reduces answer flips under user pressure and raises accuracy without any labels, roughly matching label-supervised desycophancy fine-tuning at higher compute cost.Telling the user what they want to hear · unreviewed
- For persistent-memory agents, any memory-write rule that decides using only recency and provenance is stuck on a single tradeoff — accepting more genuine preference updates means admitting more poisoned ones — because an adversary who can launder a claim through the user's own channel matches the statistics a genuine revision produces; conditioning the write decision on the inferred authenticity of the claim given the interaction history moves both axes at once.Remembering across sessions · unreviewed
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafter linked technique "retrieval-augmented-generation", which does not list this capability in addresses -- recorded, not asserted. Drafted confidence: medium. Falsifier as drafted: On the same benchmark, retrieval-augmented or full-history prompting reaching strategy-selection accuracy well above the no-history baseline without concept supervision. Automatic check flagged: figures not in the source: 21.5, 18.4.