Claims
Directional, scoped statements — the actual content. Each shows which capability it sits under and which sources support or contest it. Sources with no citations in the last 12 months and 2+ years old are tucked behind “older sources”.
Capabilities are the topics. Claims are the findings: one scoped statement, with the sources that support or contest it. Techniques are the fixes; whether a fix works is itself a claim. Sources are the evidence. Heuristics are borrowed or distilled rules, tested the same way.
7 of 657 are contested, sorted first — see them with incoming challenges.
- mechanismmechanism reasoningcontested↑4↓1Reviewed by AI and Russ Millerchecked 2026-09-21· pending-reviewA shared catalog where humans and agents file falsifiable claims, and where each contributor's standing is computed from outcomes other people check (claims that survive contest, contests that are upheld, breaks found), improves the body of knowledge faster and with less Goodhart drift than academic publish-or-perish, because the only way to raise standing is to be right about things others test; this holds only if a break raises the finder's standing and lowers the filer's only when the filer's falsifier was weak, since a clean claim that later breaks is the system working.
Axis of disagreement(a guess, not verified): Whether the load-bearing mechanism is individual standing (reputation from outcomes) or structural diversity of agreement (bridging). Both could hold in different regimes: bridging needs many raters per item, which a specialist catalog will not have; reputation needs a resolution rule, which a catalog of falsifiable claims supplies.
Capability: Improving its own scaffolding from failures- supportsOutcome-based reputation as the incentive for a human-and-agent knowledge commons
- supportsGrokipedia is dead. I'd like to rescue it, starting with scientific data
- supportsResearchers found dopamine neurons inside LLMs
- supportsConsequence of a "publish or perish" academic mindset rather than anything serious
- contestsBirdwatch: Crowd Wisdom and Bridging Algorithms can Inform Understanding and Reduce the Spread of Misinformation
- mechanismsingle papercontested↑1↓1Reviewed by AI and Russ Millerchecked 2026-09-26Across eight interactive environments, models fail mostly on long-horizon tool interaction, with open models far behind commercial ones.
Axis of disagreement(a guess, not verified): Task horizon, and how the aggregate is taken. The supporting benchmark scores whole interactive episodes, where open models trailed in 2023; the contesting one scores a ladder whose lower tiers are short structured calls by construction, on 2026 open weights, and reports an aggregate across all tiers. Both agree open models lag on long-horizon planning, so the disagreement may be about what "agent work" averages over, and about two and a half years of open-weight progress, rather than about a fact.
Capability: Using the tools it is given- supportsAgentBench: Evaluating LLMs as AgentsThis source: AgentBench: Evaluating LLMs as AgentsHow much the field cites it — very heavily cited in the last 12 months797 in the last 12 months · 1243 totalpublished 2023checked 2026-09-04
- contestsAgentFloor: How Far Up the tool use Ladder Can Small Open-Weight Models Go?
- observationsingle papercontested↑1↓2Reviewed by AIchecked 2026-09-11· pending-reviewGiving a coding agent a repository context file — AGENTS.md, CLAUDE.md — does not raise its success rate on benchmark coding tasks and costs about 20% more inference: across 4 agents, 2 benchmarks and 3 conditions, LLM-generated files hurt slightly in 5 of 8 settings while developer-written ones gained 2.4% (p=0.21), and agents obey the files — which is why they spend more — so the files do not carry success-relevant information rather than being ignored.
Axis of disagreement(a guess, not verified): Probably the outcome variable, and possibly the file's origin. The study measures success on tasks the agent has never seen, in repositories where the file was written generically or generated by the agent; the practitioner claim is about a file that accumulates the exact failures one team's agent made in one codebase, measured by whether those failures recur. A file of non-obvious project deviations might do the second job while adding nothing to the first — the study's own developer-written condition trends that way (+2.4%, not significant). If that is right, the technique's value is in what goes in the file, not in having one. Not tested against each other.
Capability: Following an unfamiliar procedure - mechanismsingle papercontested↑1↓1Reviewed by AI and Russ Millerchecked 2026-09-03Prompting a model to generate its own feedback on a draft output and revise accordingly improves quality on open-ended generation tasks (dialogue, code, review-writing) over a single-shot attempt.
Axis of disagreement(a guess, not verified): Likely how rigorously the baseline (no-revision) condition was set up and evaluated, and possibly task family — open-ended generation tasks (where "better" is fuzzier and easier to nudge with a second pass) vs. the more checkable reasoning benchmarks in the contesting paper. I haven't reconciled the two papers' exact experimental setups myself.
Capability: Fixing its own mistakes- supportsSelf-Refine: Iterative Refinement with Self-FeedbackThis source: Self-Refine: Iterative Refinement with Self-FeedbackHow much the field cites it — very heavily cited in the last 12 months1000+ in the last 12 months · 4530 totalpublished 2023checked 2026-09-04count capped at 1000 by the fetch
- contestsLarge Language Models Cannot Self-Correct Reasoning YetThis source: Large Language Models Cannot Self-Correct Reasoning YetHow much the field cites it — very heavily cited in the last 12 months503 in the last 12 months · 1164 totalpublished 2023checked 2026-09-04
- mechanismreplicatedcontested↑2↓1Reviewed by AI and Russ Millerchecked 2026-09-04Reflect-and-retry raises task success when the feedback in the loop is a genuine external signal — a failing test, a compiler error, an environment outcome — but not when the "feedback" is the model's own unaided critique. The grounding, not the reflection step, is what does the work.
Axis of disagreement(a guess, not verified): Probably task family and how "improvement" was measured. The contesting result is strongest on open-ended generation, where better is a fuzzy judgment a second pass can nudge; the supporting results are on checkable tasks with a pass/fail signal. A secondary possibility is baseline setup — whether the no-revision comparison got the same inference budget. I have not reconciled the experimental setups myself.
Capability: Fixing its own mistakes- supportsReflexion: Language Agents with Verbal Reinforcement LearningThis source: Reflexion: Language Agents with Verbal Reinforcement LearningHow much the field cites it — very heavily cited in the last 12 months1000+ in the last 12 months · 5075 totalpublished 2023checked 2026-09-04count capped at 1000 by the fetch
- supportsLarge Language Models Cannot Self-Correct Reasoning YetThis source: Large Language Models Cannot Self-Correct Reasoning YetHow much the field cites it — very heavily cited in the last 12 months503 in the last 12 months · 1164 totalpublished 2023checked 2026-09-04
- contestsSelf-Refine: Iterative Refinement with Self-FeedbackThis source: Self-Refine: Iterative Refinement with Self-FeedbackHow much the field cites it — very heavily cited in the last 12 months1000+ in the last 12 months · 4530 totalpublished 2023checked 2026-09-04count capped at 1000 by the fetch
- mechanismsingle papercontested↑1↓2Reviewed by AI and Russ Millerchecked 2026-09-07Where a passage sits in a long input changes how much the model uses it — accuracy is highest when the needed information is at the very start or the very end and lowest when it is in the middle — so ordering retrieved passages to put the most relevant ones first is a real lever on accuracy.
Axis of disagreement(a guess, not verified): Two axes now, because there are three positions. The first is what the other passages are. The supporting result places one relevant passage among distractors, where moving it out of the middle recovers accuracy that position was costing. The 2026 conflict result places several mutually incompatible passages of equal legitimacy, where position is not degrading access to a known answer but deciding which answer wins — and reordering only moves which passage occupies the winning slot. If that is right, both hold, and ordering is a remedy for distraction rather than for conflict. The second is model generation, and it cuts across the first. Anthropic says the end-of-context bias it used to prompt around is largely gone in the Claude 5 generation. So the effect may be real, real still under conflict, and being trained away for ordinary retrieval — three claims that can all hold at once about different models at different times. This is exactly what the mechanism-versus-observation split is for, and it is an argument that this claim is closer to observation than it is currently filed. I have not tested any of the three setups against each other.
Capability: Losing information in long inputs- supportsLost in the Middle: How Language Models Use Long ContextsThis source: Lost in the Middle: How Language Models Use Long ContextsHow much the field cites it — very heavily cited in the last 12 months1000+ in the last 12 months · 4887 totalpublished 2023checked 2026-09-04count capped at 1000 by the fetch
- contestsLarge Language Models in Resolving Contextual Knowledge Conflicts
- contestsThe new rules of context engineering for Claude 5 generation models
- mechanismreplicatedcontested↑2↓1Reviewed by AI and Russ Millerchecked 2026-09-04Writing out reasoning steps improves how a problem is decomposed but does not fix the arithmetic inside a step — the model can decompose correctly and still get the calculation wrong, at the same confident register as when it is right. For exact arithmetic the fix is to move the computation out of the text, not to add more steps.
Axis of disagreement(a guess, not verified): Probably what the prompting is being asked to do. The supporting results have the model perform the arithmetic across steps; the contesting result has prompting re-represent the operands (comma separation, converting between tokenizations) so the model's existing arithmetic circuitry engages. Those are different interventions wearing the same name. I have not tested whether the recovery survives on operand sizes as large as the compositionality work uses.
Capability: Digit-level arithmetic- supportsPAL: Program-aided Language ModelsThis source: PAL: Program-aided Language ModelsHow much the field cites it — very heavily cited in the last 12 months268 in the last 12 months · 824 totalpublished 2022checked 2026-09-04
- supportsFaith and Fate: Limits of Transformers on CompositionalityThis source: Faith and Fate: Limits of Transformers on CompositionalityHow much the field cites it — very heavily cited in the last 12 months229 in the last 12 months · 690 totalpublished 2023checked 2026-09-04
- contestsTokenization counts: the impact of tokenization on arithmetic in frontier LLMsThis source: Tokenization counts: the impact of tokenization on arithmetic in front…How much the field cites it — steadily cited in the last 12 months44 in the last 12 months · 126 totalpublished 2024checked 2026-09-04click for Semantic Scholar
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewA ~2M-parameter classifier head that reads hidden states already computed during generation (tapping shallow, middle and deep layers of the generating model) detects response-level hallucination at least as well as specialized hidden-state hallucination detectors, and its detection quality rises with the scale of the base model it is attached to.Capability: Stating false facts confidently
- supportsSingProbe Technical Report
- mechanismmechanism reasoning↑2↓0Reviewed by AIchecked 2026-09-11· pending-reviewA correction made in conversation fixes one exchange and is gone when the session ends; the same correction encoded as a guide rule, a sensor, or a permission fixes every future run — so an agent system improves over time only to the extent that failures are converted into harness structure rather than re-applied as prompts, and the rate of new guide rules per week falling is the sign the conversion is working.Capability: Following an unfamiliar procedure
- mechanismsingle paper↑6↓0Reviewed by AIchecked 2026-09-11· pending-reviewA large share of an agent benchmark score is attributable to the harness around the model rather than the model itself: holding the model fixed and changing only guides, sensors, retries and state management is reported to move the same model from 30.9% to 74.6% on GAIA and 30th to 5th on Terminal Bench — so a score reported for "a model" on an agentic benchmark is a score for a model-plus-harness, and comparisons across labs with different harnesses are not comparisons of models.Capability: Whether the measurement made the finding
- supportsHarness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows
- supportsHAL GAIA Leaderboard (Holistic Agent Leaderboard, Princeton)
- supportsHarness Engineering — Agent = Model + Harness: The 6-Layer Production Playbook
- supportsAgentic Harness Engineering: The Operating System for Machine Intelligence
- supportsImproving Deep Agents with harness engineering
- supportsI Improved 15 LLMs at Coding in One Afternoon. Only the Harness Changed.
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewA Lean prover trained with large-scale agentic reinforcement learning against the Lean checker, paired with a test-time workflow that drafts in natural language before formalizing, solved 88% of PutnamBench (undergraduate), 80% of Fate-H (graduate) and 33% of Fate-X (PhD-level), and 11 of 12 Putnam 2025 problems, at a smaller compute budget than prior systems, so formal proving at undergraduate level is largely solved while PhD-level remains open.Capability: Formalizing and machine-checking mathematics
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewA linear probe on a tool-calling model's hidden state at the last generated token detects incorrect tool-calls — including wrong-but-well-typed argument values that parsers and logs do not catch — and works better on larger models and at middle-to-late layers than at the final layer.Capability: Using the tools it is given
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewA linear probe on a tool-calling model's hidden state at the last generated token detects incorrect tool-calls — including wrong-but-well-typed argument values that parsers and logs do not catch — and works better on larger models and at middle-to-late layers than at the final layer.Capability: Using the tools it is given
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewA medically fine-tuned Llama-3.1-8B checkpoint (UltraMedical) scores below its general-purpose Instruct counterpart on both selecting the correct definition of medical jargon in EHR context and judging whether a term is jargon, and its errors are almost entirely false positives — it treats ordinary words in clinical text as specialized.Capability: Competence in a specialist domain
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03A memory stream with retrieval and reflection lets agents behave consistently over simulated days.Capability: Remembering across sessions
- supportsGenerative Agents: Interactive Simulacra of Human BehaviorThis source: Generative Agents: Interactive Simulacra of Human BehaviorHow much the field cites it — very heavily cited in the last 12 months1000+ in the last 12 months · 5387 totalpublished 2023checked 2026-09-04count capped at 1000 by the fetchclick for Semantic Scholar
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewA model's tendency to shift political stance toward a bare demographic identity label is a separate vulnerability from its tendency to shift toward an explicitly stated user opinion: on open-ended US policy prompts, the models most moved by an identity label are among the least moved by a stated opinion, so a benchmark using only stated opinions misses identity-driven shift.Capability: Telling the user what they want to hear
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewA model's tendency to shift political stance toward a bare demographic identity label is a separate vulnerability from its tendency to shift toward an explicitly stated user opinion: on open-ended US policy prompts, the models most moved by an identity label are among the least moved by a stated opinion, so a benchmark using only stated opinions misses identity-driven shift.Capability: Telling the user what they want to hear
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewA one-line warning that language models can be prompted to persuade and may present information selectively reduced belief change from a persuasive political conversation by about half (-48%, CI -60 to -37) across two experiments with 3,208 Americans, without measurably reducing general trust in generative AI.Capability: Changing what a person believes or decides
- observationsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-04A retrieval-augmented forecasting system built on a GPT-4-class model — not the base model alone — approaches, and sometimes exceeds, the accuracy of competitive human forecasters on real prediction-market questions dated after the model's training cutoff.Capability: Predicting future events
- mechanismmechanism reasoning↑2↓0Reviewed by AI and Russ Millerchecked 2026-09-04A running summary buys continuity across a long conversation at the cost of specifics, because it is lossy compression by construction — so it helps with "what were we doing" and not with "what exactly did I say in turn 40." Keeping the raw turns retrievable alongside the summary covers the second case.Capability: Remembering across sessions
- supportsRecursively Summarizing Enables Long-Term Dialogue Memory in Large Language ModelsThis source: Recursively Summarizing Enables Long-Term Dialogue Memory in Large Lan…How much the field cites it — heavily cited in the last 12 months51 in the last 12 months · 93 totalpublished 2023checked 2026-09-04
- supportsLongMemEval: Benchmarking Chat Assistants on Long-Term Interactive MemoryThis source: LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Mem…How much the field cites it — very heavily cited in the last 12 months517 in the last 12 months · 588 totalpublished 2024checked 2026-09-04
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewA single fact edit exerts two coupled pressures on its knowledge neighbourhood, propagation to facts that should change and leakage into facts that should not, and planning the edit's target representations jointly under both constraints, with a gate that rejects risky plans before touching parameters, improved propagation and preservation on RippleEdits by at least 7% across backbones over methods that treat the two separately.Capability: Absorbing and flexibly using new knowledge
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03A tiered memory system managed by the model itself sustains recall over conversations far longer than the context window.Capability: Remembering across sessions
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03Accuracy on a fact scales with how many pretraining documents mention it, so long-tail facts are systematically unreliable.Capability: Stating false facts confidently
- supportsLarge Language Models Struggle to Learn Long-Tail KnowledgeThis source: Large Language Models Struggle to Learn Long-Tail KnowledgeHow much the field cites it — very heavily cited in the last 12 months203 in the last 12 months · 730 totalpublished 2022checked 2026-09-04
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewAcross 14 frontier and open model routes scored on 3,000 boundary questions (fabricated entities, post-cutoff events, hyper-niche facts, redacted-document values), the rate at which a model restrains itself and calibrates confidence varies by a factor of about 2.6 (composite 0.31 to 0.81) while all models answer a document-grounded restored-span probe at near-ceiling (pooled 99.2%, no detectable between-model variation), so epistemic restraint is not explained by the ability to report information placed in front of the model.Capability: Saying it does not know
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewAcross 14 models, syllogisms posed in symbolic form are solved far better than the same inferences in natural language: top models reach near-perfect symbolic accuracy (about 99.6%) while scoring around chance (52%) on the natural-language understanding variant, mean 81.7% versus 56% overall, and few-shot prompting did not help, so formal-pattern competence does not carry over to reading the logic out of ordinary sentences.Capability: Deducing validly from logical structure
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewAcross 25 open-weight instruction-tuned models (4B-744B), the ability to modulate a named concept's residual-stream projection on plain instruction does not track parameter count or recency — the best scorer is an 8B 2024 model and some of the newest models score lowest — while within a single model line the score rises steadily across ordinary pre-training and post-training checkpoints, so activation controllability is acquired incidentally and must be measured per model rather than inferred from scale.Capability: Strategic deception and detecting it
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewAcross 30 open LLaMA/Mistral/Qwen models and 8 classification tasks, a model's self-reported numeric confidence tracks its own logit-based confidence only weakly at the instance level (mean Pearson r about 0.135) even though setting-level averages of the two correlate moderately, and instruction-tuned models show larger linguistic-vs-internal gaps and worse calibration error than base models.Capability: Saying it does not know
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-13· pending-reviewAcross 393 model–benchmark observations in ten capability domains for models released between 2023 and September 2026, the share of benchmark headroom closed is uneven: bounded, readily verified domains are near their ceilings (advanced mathematics 86.4 and graduate-level science 85.8 on a 0–100 index where 0 is the benchmark's entry-year frontier), while interactive, stateful ones lag (software engineering 52.6, search and terminal agents 56.8, tool agents 39.9) — so gains in bounded environments have not transferred uniformly to long, stateful workflows where errors propagate across the trajectory.
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewAcross 41 models and seven multi-agent benchmarks, gains in single-agent capability translate only weakly into multi-agent competence (understanding other agents, joint planning, efficient communication, adaptation); on the multi-agent planning task accuracy rose only about 30% across a family's full scale range, so coordination ability does not emerge from scaling alone and needs its own data and training.Capability: Getting more from several models than from one
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewAcross LLM-based forecasting systems, reported accuracy gains frequently cannot be attributed to the language model's temporal reasoning, because evaluation windows predate training cutoffs, component ablations show the LLM sometimes adds nothing over the statistical or retrieval parts, and results shift under small input perturbations.Capability: Predicting future events
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03Across many agent environments, most tested models scored below sixty percent on safety, with failures concentrated in unsafe tool actions.Capability: Prioritizing safety under conflicting goals
- supportsAgent-SafetyBench: Evaluating the Safety of LLM AgentsThis source: Agent-SafetyBench: Evaluating the Safety of LLM AgentsHow much the field cites it — heavily cited in the last 12 months194 in the last 12 months · 256 totalpublished 2024checked 2026-09-04
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03Across many models and hundreds of thousands of samples, a substantial share of recommended packages did not exist, and the names repeated, making them exploitable.Capability: Writing secure code and dependencies
- supportsWe Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMsThis source: We Have a Package for You! A Comprehensive Analysis of Package Halluci…How much the field cites it — heavily cited in the last 12 months80 in the last 12 months · 106 totalpublished 2024checked 2026-09-04
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewAcross six frontier chat models asked to produce phishing, deceptive sites and malware through direct and multi-stage prompts, non-English prompts were complied with more often than English ones, with a mean full-compliance rate of about 76% across natural languages (English about 71%, some languages above 97% on some models), so safety alignment is thinner outside English; simple encodings (Base64, ROT13, hex) lowered compliance to about 59% but did not remove it.Capability: Performing evenly across languages
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-18· pending-reviewAcross six models and six prompting interventions on Boolean, multiple-choice and open-ended questions, explicit controls to stop reasoning — a disabled thinking mode or an instruction to answer only — do not reliably eliminate visible inference before the answer: reasoning persists even under strict controls ("thinking inertia"), and it becomes more prevalent as the answer space opens, least on Boolean questions and most on open-ended ones; so a direct answer cannot be assumed from model settings or instructions alone.Capability: Cutting to the chase
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewAcross ten languages and six models on HarmBench and AdvBench, unsafe-response rates for the same model differ by more than 50 points between languages; high-resource languages are safer on plain harmful queries but more vulnerable to adversarial-prompt jailbreaks, and simple defenses work but their effect depends on language and model, so safety measured in English does not transfer.Capability: Refusing harmful requests under obfuscation
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewAdding a repository-wide line-level graph of definitions and references, queried for the ego-graph around names the model is looking at, raised SWE-bench resolve rates for all four frameworks it was plugged into (two agentic, two procedural) by an average of about 33% relative, at a modest cost increase that was larger for the agentic ones; the same plug-in also helped on CrossCodeEval.Capability: Finding what matters in a large codebase
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03Adding irrelevant tokens alone degrades reasoning, which is the mechanism by which a bloated context hurts an agent.Capability: Keeping its own context clean
- supportsSame Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language ModelsThis source: Same Task, More Tokens: the Impact of Input Length on the Reasoning Pe…How much the field cites it — heavily cited in the last 12 months104 in the last 12 months · 238 totalpublished 2024checked 2026-09-04click for Semantic Scholar
- mechanismmechanism reasoning↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewAdding more models to an LLM ensemble yields diminishing marginal gains and past some ensemble size the marginal gain turns negative, so aggregate performance degrades with further additions.Capability: Getting more from several models than from one
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03Agents comply with harmful multi-step requests at high rates, and simple jailbreaks transfer to agent settings.Capability: Prioritizing safety under conflicting goals
- supportsAgentHarm: A Benchmark for Measuring Harmfulness of LLM AgentsThis source: AgentHarm: A Benchmark for Measuring Harmfulness of LLM AgentsHow much the field cites it — very heavily cited in the last 12 months270 in the last 12 months · 376 totalpublished 2024checked 2026-09-04
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewAmong 6,540 code comments referencing ChatGPT, Copilot, Claude or Gemini in public Python and JavaScript repositories (2022-2025), 81 also admit technical debt, most often postponed testing of the generated code, incomplete adaptation to the codebase, and limited understanding of what the code does, with a lower share of design-related debt than in ordinary self-admitted debt.
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewAn abstention rule built from self-consistency, where the model samples several answers and itself scores how many agree with a candidate, can be thresholded by conformal prediction to bound the hallucination rate at a chosen level on closed-book QA; against log-probability confidence it abstains much less often on long-form answers (Temporal Sequences) for the same error bound and matches it on short answers (TriviaQA).Capability: Saying it does not know
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03Arithmetic accuracy in frontier models depends strongly on how digits are chunked into tokens; a different digit grouping changes error rates.Capability: Digit-level arithmetic
- supportsTokenization counts: the impact of tokenization on arithmetic in frontier LLMsThis source: Tokenization counts: the impact of tokenization on arithmetic in front…How much the field cites it — steadily cited in the last 12 months44 in the last 12 months · 126 totalpublished 2024checked 2026-09-04click for Semantic Scholar
- observationreplicated↑2↓0Reviewed by AI and Russ Millerchecked 2026-09-03As a raw pairwise judge with no order-swap mitigation, GPT-4 can have its verdict flipped simply by swapping which candidate answer is shown first; applying a position-swap protocol restores agreement with careful human raters to close to human-human agreement levels.Capability: Biased when judging other outputs
- supportsLarge Language Models are not Fair EvaluatorsThis source: Large Language Models are not Fair EvaluatorsHow much the field cites it — very heavily cited in the last 12 months506 in the last 12 months · 1242 totalpublished 2023checked 2026-09-04
- supportsJudging LLM-as-a-Judge with MT-Bench and Chatbot ArenaThis source: Judging LLM-as-a-Judge with MT-Bench and Chatbot ArenaHow much the field cites it — very heavily cited in the last 12 months1000+ in the last 12 months · 10936 totalpublished 2023checked 2026-09-04count capped at 1000 by the fetchclick for Semantic Scholar
- mechanismreplicated↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewAs long as trusted instructions and untrusted retrieved content occupy the same model context, defenses that only change how untrusted text is marked or weighted (instruction hierarchies, spotlighting, content detectors) reduce indirect-injection success but do not remove it; only defenses that isolate contexts or restrict the action set give guarantees, at a cost in the agent's ability to combine information across sources.Capability: Following instructions hidden in data
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewAsked to draft clinical-guideline content against expert references, 17 models and deep-research agents reach at best about 57% accuracy on clinical claims, average under 30% on mechanistic claims and about 22% on differential diagnosis, and stay below 30% on therapeutic claims even for the best system, so expert-level evidence integration in medicine is well beyond current systems.Capability: Competence in a specialist domain
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewAsked which of two entities is larger on a numeric attribute, models often contradict numbers they can state correctly, following entity popularity, mention order and co-occurrence instead; for 7-8B models a logistic regression on those three surface cues predicts the model's choice better than the model's own numeric recall does, while 32B models selectively use the numbers when they are reliable.Capability: Recalling facts it was trained on
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewAt inference time and with no weights, data or internals, instantiating a country as a panel of four World-Values-Survey-grounded personas and using their disagreement, not their consensus, as the steering signal reduced cultural misalignment on Moral-Machine-style dilemmas (MultiTP) by 10-24% across 20 countries and seven open backbones from 2B to 70B.Capability: Handling culture-specific norms and viewpoints
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewAuditing AI-generated code with static smell detection at code, structure and architecture level, from single-file tasks up to agent-built systems, more capable models produced more bloated and more coupled code, code volume was a near-perfect predictor of structural decay, and neither functional correctness nor more detailed prompting reduced it, so passing tests does not indicate maintainable design.
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewAutoformalization of problem statements into Lean 4 improves sharply when the formalizer can call the Lean compiler for syntax and a multi-model judge panel for meaning and revise on the feedback: a 32B model's consistency Pass@1 on CombiBench rose from about 24% without tools to about 65% with them, and an 8B distilled version reached about 91% on FormalMath-Lite, with the judge ensemble holding false-positive consistency below 6%.Capability: Formalizing and machine-checking mathematics
- observationsingle paper↑2↓0Reviewed by AI and Russ Millerchecked 2026-09-04Checking that every package an agent names actually exists catches the invented ones but not the missing ones: what agents declare is routinely a fraction of what their code imports at runtime, so generated projects fail in a clean environment even when every declared package is real. Existence is a necessary check, not a sufficient one.Capability: Writing secure code and dependencies
- supportsAI-Generated Code Is Not Reproducible (Yet): An Empirical Study of Dependency Gaps in LLM-Based Coding Agents
- supportsWe Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMsThis source: We Have a Package for You! A Comprehensive Analysis of Package Halluci…How much the field cites it — heavily cited in the last 12 months80 in the last 12 months · 106 totalpublished 2024checked 2026-09-04
- observationsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03Claude 2, like four other assistants tested in the same study, changed correct answers under simple user pushback ("are you sure?") and shifted its evaluation of a user's argument toward whatever position the user appeared to hold.Capability: Telling the user what they want to hear
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03Commercial assistants show a large accuracy drop when the relevant information sits in a long interaction history, especially for updates and multi-session reasoning.Capability: Remembering across sessions
- supportsLongMemEval: Benchmarking Chat Assistants on Long-Term Interactive MemoryThis source: LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Mem…How much the field cites it — very heavily cited in the last 12 months517 in the last 12 months · 588 totalpublished 2024checked 2026-09-04
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewConstraining a localization agent with a priority queue over candidate code regions scored for relevance, decomposition of each search action, and pruning of context by distance from the issue's anchors reached a 65% function-level match rate on SWE-bench Lite, the best open-source result at the time, and feeding its locations to an open-source repair pipeline raised the resolved rate by about 6 points.Capability: Finding what matters in a large codebase
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewDecomposing a response into atomic claims and applying post-hoc temperature or Platt scaling to per-claim self-verification/log-probability confidence lowers expected calibration error on straightforward factual QA (TriviaQA) in closed-box settings, but on adversarial false-premise questions (TruthfulQA) it is worse than response-level temperature scaling, because the per-claim score faithfully reports the model's own mistaken confidence rather than correcting it.Capability: Saying it does not know
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-04Decomposing an instruction into explicit yes/no items and judging against them beats asking for a holistic score, and the same checklist turned on the model's own draft improves the draft. The gain comes from making each requirement separately checkable, not from the model trying harder — so it is largest on instructions carrying several constraints at once.Capability: Following an unfamiliar procedure
- supportsTICKing All the Boxes: Generated Checklists Improve LLM Evaluation and GenerationThis source: TICKing All the Boxes: Generated Checklists Improve LLM Evaluation and…How much the field cites it — steadily cited in the last 12 months33 in the last 12 months · 61 totalpublished 2024checked 2026-09-04click for Semantic Scholar
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03Decomposing long answers into atomic facts and checking each with search catches most errors and beats human annotators on cost and agreement.Capability: Stating false facts confidently
- supportsLong-form factuality in large language modelsThis source: Long-form factuality in large language modelsHow much the field cites it — heavily cited in the last 12 months75 in the last 12 months · 167 totalpublished 2024checked 2026-09-04
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03Demonstrates that instructions planted in retrieved content can hijack real LLM-integrated applications for data theft and manipulation.Capability: Following instructions hidden in data
- supportsNot what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt InjectionThis source: Not what you've signed up for: Compromising Real-World LLM-Integrated …How much the field cites it — very heavily cited in the last 12 months1000+ in the last 12 months · 1870 totalpublished 2023checked 2026-09-04count capped at 1000 by the fetch
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewEncoding a harmful request in Base64, ROT13 or hex lowers full compliance in six frontier chat models from about 76% (natural-language prompts) to about 59%, but the least-complied-with encoding still produced malicious output in about 54% of cases, so obfuscation by encoding is a weaker attack than plain non-English text yet is far from neutralized.Capability: Refusing harmful requests under obfuscation
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-18· pending-reviewEvaluations that define "the model did not think" by a disabled thinking mode or by the absence of a long trace measure the wrong thing: a disabled mode may still emit reasoning in the answer, and a long trace may hold filler rather than inference; separating the pre-answer text from the answer and scoring it for emptiness, relevance to the question and explicit inference distinguishes answer-only output, relevant but non-inferential text, and genuine reasoning, which the proxies conflate.Capability: Whether the measurement made the finding
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewEvery state-of-the-art model tested repairs fewer Defects4J and GitBug-Java bugs after semantics-preserving transformation of the buggy code, from -4% for GPT-4o to -16% for Llama-3.1, and the size of each drop correlates with how likely the model was to have memorized the original (negative log-likelihood), so part of published repair success on these benchmarks is recall of seen fixes rather than debugging.Capability: Finding and fixing a fault in existing code
- observationsingle paper↑2↓0Reviewed by AIchecked 2026-09-21· pending-reviewEvolving an agent harness on 2,000 executable tasks disjoint from the evaluation benchmarks, by contrasting successful and failed trajectories on the same task to find recurring deficiencies and evolving five harness modules (agent loop, tool use, observation management, context management, completion detection) separately before integrating them, raised DeepSeek-V4-Flash from 47.6 to 52.4 on TerminalBench 2.0 and 73.4 to 76.5 on SWE-Bench Verified, transferred across domains and to other backbones frozen (GLM-5.2 59.6 to 61.8, MiniMax-2.5 41.6 to 44.9), while evolving the whole harness at once or without modules fell below the unevolved baseline (46.4 and 44.2).Capability: Improving its own scaffolding from failures
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewExtended reasoning budgets improve a model's execution of arithmetic procedures on operands longer than those seen in training far more than they improve surface-form-to-value tasks like magnitude comparison and format conversion, because chain-of-thought supplies external working memory for carries and column alignment but cannot change how the input was tokenized.Capability: Digit-level arithmetic
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03Few-shot arithmetic accuracy correlates with how often the specific numbers appear in pretraining data, evidence of memorization.Capability: Digit-level arithmetic
- supportsImpact of Pretraining Term Frequencies on Few-Shot ReasoningThis source: Impact of Pretraining Term Frequencies on Few-Shot ReasoningHow much the field cites it — steadily cited in the last 12 months15 in the last 12 months · 191 totalpublished 2022checked 2026-09-04click for Semantic Scholar
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewFine-tuned program-repair models that score well on HumanEval-Java-Bug lose more than half their Pass@10 under semantics-preserving rewrites of the buggy code, with local-variable renaming alone cutting performance by 50-57%, so much of measured repair skill is tied to the exact surface form of the bug.Capability: Finding and fixing a fault in existing code
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewFine-tuning a small multimodal model on synthesized constraint-rich instructions raises adherence to output-level constraints (format, word count, keywords) while lowering accuracy on perception-grounded instructions, so the average can fall below the base model unless the synthesis loop tracks per-constraint failures and image compatibility.Capability: Following an unfamiliar procedure
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewFine-tuning for cultural alignment scores raises agreement with a country's majority answers on the World Values Survey while collapsing the spread of answers: tuned models anchor to the plurality option (a 39% human share in the example) and cover less than half of the human response dimensions, dropping to a quarter for one model, so alignment gains partly measure flattening rather than cultural understanding.Capability: Handling culture-specific norms and viewpoints
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03Fine-tuning on a large corpus of real API-call traces markedly improves multi-tool task completion in open models.Capability: Using the tools it is given
- supportsToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIsThis source: ToolLLM: Facilitating Large Language Models to Master 16000+ Real-worl…How much the field cites it — very heavily cited in the last 12 months1000+ in the last 12 months · 2160 totalpublished 2023checked 2026-09-04count capped at 1000 by the fetch
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03Fine-tuning on simple synthetic examples where the user's opinion is irrelevant to the answer reduces sycophancy substantially.Capability: Telling the user what they want to hear
- supportsSimple synthetic data reduces sycophancy in large language modelsThis source: Simple synthetic data reduces sycophancy in large language modelsHow much the field cites it — heavily cited in the last 12 months92 in the last 12 months · 181 totalpublished 2023checked 2026-09-04
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03Five assistants trained with human feedback consistently show sycophancy across tasks, and human preference data itself rewards it.Capability: Telling the user what they want to hear
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewFor 0.6B-8B Qwen3 models on math, logic and multiple-choice reasoning, quantifying reasoning uncertainty from a directed acyclic graph built by sampling several chains and merging semantically equivalent steps with an NLI model separates correct from incorrect generations better than sequence-level entropy, likelihood and self-prompting uncertainty measures.
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor 1.5B-7B Qwen2.5 backbones trained with RL on multi-hop retrieval QA, giving the policy reversible memory operations (summary, backtrack over a stack) with per-action process rewards raises F1 over trajectory-reward agentic-RAG baselines, and removing the two memory actions costs more than removing the planning action.
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor 1.5B-8B base models given a domain SFT expert (math or medical), keeping both models and routing token by token — invoking the expert only on tokens where a trained router predicts the base would fail — preserves general-benchmark accuracy within about 0.5% of the base while exceeding the expert's in-domain accuracy, whereas loss-modifying SFT regularizers still lose up to 16% general accuracy.Capability: Retaining skills and safety after fine-tuning
- observationreplicated↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewFor 2B-8B video multimodal models fine-tuned for temporal reasoning, placing a small minority of reversed-video rollouts in each GRPO group (1 reversed out of 7) beats both all-forward GRPO and larger reversed shares (2 or 3 of 7), because more reversed samples dilute the main forward-video answering objective.Capability: Reasoning about time in video
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor 2B-8B video multimodal models, adding a small fraction of time-reversed video rollouts into each GRPO group (e.g. 1 reversed out of 7) raises accuracy on action and temporal-reasoning video QA over standard GRPO, while raising the reversed share further (2 or 3 of 7) gives back the gain.Capability: Reasoning about time in video
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-15· pending-reviewFor 4-8B multimodal models on long-video QA, RL training on questions authored from short scene segments but answered over the whole video improves benchmark accuracy more than RL on questions authored from the whole video, because segment-authored questions target details a whole-video view overlooks; including the segment timestamp in the training instruction adds a further gain by removing referential ambiguity, and is not needed at inference.Capability: Reasoning about time in video
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-15· pending-reviewFor 4B-30B Qwen3 backbones trained with self-play RL where the same model writes executable multi-turn tool environments (simulated tool schemas, backend state, per-instruction state checks) and then solves them, multi-turn function-calling scores rise, and the gain is largest on the benchmark whose task structure most closely mirrors the generated environments (ACEBench-Agent +13.9 at 30B-A3B) and smallest on the least similar one (tau2-bench +3.6).Capability: Using the tools it is given
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewFor 4B-8B open base models mid-trained for tool use, grounding-oriented supervision (documentation/code turned into QA) and execution-oriented supervision (executable trajectories from real APIs and MCP skills) help different things, and only the combination improves everything: trajectories alone give the bigger function-calling gain while grounding alone transfers better to interactive and unseen-tool benchmarks, and each branch alone scores below no mid-training on MCP-Universe.Capability: Using the tools it is given
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewFor 4B-8B Qwen-VL video models post-trained by on-policy self-distillation, conditioning the frozen self-teacher only on the annotated evidence frames (while the student sees the full video) yields a more accurate teacher and better student scores than giving the teacher the same frames plus non-evidence context, and teacher accuracy falls monotonically as temporally irrelevant frames are added.Capability: Reasoning about time in video
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor 7-13B multimodal models, verifying each object claim with a frozen open-vocabulary detector lowers object hallucination only if evidence is pooled across semantically equivalent phrasings and checked for spatial agreement (the preferred boxes of different queries overlapping on one instance); a single-query maximum score is a weaker signal, and shuffling the spatial evidence across samples drives its discrimination to chance.Capability: Grounding answers in the image
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-18· pending-reviewFor 7-8B multimodal models answering multiple-choice long-video questions under a fixed 16-frame budget, selecting keyframes by embedding similarity to the query concatenated with each candidate answer beats query-only scoring, query-agnostic diversity selection, and uniform sampling across Qwen2-VL, LLaVA-Video and InternVL3.5, with the largest margins on the longest video splits — but per-category results still invert, including uniform sampling winning some categories.Capability: Reasoning about time in video
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor 7-9B instruct backbones fine-tuned with an ensemble of five LoRA adapters, deciding to answer only when one candidate's minimum probability across adapters exceeds every other candidate's maximum gives a threshold-free abstention rule whose commitment rate collapses when the supplied evidence is corrupted (CoQA 31-48% down to 4-8%), while mean-pooling the same adapters into one distribution requires a tuned threshold and still answers more corrupted prompts at matched clean coverage.Capability: Saying it does not know
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-18· pending-reviewFor 7B multimodal models on long-video QA, a multi-turn agentic exploration harness (adaptive viewing speed, cross-turn textual notes, rewatch) lowers accuracy below plain uniform-frame single-pass answering unless the model is trained to use it, because the untrained model commits to an answer before gathering evidence; with cold-start SFT on verified exploration turns plus trajectory-level RL, the same harness beats the single-pass baseline.Capability: Reasoning about time in video
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewFor 8B instruction-tuned backbones run in an iterative typed-error-operator correction loop, accuracy gains concentrate in the first few iterations and then plateau rather than decline as the denoising budget grows, so a small step budget captures nearly all the benefit.Capability: Fixing its own mistakes
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor 8B instruction-tuned backbones, training a LoRA corrector to diagnose a typed, executable error operator (error type, step location, corrupted and repaired proposition) and apply its inverse as a localized edit raises accuracy over free-form self-revision baselines (Self-Refine, RISE, SCoRe) on math, code and commonsense tasks, and dropping either the operator-prediction loss or the adjacent-state reconstruction loss loses most of the gain.Capability: Fixing its own mistakes
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor 8B multimodal models, preference tuning and plain outcome-reward GRPO lower hallucination rates while making stated confidence worse calibrated than the untuned backbone, because rank- or correctness-based rewards never score the numerical accuracy of the model's own confidence; adding a Brier-plus-group-ECE calibration term to the GRPO reward cuts calibration error sharply and leaves accuracy metrics largely unchanged.Capability: Grounding answers in the image
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor 8B-14B generative reward models trained as judges, making the model generate input-adaptive evaluation criteria inside its own reasoning chain and rewarding the quality of those criteria with a stronger teacher model raises agreement with preference labels, and the gain scales with the teacher's capability rather than with the presence of the rubric step alone.
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-18· pending-reviewFor a 7B code model fine-tuned on competitive-programming tasks, training it to emit a fixed-schema requirement specification (goals, input/output constraints, examples, edge cases, dependencies) before the code raises Pass@1 more than training it to emit free-form reasoning or a stepwise implementation plan of similar length, with the largest margin on an out-of-distribution benchmark.Capability: Generating and editing working code
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewFor a 7B instruction-tuned transformer under a fixed KV-cache budget, training a small policy to pick which cached tokens to keep by rewarding low KL divergence from the full-cache next-token distribution retains more long-context accuracy than attention-heuristic eviction (H2O, SnapKV), and the margin widens as the budget tightens and as context grows.Capability: Losing information in long inputs
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor a domain-specific language essentially absent from pretraining corpora (FORM symbolic algebra), QLoRA fine-tuning an 8B model on a few thousand examples validated by executing them through the language's own interpreter beats frontier open-weight models of up to 756B parameters given syntax documentation in context, on execution rate and strict output match for the larger benchmarks, while general reasoning scores drop under 3 points.Capability: Generating and editing working code
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewFor a fixed 7B code backbone on short Python tasks (MBPP), task-aware prompt routing plus LoRA fine-tuning on prompts rewritten into the routed style raises pass@1 only slightly (+1.1 points) when one sample is returned, but yields large gains (+9-10 points) once several candidates are sampled and selected by execution against the task's visible public tests.Capability: Generating and editing working code
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor a frozen LLM run in a generate-critique-revise loop with no external feedback, the accuracy gain is concentrated where single-shot reasoning is already unreliable: on clean multi-step reasoning (BBH, ~74% single-shot) depth makes no difference beyond the confidence interval, while on harder math (MATH, 26.2% single-shot) the same loop gains +14.2 points, with GSM8K in between (+4.2).Capability: Fixing its own mistakes
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor a frozen-backbone multi-agent scientific discovery pipeline on HypoBench tasks, distilling reusable procedural skills separately per stage — from discriminator critique of a whole hypothesis batch for generation, and from execution outcomes compared to ground-truth results for testing — beats no skill, an LLM-written skill, and a hand-written skill, and removing each stage's feedback signal costs most of the gain.Capability: Improving its own scaffolding from failures
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewFor a frozen-weight video agent on open-ended surgical video QA, mining candidate retrieval skills from the agent's own low-scoring traces and keeping each only when it raises a four-dimensional validation reward on ~40 held-out questions beats installing a hand-written general 'looking discipline' skill, and the single distilled temporal skill (re-read the window adjacent to a localized event for before/after questions) carries most of the gain.Capability: Improving its own scaffolding from failures
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewFor a same-family pair of reasoning and instruction-tuned 4B models, learning one interpolation ratio per module (attention block, FFN block, embedding, final norm) with a reward that scores correctness and ranks correct responses by length cuts average generated tokens by about 27% relative to the reasoning model while holding average accuracy, and beats fixed global-ratio merging and CMA-ES search over the same ratios.Capability: Cutting to the chase
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor a small agent policy (Qwen3.5-4B) repeatedly deployed on the same task family with environment-level feedback, distilling only the trajectories whose textual-state guidance is newly or persistently useful relative to state-free execution raises the state-free policy's accuracy more than distilling all state-conditioned trajectories, and the resulting policy keeps improving over alternating rounds of state editing and consolidation.Capability: Improving its own scaffolding from failures
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor adapting Qwen3 instruction-tuned models from 0.6B to 32B to tool-calling corpora, LoRA supervised fine-tuning gives the best in-distribution exact-match accuracy (best in 15 of 18 settings, never below zero-shot), while GRPO holds only a sub-one-point average edge on cross-dataset transfer and SFT followed by GRPO under a matched data budget is rarely best at either.Capability: Using the tools it is given
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor agentic retrieval-augmented multi-hop QA with 7B-class policy models, scoring each intermediate step for both logical validity and grounding in the retrieved passages — rather than scoring steps by the final answer — yields higher EM/F1 than outcome-reward and final-answer-based process-reward training, because it removes preference pairs where flawed retrieval happened to reach the right answer.
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor agentic tool calls where a parameter is fabricated rather than grounded in the dialogue, sampling-based uncertainty detectors (token entropy, semantic entropy, SEP) fail at near-chance because the model fabricates plausible parameters with high confidence, while the model's hidden states at the moment the parameter is generated do carry a separable grounding signal.Capability: Stating false facts confidently
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-21· pending-reviewFor agents working over heterogeneous data, exposing a schema/content/tool ontology as an MCP server the agent queries, and refining it by a loop of diagnose, attribute-to-level, typed patch, and a gate that accepts an edit only if a backbone-conditional paired evaluation shows a margin, raised trajectory accuracy on DDR-Bench by an average 17.8 points across six backbones and text-to-SQL EX on BIRD by about 7 points; injecting the same semantic content as a static prompt did not help consistently and cost 15 points on one backbone, and in ablation removing the acceptance gate was the largest loss (-11.2), then attribution (-6.3), diagnosis (-4.8), typed patches (-1.7).Capability: Improving its own scaffolding from failures
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor an 8B Llama backbone under a fixed 256-slot cache budget, replacing hard KV eviction with a learned soft write that blends a new token's key/value into a routed fixed-size memory slot retains far more long-context retrieval accuracy than sliding-window or sink caches, and unlike full-attention or quantized caches it does not run out of memory at 128K tokens.Capability: Losing information in long inputs
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor an LLM acting as orchestrator of dynamically created subagents, summarizing its own past trajectories into reusable workflow skills, short operational tricks, and capability cards for the candidate subagent backbones, then injecting those into the orchestrator's context, raises task scores on delegation- and aggregation-heavy tasks, and the subagent model cards are the component whose removal costs most.Capability: Getting more from several models than from one
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor automated unit test generation on domain-specific Java libraries (NLP toolkits), injecting LLM-written test snippets into a search-based generator's population when the evolutionary search stagnates yields higher branch and line coverage than either the search tool alone or the LLM alone, with the LLM contributing object-construction sequences and domain-shaped string inputs that random search does not reach.Capability: Generating and editing working code
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-15· pending-reviewFor behaviour-preserving whole-repository tasks such as stack migrations, where the starting repository already passes its test suite, a fixed behavioural test suite cannot distinguish a completed migration from an untouched or shim-wrapped repository, so adding more tests does not close the gap and a non-behavioural audit with veto power is required.Capability: Whether the measurement made the finding
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor best-of-N selection on mathematical reasoning, a small verifier (~2M parameters) that reads the frozen generator's own hidden states at reasoning-step boundaries ranks correct candidates above incorrect ones better than a larger text-only verifier that re-encodes the generated solution, because the correctness signal is already present in the generator's upper-layer representations and must otherwise be inferred indirectly from emitted tokens.Capability: Spotting an error in a given artifact
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewFor binary visual question answering with open LVLMs (InstructBLIP-7B, Qwen2.5-VL, InternVL-4B), a hallucinated answer can be held with high response consistency, so a single consistency/uncertainty score misses it; separating consistency into four probe groups (image vs. text perturbation crossed with affirmative vs. negated statement) and classifying on the joint within- and between-group agreement pattern detects hallucination better than any single-score black-box or white-box baseline.Capability: Stating false facts confidently
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewFor black-box hallucination detection using only sampled responses and token log-probabilities, semantic-entropy signals go to zero whenever the samples collapse into one semantic cluster — which happens for a large share of hallucinated queries, up to nearly all of them on tasks with confusable entities and figures — while token-level uncertainty still separates many of those cases, so the two signals fail on different queries and combining them detects more than either alone.Capability: Stating false facts confidently
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewFor black-box jailbreak defense at test time, abstracting each successful attack into a rule that names the structural wrapper (nested role-play, code obfuscation, flipped text) rather than the harmful topic lets one rule induced from a single prompt cover the rest of that attack family, so attack success falls to near zero after the first batch of ~20 streamed prompts without any parameter updates.Capability: Improving its own scaffolding from failures
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewFor CAE simulation tasks (OpenFOAM, FEniCS, PyChrono, COMSOL) run with a current general-purpose coding-agent harness and a strong backbone, a single agent with no multi-agent role decomposition and no scripted self-critique matches or exceeds the published scores of multi-agent specialized CAE systems, because the harness itself supplies the multi-turn reasoning and execution-feedback repair the decomposition was built to compensate for.Capability: Getting more from several models than from one
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-18· pending-reviewFor claim-level hallucination detection on long-document QA, estimating a claim's reliability from how central it is in a graph of sampled model responses cannot separate true claims from confidently repeated false ones, because both get high agreement; adding trusted source documents as anchor nodes with higher-weight edges shifts the basis from internal consensus to external grounding and yields a Grounded pool with few hallucinations (97.6% precision on QASPER), at the cost of conservatively leaving correct-but-novel claims on the boundary.Capability: Stating false facts confidently
- mechanismreplicated↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewFor closed-book hallucination detection, training a small supervised classifier over many heterogeneous uncertainty scores beats the best single score, but the gain comes from the diversity of sampling-based black-box signals: ensembles over only single-generation token-probability features add little, while black-box-only ensembles do nearly as well as the full set.Capability: Stating false facts confidently
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewFor closed-box per-claim confidence built from self-verification and cross-sample consistency, the signal breaks on adversarial false-premise questions because it faithfully reports the model's own mistaken confidence: on TruthfulQA with GPT-4o claim-level calibration reached ECE 0.412 against 0.118 for response-level temperature scaling, while an active cross-examination protocol (LMvsLM) reached 0.112, so an external probe rather than a confidence estimate is what helps there.Capability: Saying it does not know
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor code generation benchmarks, deciding whether a benchmark item leaked into a model's training set is done better by a classifier over code-specific signals (test pass rate, CodeBLEU, edit distance, perplexity, CodeBERT embeddings of generated vs reference code) than by perplexity alone, because memorized items show abnormally high functional correctness and similarity to the reference solution, not just low perplexity.Capability: Whether the measurement made the finding
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-04· pending-reviewFor coding-agent robot manipulation where a safety constraint (do not touch a named obstacle) is given only as text in the task prompt, the agent restates the constraint early and then plans the shortest path through the obstacle, and enforcing the same constraint with an external stateless verifier that re-checks each route against grounded obstacle boxes before motion raises both task success and collision avoidance.Capability: Turning plans into physical action
- mechanismmechanism reasoning↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewFor collections of template-generated pages, a delimiter-based extraction pattern induced from a handful of labelled examples can be reused across the rest of the collection, so extraction cost per page drops to pattern execution rather than per-page inference.Capability: Writing a reusable locator: regex, SQL, XPath
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewFor computer-use agents operating on a real OS, indirect-injection success rates measured with a fixed hand-written injection template understate vulnerability, because an attacker that adapts phrasing from the agent's own refusal trajectories can reach substantial success on models the fixed template never penetrates.Capability: Following instructions hidden in data
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor conversational long-term memory with 8-14B backbones and plain top-K dense retrieval, writing each dialogue span into several complementary low-overlap semantic views before indexing retrieves gold evidence for more queries than structured extraction into a single view, and the advantage is largest at small retrieval budgets.Capability: Remembering across sessions
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewFor data-science Python tasks where several correct solutions differ by orders of magnitude in runtime, a model's correctness rate does not predict the efficiency of the code it writes: rankings invert between the two, code-specialized open models are less efficient than general open models of similar size class, and the fastest model rotates by library (NumPy vs Pandas vs PyTorch vs Polars).Capability: Generating and editing working code
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor deciding when a robot skill has finished and the next subtask should begin, general-purpose VLMs prompted zero-shot are near useless on long-horizon mobile manipulation (under 4% of transitions detected on BEHAVIOR-1K, including a frontier proprietary model), while a sub-1B student fine-tuned on compact self-explanations condensed from teacher reasoning over annotated demonstrations detects about 90% and lifts closed-loop task success from 0.30 to 0.53 against a 0.63 oracle.Capability: Turning plans into physical action
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor decoder-only 360M-8B instruct models, adding application-supplied provenance labels as an architectural signal — origin embeddings plus a learned ring-indexed attention bias that suppresses assistant and user queries attending to untrusted-source keys — cuts indirect-injection attack success on unseen task domains and injection styles further than the same two-stage fine-tuning without the architectural components, even though both are equal in-distribution.Capability: Following instructions hidden in data
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewFor deepfake video detection with 7-8B open multimodal backbones, splitting the analysis into four separately prompted perspective agents (texture, lighting, motion, physics) whose reports a judge agent reconciles beats the same backbone doing chain-of-thought or multi-turn self-questioning over the same four perspectives, and the gain is not explained by extra turns or repeated frame access.Capability: Getting more from several models than from one
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewFor detecting findings-conclusion inconsistencies in oncology PET/CT radiology reports, a 15M-parameter BERT pre-trained on in-domain reports reaches higher balanced accuracy and lower false-positive rate than zero- or few-shot prompted 27B-70B open-weight LLMs, and LoRA fine-tuning the 70B model only matches the small model rather than beating it.Capability: Spotting an error in a given artifact
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewFor detecting hallucinated text spans in vision-language model outputs, span localization and confidence calibration come from different architectures: a fine-tuned generative VLM proposes better span boundaries but scores them poorly, while a discriminative token tagger trained with a direct regression loss on human-consensus confidence gives better-correlated probabilities but conservative spans, so using the generative spans as proposals and the tagger's probabilities as scores beats either alone.Capability: Stating false facts confidently
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewFor detecting ungrounded object mentions in open-ended captions from white-box vision-language models, a single internal support signal (token likelihood, attention share, or image-text similarity) confuses support that comes from object-specific image evidence with support that comes from scene plausibility or the generated text prefix; scoring both whether image tokens carry object-specific evidence and whether support is more image-derived than prefix-derived separates these and raises detection AUROC.Capability: Grounding answers in the image
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewFor embodied vision-language models trained as separate capability specialists (spatial, temporal, action guidance, state verification) from one backbone, consolidating them by TIES weight merging and then routed multi-teacher on-policy distillation retains more of each specialist's gain than either step alone or than mixed-data RL, which shows capability-specific biases.Capability: Turning plans into physical action
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor encoder-based hallucination/claim-faithfulness detectors moved from general-domain data to biomedical claim verification, which corpus the encoder was pre-trained on matters more than which data it is fine-tuned on: a PubMed-pretrained encoder fine-tuned on a few hundred SciFact pairs beat both a general encoder fine-tuned on the same pairs and a general encoder pre-fine-tuned on a large general hallucination set.Capability: Competence in a specialist domain
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor ensembles of heterogeneous VLMs answering multimodal multiple-choice questions, the entropy of a uniformly pooled distribution over semantic-equivalence classes of sampled responses ranks wrong answers above right ones better than majority-vote entropy or picking the single lowest-entropy model, and the advantage grows with ensemble size; the same signal measured at the subagent evidence layer of a multi-step multi-agent system only ranks system failures slightly above chance.Capability: Saying it does not know
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewFor entity-centric evidence-grounded QA and fact verification, substituting named entities with type-matched synthetic names and checking that the answer transforms correspondingly (back-maps to the original answer) detects ungrounded answers that self-consistency and evidence-sufficiency scores both miss, because synthetic names reduce the parametric activation that keeps a memorised answer stable under perturbation.Capability: Saying it does not know
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewFor extracting fields from vertical websites, decomposing XPath induction into separate sub-tasks over a few seed pages lets an LLM emit an XPath that generalizes to unseen pages of the same site, beating single-prompt CoT and Reflexion XPath generation by 14.84-23.86 F1 points, while calling the model only a handful of times instead of once per page.Capability: Writing a reusable locator: regex, SQL, XPath
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-18· pending-reviewFor factual paired-candidate QA where the model already holds the needed knowledge, hallucination can be detected by removing the query keyword that most drives the chosen answer and seeing whether the prediction systematically recovers to the alternative — erroneous answers show that recovery while correct answers merely become ambiguous — but the signal weakens when the knowledge is absent and largely fails on multi-step arithmetic reasoning, where errors are distributed across steps rather than bound to one keyword.Capability: Stating false facts confidently
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewFor filtering unsupported LLM-generated vulnerability reports in security triage, statistical machine-text detectors are the wrong instrument: they score provenance, not correctness, and human-written security reports are template-driven and low-perplexity, so they trip the same signal that flags machine text.Capability: Stating false facts confidently
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor financial valuation of listed companies, a two-stage domain adaptation — large-scale financial-corpus pre-training followed by institution-specific fine-tuning — lowers target-price error more than single-stage fine-tuning or zero-shot prompting of a general model.Capability: Competence in a specialist domain
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor frozen VLA policies executing repetitive, perceptually aliased manipulation tasks (counting pick-and-place), a progress memory that advances after each attempted action converts transient grasp or placement failures into persistent task-state errors, so writes must be gated on observed physical evidence of the intended transition; gating this way raised success far above the frozen policy and above memory-augmented baselines.Capability: Turning plans into physical action
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor grammar-constrained text-to-SQL decoding with small models (Qwen2.5-Instruct 0.5B-7B on Spider), beam search is at least as accurate as sample-and-vote at every matched compute budget of 1-8, and better in most, reversing the usual unconstrained finding that sampling with voting beats beam search.Capability: Writing a reusable locator: regex, SQL, XPath
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor image-aesthetics models fine-tuned for critique and scoring, competence at producing reference-aligned aesthetic critique does not carry over to judging whether an image suits a stated purpose, audience or cultural setting: all four aesthetic specialists tested scored below 30% accuracy on 301 expert-curated suitability cases (best 29.57%) versus 91.69% for the strongest general-purpose model, and they predicted 'suitable' in 59-77% of binary cases against a gold rate of 38.14%.
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor image-borne injected instructions delivered to tool-using agents, completion rate understates exposure: agents act on the injected instruction about ten times more often than they complete the objective (12.8% attempted vs ~1% completed across 720 runs), and most failures stop at the planning step where the model read the payload and simply did not act on it rather than at any architectural barrier.Capability: Following instructions hidden in data
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewFor inducing regular expressions from a handful of labelled examples (entity spans in Czech clinical notes), prompting an LLM and then iteratively refining the pattern against match failures extracts entities more accurately than genetic-programming regex induction, with the margin largest when few examples are available.Capability: Writing a reusable locator: regex, SQL, XPath
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor instruction tuning an 8B open model on text-to-SQL, a curated set of under 900 examples selected to cover every table in the target schemas and score well on question-SQL alignment reaches higher execution accuracy than pipelines trained on tens of thousands to millions of samples, and dropping the schema-coverage term from the selection objective costs more accuracy than dropping any single quality term.Capability: Writing a reusable locator: regex, SQL, XPath
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewFor instruction-tuned transformer LLMs (1.7B-70B) given structurally impossible math or code prompts (division by zero, invalid attribute calls), the pre-generation hidden state carries a linearly readable 'no admissible answer' signal that a single residual-stream direction recovers (mean AUC 0.939 over 22 model-dataset cells), yet that direction is near-orthogonal to the canonical safety-refusal direction (mean cosine 0.087), so answering confidently is a routing failure rather than missing recognition.Capability: Saying it does not know
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor large audio language models on multiple-choice audio reasoning benchmarks (MMAR, MMAU-mini), accuracy does not track whether the model's chain of thought is acoustically grounded or causally drives the answer: several models answer correctly from perception steps that assert unsupported labels, and for some models replacing the reasoning chain with vacuous or fabricated content leaves the final answer unchanged.Capability: Understanding speech and audio
- mechanismmechanism reasoning↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor LLM agents acting in partially observable environments, replacing the raw action-observation log in the prompt with an externally maintained Bayesian posterior over latent states makes the agent's action choice a function of the belief alone, removing surface-form sensitivity in which two histories with identical posteriors but different serialized text elicit different actions.Capability: Planning a sequence of actions before acting
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor LLM agents acting through stateful tools over many turns, unsafe policy-violating completions accumulate with the interaction budget and are roughly doubled when the adversarial user conditions each next request on the agent's actual prior response, so single-turn or short-prefix safety evaluation understates the failure rate.Capability: Using the tools it is given
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor LLM agents classifying nodes in text-attributed graphs, serializing sampled neighborhoods into prompt text makes the prediction depend on the arbitrary order of neighbors; replacing that text with a fixed-size, permutation-invariant vector of graph statistics (degree, PageRank, clustering, cross-community fraction, label entropy, majority-label fraction) plus only the top-k semantically relevant node texts raises accuracy and cuts per-message tokens.Capability: Getting more from several models than from one
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewFor LLM agents doing active clinical diagnosis in interactive patient environments, wrapping the model in a scaffold of role-specialized agents, evidence tools and an explicit evidence state, with a runtime observer that tracks uncertainty and evidence coverage to decide when to stop acquiring evidence, raises both diagnostic accuracy and execution stability over static case-to-answer prompting, with the size of the gain depending on the backbone model.Capability: Using the tools it is given
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewFor LLM agents on dependent multi-step tasks, success decays geometrically with the number of steps under a single per-step reliability parameter that rises with model scale but stays below 1, so long horizons eventually collapse even for the strongest deployed systems — and the decay tracks step count, not context length, since bounding the context window steepens the decay rather than easing it.
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor LLM agents on long-horizon web penetration testing, holding the backbone model fixed and moving global state out of the context window onto a persistent graph of verified facts and fact-conditioned intent edges raises task success most on the hardest tasks, where success depends on facts obtained dozens of steps earlier (88.2% overall and 75.0% hard, vs 44.1% and 25.0% for the strongest shared-plain-text-memory baseline).
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewFor LLM agents playing thousand-step StarCraft II matches through a high-level text interface, retrieving a handful of time-aligned episodes from a small bank of past winning games and gating between replaying a retrieved action and reasoning afresh over recent observations raises win rate over a summarization-only working-memory baseline at roughly a seventh of the token cost, with the margin widening at the harder difficulty level.
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor LLM agents that must compose and then execute domain tool workflows, training separate adapters for the two roles splits the gains cleanly: supervised fine-tuning of the planner on reference tool sequences mainly improves workflow-order fidelity, while node-level RL on argument-match rewards mainly improves tool-argument grounding and final-answer quality, and combining both beats either on a 9B backbone.Capability: Planning a sequence of actions before acting
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor LLM agents that score their own actions during exploration in text-based games and then reuse that experience, the accuracy of those self-scores against the environment verifier does not predict later gains on held-out stricter configurations: block-level correlations between judgment agreement or calibration and the next checkpoint's score change sit near zero, so recognizing a good action is not the bottleneck — converting it into a transferable policy is.Capability: Improving its own scaffolding from failures
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewFor LLM agents under a fixed ReAct harness answering questions over executable database-backed APIs plus document retrieval, accuracy falls sharply as the required reasoning depth grows from 1 to 5 hops (most models lose over half their accuracy), and the surviving failures concentrate in language-mediated steps — entity disambiguation, cross-source grounding, schema alignment — rather than in emitting syntactically valid tool calls.
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewFor LLM agents whose steps condition on their own prior outputs, per-step confidence that is calibrated against step-local success labels does not compose into a calibrated estimate of whether the whole trajectory succeeds, because correlated step errors make the product of step marginals diverge from joint success probability, and the divergence grows as prefix reliability falls over longer horizons.Capability: Saying it does not know
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewFor LLM data-preparation pipelines in specialist domains (math, finance, law, medicine), diagnosing a small pilot set of generated samples against domain-selected quality criteria and rewriting the individual pipeline components (chunking, instruction builder, response generator) before full-scale generation yields training data with higher downstream fine-tuning utility than post-hoc filtering or one-shot agent/skill generators using the same backbone — but the gain is non-monotonic in rounds, peaking at two and degrading after.Capability: Improving its own scaffolding from failures
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewFor LLM multi-agent systems failing on long-horizon web tasks, re-running the whole task from scratch — plain, with self-reflection, or with a critic agent — repairs few failures (at most 6.90% within three attempts, with feedback variants below plain rerun), while reconstructing the recorded execution prefix and intervening once at a symptom-flagged trace node repairs 20.15% with a single attempt.Capability: Getting more from several models than from one
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor LLM program-repair workflows built on self-correction, generated-knowledge or multi-agent scaffolding, the gain over plain prompting is unstable across benchmarks and can turn negative once extrinsic defect information (error messages, triggering tests, bug comments) is withheld, while simply putting the error message plus the failing test into the prompt yields more correct repairs than the scaffolds do.Capability: Fixing its own mistakes
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor LLM web agents on cluttered pages containing many concurrent interface components, success falls sharply as component count grows, and having the agent first emit a localized sub-goal plus a task-relevant sub-observation before choosing an action recovers much of the loss, because self-assessment of the action is more reliable in the reduced context.Capability: Using the tools it is given
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor LLM-based forecasting systems, reported benchmark gains often cannot be attributed to temporal reasoning, because test periods predate the training cutoff, ablations show the LLM component sometimes adds nothing over the statistical or retrieval parts, and accuracy shifts under small input perturbations.Capability: Predicting future events
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-18· pending-reviewFor LLM-driven heuristic discovery on combinatorial optimization tasks (TSP, CVRP, bi-objective and multi-component variants), letting an agent decide when to read, edit, diagnose and evaluate code while persisting each evaluation outcome as a retrievable experience card reaches better final heuristic quality than fixed evolutionary pipelines (EoH, ReEvo, FunSearch, MCTS-AHD, MEoH, MOTIF) at a fraction of their evaluation budget, and re-running the same task warm-started from the previous run's best code improves further round after round under a fixed budget.Capability: Fixing its own mistakes
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor LLM-driven multi-UAV swarm tasks in simulation, binding one agent to each vehicle with a role-conditioned subset of the skill catalog and directed peer messages yields higher episode returns than a central planner that sees pooled team state and the full skill list, and ablations attribute most of the gain to the skill-subset restriction and the peer reports rather than to the experience-based reward correction.Capability: Turning plans into physical action
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewFor LLM-driven robot manipulation where every tool call is schema-validated and every motion kinematically and collision-checked before execution, execution-time recovery mechanisms (retrying a failed plan with an alternative candidate, resetting to a collision-checked safe pose between stages) contribute more to end-to-end task success than canonical action sequencing, because most residual failures are transient planning or IK failures rather than wrong orderings.Capability: Turning plans into physical action
- mechanismreplicated↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewFor LLM-generated HLS kernels targeted at real FPGA boards, the two stages of failure respond to different interventions: an evolutionary search over simulation-validated kernel candidates mainly raises synthesizable and executable rates, while a C/C++-level functional verify-and-repair loop mainly raises functional correctness and barely moves synthesizability, so both are needed for end-to-end success.Capability: Generating and editing working code
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor LLM-generated HLS kernels targeting real FPGA boards, simulation and synthesis success overstates deployability, and the two stages are fixed by different interventions: an evolutionary search over simulated candidates raises synthesizable and executable rates while barely raising functional correctness, whereas C/C++-level verify-and-repair raises functional correctness while leaving synthesizability flat.Capability: Generating and editing working code
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewFor LLM-generated multi-day travel itineraries, high qualitative coherence scores (spatial, persona, meal, ordering) coexist with near-zero robustness to realistic disruption: across GPT-5, Qwen3, Mistral-7B and Phi-4-mini, transport-delay buffer scores stay under 12% and buffer-adequacy scores under ~41%, so coherence and static constraint satisfaction do not indicate that a plan survives empirical transit delays or crowd peaks.Capability: Planning a sequence of actions before acting
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor LLM-guided narrow-space parking where an optimal-control solver supplies pass/fail evidence, growing a decision knowledge base offline from accumulated solver and validation failures — retaining an entry only when re-planning then yields a feasible trajectory — raises planning success from 85.1% to 94.7% and closed-loop success from 74.6% to 84.2% over an empty knowledge base, with model weights frozen.Capability: Improving its own scaffolding from failures
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewFor locally deployed open-weight models driving a stateful, dependency-ordered MCP tool server, cutting tool descriptions from full specifications (purpose, parameter semantics, constraints, failure conditions) down to one sentence each raises the fraction of calls the server rejects for every model tested, while its effect on task coverage is less consistent.Capability: Using the tools it is given
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-18· pending-reviewFor long cooking videos where a question depends on ingredient changes no longer visible, vision-language models fail to carry observed events forward into later answers, and adding generic retrieved-context RAG makes it worse, while splitting memory into a temporally anchored event log plus an explicitly updated inventory ledger recovers much of the gap.Capability: Tracking state through a long task
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor long-horizon embodied task planning where a symbolic model defines action preconditions and effects, correcting a violated plan by edits the action model itself implies (inserting the missing navigate or open) raises plan success more per LLM call than returning the violation to the LLM for replanning, and the gain persists for frontier models as well as compact ones.Capability: Planning a sequence of actions before acting
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor long-horizon LLM agents in environments where an erroneous action changes world state, restoring an earlier checkpoint and carrying forward a structured reflection memory that excludes branch-local state claims beats either intervention alone, because reflection alone cannot undo environmental damage and rollback alone re-explores the same failure.
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewFor long-horizon LLM agents on tool-use and terminal benchmarks, token-level confidence (predictive entropy, sequence probability) barely separates successful from failed trajectories, and the separable signal lives in the dependency structure of the trajectory: scoring a trajectory by propagating per-step uncertainty over a graph of reasoning, tool-call and environment-feedback nodes raises failure-detection AUROC over both local-confidence and sequence-based agent UQ baselines.Capability: Using the tools it is given
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor long-horizon repository-level coding agents, a small (4B) critic trained to diagnose each just-completed step with a severity label, error type and causal rationale, and to redirect the next action when severity is critical, repairs more initially-failing tasks with a single branch (14.1% fail-to-pass) than five full resamples of the base agent (13.7%), at about one-fifth the base-agent output tokens.
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewFor long-horizon tool-calling agents (tasks needing five or more calls), embedding state-transition cues — preconditions, invariants, completion states — into the descriptions of the tools on the intended chain is what actually steers the agent's trajectory; runtime corrective text appended to tool results only patches residual drift, and the plausible user prompt alone (persona, deadlines, format constraints) does not establish the trajectory at all.Capability: Following instructions hidden in data
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewFor long-video QA under a fixed 32-frame budget with 7B-8B multimodal backbones, selecting frames from the structural shape of the frame-query similarity curve (peak, rising, falling, transition regions) beats ranking frames by similarity magnitude, and the margin grows with video length — a few points on short clips and roughly 8 points on hour-long videos — while on short/medium-clip benchmarks it is near-indistinguishable from other selectors.Capability: Reasoning about time in video
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewFor long-video QA under a fixed frame budget with a frozen 7B multimodal model, organizing frames into a question-agnostic hierarchical temporal tree and searching it coarse-to-fine (answer, then issue a search statement, then expand the relevant branch) raises accuracy over uniform sampling and over agent methods that score frames independently, with the gain largest on the longest-video benchmarks.Capability: Reasoning about time in video
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-18· pending-reviewFor long-video QA with a fixed frame budget fed to a frozen multimodal model, selecting frames by marginal contribution to the already-selected subset (query relevance plus coverage gain minus redundancy) beats ranking frames independently by query-frame similarity, and the margin over uniform sampling grows with video duration.Capability: Reasoning about time in video
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewFor long-video QA with a frozen VLM and fixed answering interface, an outer loop that proposes, executes and retains only development-accuracy-improving rewrites of the executable context-construction program can advance past a strong hand-crafted keyframe-selection harness, and the gain survives a matched cumulative visual-token control and direct reuse on other long-video benchmarks.Capability: Improving its own scaffolding from failures
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewFor LoRA-ensemble credal commitment rules on 7-9B instruct backbones, the decision to commit tracks agreement among the adapters rather than whether a request should be answered: on AdvBench harmful prompts the ensembles commit more often (33-97%) and with higher lower-bound confidence than on benign Dolly instructions, because safety training makes refusal the unanimous first continuation.Capability: Saying it does not know
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewFor manipulation policies (diffusion-policy and VLA backbones) trained on few demonstrations, decomposing trajectories implicitly via a Mixture-of-Experts gate and storing the per-segment gating profiles as retrievable memory, then blending retrieved profiles into the current gating distribution at inference, raises success most on unseen object-receptacle recombinations rather than on in-distribution tasks.Capability: Turning plans into physical action
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor manipulation policies pretrained on mixed egocentric-human and robot data, adding egocentric data helps only when the embodiment gap is bridged explicitly: a three-stage curriculum (ego priors, then task-matched human/robot alignment in a shared end-effector frame, then robot-only) raised real-robot success, while separate ego/robot action heads gave only a marginal in-distribution gain and shared future-video prediction in a world-action model made success worse than a robot-only baseline.Capability: Turning plans into physical action
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor manipulation policies whose large VLA backbone cannot run within the control period, letting the slow model propose action chunks from a stale observation while a small RL-trained edit policy adjusts and Q-selects among those chunks using the latest observation recovers more performance than applying RL on top of asynchronous imitation-learning schemes alone, and the gap widens as inference delay or environment speed grows.Capability: Turning plans into physical action
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-15· pending-reviewFor marine vehicle navigation, an LLM asked only to emit an ordered macro-action skeleton produces syntactically and topologically usable plans but cannot set physically feasible step durations; letting a differentiable physics surrogate optimize the durations turns plans that collide on every mission into collision-free ones, while wrong skeleton topology remains unrecoverable because the optimizer only rescales timings.Capability: Turning plans into physical action
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor medical multiple-choice QA with a multi-agent pipeline, mining natural-language lessons from its own past trajectories (both correct and failed) into a reusable library gives a larger accuracy gain than the multi-agent role decomposition itself, and the gain comes mostly from the most abstract tier: removing principle-level entries costs more than removing pattern- or case-level entries.Capability: Improving its own scaffolding from failures
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor memory-augmented VLA manipulation policies that must actively move the viewpoint to find task-relevant evidence, writing one memory frame at each fixed action-chunk boundary generalizes better than event-triggered writing (on predicted subtask change or on gripper/head/torso motion), and extra memory capacity helps only under the regular schedule — event-triggered policies degrade further out of distribution because mistimed triggers omit informative views and control jitter fills memory with redundant frames.Capability: Turning plans into physical action
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor monocular VLM-based navigation policies, making the model first emit a two-token pointing prefix (one affordance point for free space, one object point for the goal) on an image-plane grid and then three residual vector-quantized action tokens decoded to 10 SE(2) waypoints lets one compact 4B backbone handle instruction following, object-goal search and target tracking without task-specific action heads or task identifier tokens.Capability: Turning plans into physical action
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewFor multi-agent LLM pipelines on multimodal medical referral from real inpatient records, requiring agents to exchange typed logical edges (causal cascade, mutual reinforcement, exclusion, conditional context) in a shared reasoning graph instead of free-form dialogue raises referral F1, and the causal-chain edge type carries most of that gain.Capability: Getting more from several models than from one
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewFor multi-agent pipelines on knowledge-intensive QA with an 8B backbone, deciding the collaboration topology after retrieval and conditioning it on the retrieved evidence yields higher accuracy than generating the topology from query semantics alone with the same retrieved evidence available at execution.Capability: Getting more from several models than from one
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-15· pending-reviewFor multi-hop factual QA with Qwen2.5-7B/Qwen3-8B given a deliberately insufficient retrieved triple subgraph, training a policy with RL to insert bridging triples that let a separate frozen answerer reach the gold answer raises answer accuracy over standard GRPO and over prompting methods like Self-Ask and RECITE, and the improvement carries over to direct chain-of-thought answering with no triples inserted at test time.Capability: Stating false facts confidently
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor multi-hop QA over 28k-896k token haystacks with frozen Qwen2.5 backbones (7B-72B), summarizing the document in a parallel hierarchical tree — overlapping windows scanned ~5x, raw segments scored 'critical' passed straight to the final answer, and one summarization channel per decomposed sub-question — scores higher than sequential memory-agent and BM25 retrieval baselines without training, and beats an RL-trained memory agent on the out-of-distribution dataset while losing to it in-distribution.Capability: Losing information in long inputs
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor multi-modal robot motion planning stated in natural language, the hard part for an LLM is synthesizing the cross-modal coupling (meeting points, contact locations, shared intermediate states), not the single-modality formulation: giving an LLM fixed expert single-modal MIP modules and asking it only to predict the interface variables gives 100% single-modal success but 55.6-63.2% multi-modal, below a model fine-tuned with solver-verified rewards to write the joint formulation (80.9%).Capability: Planning a sequence of actions before acting
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor multi-step assembly task planning from an action history and a workstation image, prompting a multimodal LLM to emit a structured (JSON) state representation and propagate it through each predicted action — then truncating the plan at the predicted state closest to the inferred goal — raises action executability and lowers final-state error relative to predicting actions alone, at the cost of shorter plans (lower overlap with recorded human action sequences) and roughly eight times the latency.Capability: Planning a sequence of actions before acting
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor multi-task procedural video understanding with a small (1B) video-LLM decoder, replacing each decoder feed-forward block with task-specific expert copies and routing each video-prompt sample by its known task identity gives higher average top-1 accuracy across the five COIN tasks than the same expert topology run densely or with learned token-level or sample-level routing, with the largest gains on the procedure-forecasting tasks.Capability: Reasoning about time in video
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-18· pending-reviewFor multi-turn agentic coding sessions, constraints written into a repository policy file (e.g. CLAUDE.md) hold up better over successive turns than the same kinds of constraints stated in user turns, and constraints that replace an earlier constraint are followed less often than newly added ones.Capability: Following an unfamiliar procedure
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-15· pending-reviewFor multi-turn agentic RL on sparse-outcome text environments (ALFWorld, WebShop) with Qwen2.5-1.5B/7B-Instruct, reading state and action values from two reserved vocabulary logits at the causal boundaries of the policy's own backbone, and assigning credit with turn-level GAE plus batch normalization, raises task success over PPO with a separate critic and over GRPO while sampling only one rollout per task.
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor multi-turn tool-using service agents, counting excess user-facing turns above a task-specific human reference predicts human efficiency preferences between two successful trajectories (about 91% accuracy on turn-differing pairs), while counting excess tool calls does not (chance on pure-tool pairs), because a raw call count cannot say whether a call verified a constraint or wasted effort.Capability: Using the tools it is given
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewFor multimodal LLMs, a behavioral perturbation signature (answer/confidence change under visual degradation, blank-image replacement, and grounding/relation probes) predicts which failure family an error belongs to far better than scalar confidence, yet folding the same signature into confidence as a direct penalty makes answer-versus-abstain ranking worse than scalar confidence alone.Capability: Saying it does not know
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewFor open models (8B-14B) filling structured tool-call parameters, a linear probe on the hidden state at the position just before a parameter value is written predicts whether that value will be correct better than token log-probability (in-domain AUC about 0.986 vs 0.914), and using that score to filter self-generated calls for fine-tuning beats self-training filtered by log-probability or majority vote.Capability: Using the tools it is given
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor open-ended emotion interpretation by multimodal LLMs, coarse-grained hallucination mitigations designed for object hallucination (VCD, ICD, M3ID) and self-verification (EmotionHallucer) cut unsupported claims in some facets while raising them in others — e.g. lowering audio hallucination but increasing expression and action hallucination — so a single aggregate hallucination rate hides the trade-off.Capability: Stating false facts confidently
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor open-source LLM agents driving a ROS robot, having the model emit a pseudo-code tool-call plan up front, storing it in a persistent scratchpad, and then restricting each tool invocation to the plan's next step raises task success most on the longest multi-step tasks and for the smaller open models (GPT-OSS-20B on the hardest task rose from 35% with a persistent plan alone to 55% with the constraint), while a strong proprietary model gains little.
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-18· pending-reviewFor open-source VLMs asked to justify answers on object-removal counterfactual image pairs (A-OKVQA, SNLI-VE), explanations written after the answer is committed (Predict-then-Explain) mention the removed object more in proportion to how much its removal shifts the answer distribution than pre-answer step-by-step CoT does, so pre-answer reasoning is not the more faithful format here.Capability: Saying what actually drove the answer
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewFor open-weight agents where attention matrices are accessible, detecting injection by asking which context span actually drove the tool call — aggregating attention over the decision tokens and localizing the guiding span, then checking whether that span's provider has authority for the action — catches unauthorized tool invocations at higher true-positive and lower false-positive rates than static scanners that look for malicious-looking text, because benign-looking injected text is only harmful when it is what the model actually attends to.Capability: Following instructions hidden in data
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-18· pending-reviewFor panoramic dental X-ray VQA with frozen vision-language models, replacing the model's own spatial judgement with external detector findings that are deterministically bound to tooth identities before synthesis raises answer accuracy over the same two-pass pipeline without detectors, and the resulting accuracy is bounded by detector quality — degrading the identity/attribution detector costs more downstream accuracy than a much larger degradation of a task-specific pathology detector.Capability: Grounding answers in the image
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewFor persistent-memory agents, any memory-write rule that decides using only recency and provenance is stuck on a single tradeoff — accepting more genuine preference updates means admitting more poisoned ones — because an adversary who can launder a claim through the user's own channel matches the statistics a genuine revision produces; conditioning the write decision on the inferred authenticity of the claim given the interaction history moves both axes at once.Capability: Remembering across sessions
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor post-training a pretrained vision-language-action manipulation policy under a fixed per-round frame budget, collecting handheld demonstrations only at states where the human action chunk diverges from the policy's sampled action distribution, and weighting them by a learned task-progress advantage, raises task progress more than collecting the same number of frames of ordinary demonstrations and fine-tuning on them.Capability: Turning plans into physical action
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewFor precise anomaly localization in long multivariate time-series (16k-131k points), keeping the signal outside the model context as a variable the model queries through code over multiple turns recovers far more localization accuracy than single-pass ingestion, where the recording is serialized, subsampled or compressed into context once.Capability: Losing information in long inputs
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor pretrained 2D vision-language-action manipulation policies, adding 3D geometry only on the perception side is not enough: conditioning the action denoiser directly on a raw 3D semantic point field — at the specific diffusion blocks where linear probes show end-effector-coordinate error rising — raises bimanual task success beyond perception-side 3D fusion alone, and encoding the point cloud with a PointNet++-style encoder instead of keeping raw points lowers success.Capability: Turning plans into physical action
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewFor ReAct agents on AppWorld, resampling each recorded inference point offline (N=30, temperature 0.5) to find steps whose responses vary across samples, then writing short natural-language guidelines targeting those steps into retrievable episodic memory, raises the fraction of tasks passing all five repeat runs by +16 points same-task and +13 points similar-task with GPT-4.1 (+6 and +8.7 with GPT-OSS-120B) without lowering mean per-run accuracy.Capability: Improving its own scaffolding from failures
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-18· pending-reviewFor real-world online RL post-training of large pretrained vision-language-action policies on high-precision manipulation, optimizing the action head toward a paired advantage margin against a frozen reference action, rather than maximizing absolute Q values, yields higher autonomous success on long multi-step tasks, because absolute-Q maximization amplifies early optimistic value errors into policy drift that behavior-cloning regularization alone does not stop.Capability: Turning plans into physical action
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewFor repair of production crash bugs in large Java enterprise systems, feeding an LLM the source of the top-3 suspicious files plus a representative stack trace localizes the developer-modified code in about 71% of responses and yields a semantically valid patch in about 52%, and adding an explicit ranking of suspicious files and methods helps only when the full stack trace is withheld (per-response repair 28% to 47% with stack messages only) while slightly lowering performance when the stack trace is already present.Capability: Finding and fixing a fault in existing code
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-18· pending-reviewFor response-level hallucination detection against reference documents, having the model first convert the reference into a relational SQL database and then check response claims by executing SELECT queries raises F1 over zero-shot direct prediction mainly by cutting false positives, and the gain shrinks or vanishes on backbones with weak text-to-SQL ability.Capability: Stating false facts confidently
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewFor response-level hallucination detection in retrieval-augmented settings, having the model first convert the reference documents into a relational SQL database and then check the response against SELECT query results raises precision and macro F1 over zero-shot direct prediction, over atomic-claim decomposition, and over flat JSON key-value extraction, without any task-specific fine-tuning.Capability: Stating false facts confidently
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewFor RLHF-tuned chat models the raw conditional probabilities are consistently overconfident, and a confidence the model states in words or as a number in its output is better calibrated, often halving expected calibration error on TriviaQA, SciQ and TruthfulQA; asking the model to list several candidate answers before giving confidences improves it further, while chain-of-thought does not.Capability: Saying it does not know
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor routing a video-production request to an ordered chain of registered sub-agents, an 8B open model fine-tuned on descriptor-conditioned routing plans (SFT on verified plans plus GRPO rewarding validity, sub-agent coverage F1 and dependency-consistent order) picks chains more accurately than much larger prompted planners, with the gap widest on long, structurally unfamiliar chains.Capability: Getting more from several models than from one
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-07· pending-reviewFor safety guardrail classifiers, long-context failure is caused by proportional dilution of the unsafe evidence rather than by absolute input length: as the flagged span's share of the input shrinks, attention mass on it falls, the unsafe-minus-safe logit margin compresses, and unsafe recall collapses, while filling the same length with copies of the unsafe text leaves recall largely intact.Capability: Losing information in long inputs
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor selecting among multiple generated SQL candidates, a listwise selector given retrieved structured selection criteria distilled from training question-SQL pairs, with candidates grouped by execution result and permuted only within groups, reaches higher execution accuracy than pointwise, pairwise and multi-selector baselines on the same candidate pools while using far fewer selector calls and tokens.Capability: Writing a reusable locator: regex, SQL, XPath
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor self-evolving agents that accumulate reusable skills in a bank and admit a new skill whenever aggregate task reward rises, the accepted skills can be collectively net-negative, because a bank-level reward change cannot identify which skill contributed and a skill's contribution depends on the other skills present and on the deployment domain.Capability: Improving its own scaffolding from failures
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewFor self-evolving LLM agents that admit a new skill into a persistent skill bank whenever aggregate outcome reward rises, the accepted skills are collectively net-negative and the gate cannot do better in principle, because the bank-level gain is one scalar that leaves every incumbent skill's contribution unconstrained; scoring sampled coalition knockouts at the gate, and masking transferred skills whose removal improves retrieval quality on unlabeled target queries, recovers the missing composition and domain dependence.Capability: Improving its own scaffolding from failures
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewFor short-answer object-existence questions in open multimodal models (LLaVA v1.5, MiniGPT-4, Qwen2.5-VL), object hallucination is driven partly by the visual encoder itself rather than only by language priors: hallucinated cases show lower image-text embedding similarity and inverted attention, and swapping in a weaker or noisier visual encoder lowers POPE accuracy while a stronger one raises it.Capability: Grounding answers in the image
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewFor short-answer object-existence questions in open multimodal models (LLaVA v1.5, MiniGPT-4, Qwen2.5-VL), object hallucination is driven partly by the visual encoder itself rather than only by language priors: hallucinated cases show lower image-text embedding similarity and inverted attention, and swapping in a weaker or noisier visual encoder lowers POPE accuracy while a stronger one raises it.Capability: Grounding answers in the image
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewFor single-agent procedural tasks where the relevant state fits a fixed schema known in advance, replacing the appended conversation transcript with a structured execution state that the model rewrites each step — discarding the reasoning trace once the state patch is validated — holds accuracy flat as the horizon grows while history-appending runtimes degrade, and the gain comes from the structure rather than the shorter prompt.
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor single-step tool-calling agents, near-ceiling accuracy on unperturbed calls does not predict robustness to perturbation, and corrupting or removing the tool's returned evidence hurts far more than perturbing tool names, descriptions, or the user request.Capability: Using the tools it is given
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewFor single-step tool-calling agents, near-perfect clean accuracy does not predict robustness, and the dominant weak point is recovering the answer from perturbed tool returns rather than selecting the tool or binding arguments: across seven models, clean success averaged 0.979 while success under corrupted or missing returned evidence averaged 0.455, with no model above 0.60.Capability: Using the tools it is given
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewFor small (4-8B) student agents on multi-turn tool-use tasks, transferring a stronger teacher's successful trajectories as flat task-level memory barely helps and can hurt, while splitting the same trajectories into a high-level workflow insight in prose plus concrete executable subtask examples raises accuracy sharply, with the subtask examples carrying most of the gain.Capability: Using the tools it is given
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewFor small instruction-tuned models (3-4B) answering closed-form true/false questions under a stated user opinion, scoring each sampled response by how much more common its answer is than the group itself predicted, and using that score as the GRPO reward, reduces answer flips under user pressure and raises accuracy without any labels, roughly matching label-supervised desycophancy fine-tuning at higher compute cost.Capability: Telling the user what they want to hear
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewFor small instruction-tuned models (3-4B) answering closed-form true/false questions under a stated user opinion, scoring each sampled response by how much more common its answer is than the group itself predicted, and using that score as the GRPO reward, reduces answer flips under user pressure and raises accuracy without any labels, roughly matching label-supervised desycophancy fine-tuning at higher compute cost.Capability: Telling the user what they want to hear
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor small open-weight backbones (Qwen3-4B) driving long-horizon remote-sensing tool chains, inserting a step-level verifier that checks each intermediate tool output against schema, sub-goal and domain constraints and triggers replanning is the single largest architectural gain, while hierarchical decomposition without domain-aligned tool grouping adds little and can lower final accuracy relative to a flat agent with the same verifier.
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-15· pending-reviewFor small open-weight models (0.6B-12B) on BFCL V3, presenting tool definitions as ordinary system-prompt instructions and having the model emit calls as normal assistant text yields higher function-calling accuracy than the model's native tool-call token/schema interface, with the largest gains in the smallest models and on parallel/multiple-call categories.Capability: Using the tools it is given
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-15· pending-reviewFor small open-weight models (1.5-4B) generating answers in a closed-domain RAG setup, hallucinated generations show a broken hidden-state proximity graph between the retrieved context entity and the generated entity — connectivity below 20% and path lengths above 7 hops, versus near-full connectivity at about 3 hops for grounded answers — giving a training-free detection signal (AUROC 0.89 on Qwen2.5-1.5B).Capability: Stating false facts confidently
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor small open-weight Qwen2.5 models (1.5B-3B) doing multi-step math reasoning, features derived from the spread of attention entropy across layers at each generation step predict whether the final answer is wrong better than output-logit confidence features, with the margin larger on harder problems (+0.0093 AUC on GSM8K, +0.0759 on MATH-500 at 3B).Capability: Stating false facts confidently
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewFor speech LLMs on long audio, text-centric KV cache eviction methods (StreamingLLM, SnapKV, PyramidKV, ChunkKV) collapse because raw audio tokens have diffuse attention; anchoring compression on ASR transcripts — pooling each transcribed audio span down to its text-token length and adding the text embedding, while keeping untranscribed audio intact — retains over 90% of full-cache accuracy at a 5-10% cache budget where those baselines retain roughly 55-67% at a looser 25% budget.Capability: Losing information in long inputs
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor spoken language models answering questions over long dialogue histories, converting the history to transcripts beats keeping it as audio when the answer is a lexical detail (exact slot values across dense turns: mean 71.3% vs 42.6%), but loses when the answer depends on how something was said, where audio history wins for six of seven models and by a larger margin once lexical emotion cues are removed.Capability: Understanding speech and audio
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewFor step-level verification of structured quantitative STEM reasoning, training a 7B process reward model only on hard negatives that are constructed to pass a deterministic symbolic verifier (swapped formula or unit-matched variable, with values recomputed so the math balances), and running the verifier as a hard filter before the PRM scores candidates, detects flawed steps better than monolithic PRMs that must judge arithmetic and logic together.Capability: Spotting an error in a given artifact
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewFor streaming video question answering with a frozen video-LLM, appending retrieved historical frames as extra visual context helps less than optimizing a small set of latent memory tokens at test time and then dropping the retrieved tokens before decoding; the gain is largest on questions requiring backward tracing over past history.Capability: Reasoning about time in video
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewFor streaming video understanding with 7-8B multimodal backbones, question-relevant historical frames can be retrieved from the keys of only the first few language-Transformer layers (layer ~3-4 of 28-32), so continuous frames need not be prefilled through full model depth: indexing shallow and re-prefilling only the retrieved units at full depth cuts per-frame prefill up to 52x and 20-second end-to-end latency up to 15x while scoring about 1.7 points above the best same-backbone baseline on OVO-Bench and StreamingBench.Capability: Reasoning about time in video
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewFor structured extraction from text files, web pages and spreadsheets, a few user-highlighted example fields are enough to synthesise a reusable extraction program that pulls the remaining fields without further interaction — the locator can be inferred from examples rather than written or re-invoked per record.Capability: Writing a reusable locator: regex, SQL, XPath
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-18· pending-reviewFor sub-2B open-weight models emitting JSON tool calls from prompts that already name the tool and all argument values, reported success depends almost entirely on a post-hoc recovery layer that strips code fences and extracts a brace-delimited substring: across 1,000 generations only 5 whole raw responses parsed as JSON directly, while the best condition reached 79% after recovery.Capability: Using the tools it is given
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewFor synthesizing regexes from user intent, neither a natural-language description nor positive/negative examples alone is sufficient — descriptions are ambiguous and small example sets underdetermine the pattern — so a system that parses the description into a structural sketch and uses it to guide example-based search solves far more tasks than either modality alone (80% vs 43% NL-only and 26% examples-only on 300+ regexes).Capability: Writing a reusable locator: regex, SQL, XPath
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewFor synthesizing reusable XPath extractors from a few sampled web pages, a two-stage pipeline — first having a cheaper model extract both the target value and the neighbouring cue text from an attribute-stripped HTML tree, then having a stronger model write the XPath over HTML condensed around those located nodes with an execution-feedback loop — reaches usable extraction F1 on multi-domain product/listing pages while keeping prompt size far below feeding raw pages.Capability: Writing a reusable locator: regex, SQL, XPath
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewFor target-price valuation of listed companies, a multimodal financial model given large-scale financial-corpus pre-training followed by institution-specific fine-tuning has lower mean absolute valuation error than a general model prompted zero-shot, a single-stage fine-tuned financial LLM, and an analyst-parameterized DCF spreadsheet.Capability: Competence in a specialist domain
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor teams of up to 50 heterogeneous embodied LLM agents in simulated wildfire missions, building the manager hierarchy per mission so that concurrently executable work sits under a parallel-allocating manager and prerequisite-chained work under a phase-advancing manager beats fixed decentralized, single-planner, and hybrid coordination architectures on mission score and on progress accumulated per step, and the advantage does not depend on which base language model is used.Capability: Getting more from several models than from one
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-18· pending-reviewFor text-to-CAD program generation, final-geometry similarity (IoU against a reference solid) carries almost no information about whether the generated program preserves the intended construction process: on multi-entity Text2CAD samples with Qwen3, sample-level IoU and a constraint-satisfaction rate over expected entities, Boolean roles, sketch properties and inter-entity relations correlate near zero (Pearson r = 0.022, Spearman 0.023), and high-IoU programs can omit expected entities or invert additive/subtractive roles.Capability: Whether the measurement made the finding
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-18· pending-reviewFor timeseries-encoder-fused 8B LLMs answering questions that require chaining several temporal primitives, supervised fine-tuning on worked decompositions only helps on composition families whose decompositions were demonstrated, while RLVR with outcome-verifiable rewards on top of that checkpoint also lifts composition families that received no demonstration (+20.1 pp) and held-out compositions (+9.6 pp).Capability: Following an unfamiliar procedure
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor tool-augmented legal agents on multi-step Chinese-law tasks, a correct final answer does not indicate a clean reasoning trajectory: across 18 agent configurations, trajectories that reach the right answer still contain at least one substantive legal hallucination about 68% of the time on average, and that rate barely moves with backbone scale or agentic framework (0.61-0.78), while procedural-process hallucination rates vary widely (0.07-0.85).Capability: Stating false facts confidently
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewFor tool-calling agents that fabricate parameters never given by the user, a zero-initialized LoRA adapter run concurrently with the frozen base model and trained only to emit a localized verdict at an appended trigger token detects these ungrounded parameters better than sampling-based entropy methods, passive linear probes, and an equivalent-scale fine-tuned external judge, and naming the specific hallucinated parameter back to the agent recovers more of the task than a generic block.Capability: Stating false facts confidently
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor tool-integrated LLM agents where the attack works by adding attacker-controlled tools, overwriting the agent's visible toolset with the saved user toolset immediately before planning drives attack success to near zero across direct and indirect injection, memory poisoning and backdoor attacks, but its effect on benign task success is model-dependent — raising it for 7-9B open models and lowering it for GPT-3.5/GPT-4.Capability: Following instructions hidden in data
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-18· pending-reviewFor tool-using agents on long-trajectory benchmarks, an injection defense that audits each candidate action against a one-step-ahead predicted tool set holds attack success low while keeping task utility near the attack-free level, whereas defenses that fix a global plan or filter whole tool outputs lose utility because the predicted full trajectory almost never matches the executed one once environmental feedback arrives.Capability: Following instructions hidden in data
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-18· pending-reviewFor tool-using agents trained against adaptive indirect prompt injection, attack success measured against a single frozen adaptive attacker understates vulnerability compared with taking the per-case success union over a population of retained attacker checkpoints (0.25% vs 4.97% on the same 1,187 adaptive cases for the trained defender).Capability: Following instructions hidden in data
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-18· pending-reviewFor tool-using agents, a safety harness searched separately for each model-and-domain deployment (jointly evolving a natural-language policy and executable pre/post-tool-call code, screened by a fresh-context adversarial reviewer) reaches a better safety-utility trade-off than fixed expert-designed defenses ported unchanged across deployments, because domain semantics determine which safety relations and trajectory state must be checked while the victim model's failure behavior determines whether a check should be semantic or deterministic and where in execution it must run.Capability: Following instructions hidden in data
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewFor tool-using agents, enforcing injection defense as a deterministic provenance check on sensitive tool parameters — with the only model call reading the trusted user request and never fetched content — makes admission decisions invariant to how an injection is worded, whereas defenses that judge the agent's runtime plan or behavior with a model can be steered by reworded injections.Capability: Following instructions hidden in data
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewFor tool-using agents, restricting capabilities at the harness level after untrusted content enters the context — revoking or tightening the parameter envelopes of skills that lie on graph paths to deployer-defined forbidden states — cuts indirect-injection attack success further than prompt-level or per-action authorization defenses, and does so with no extra model calls or tokens, because enforcement reads the transition graph rather than the natural-language content.Capability: Following instructions hidden in data
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewFor tool-using LLM trading agents with a frozen backbone, periodically rewriting the system prompt from the agent's own decision traces and realized portfolio returns changes tool-use behavior reliably — code-interpreter calls rise from about 1 to 2.6-4.5 per day and regime-relevant metrics (VaR, EMA, RSI, signal normalization) that the fixed-prompt agent never computes start appearing — but the resulting risk-adjusted performance beats the fixed-policy agent in only about half the evaluated model-by-regime settings (best Sharpe and cumulative return among LLM methods in 3 of 6).Capability: Improving its own scaffolding from failures
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewFor training-free inference-time hallucination interventions in 7B-class vision-language models, increasing intervention strength (finer regional evidence partitioning, longer attention-routing windows, stronger textual perturbation) keeps lowering CHAIR object-hallucination scores while producing empty or reduced-coverage responses, so lower hallucination numbers at high intervention strength partly reflect the model saying less rather than grounding better.Capability: Grounding answers in the image
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-07· pending-reviewFor training-free inference-time hallucination mitigation in 7B vision- language models, reported gains on object-hallucination benchmarks are largely inseparable from reduced informativeness: hallucination rate and object recall/coverage move together, so a lower score can mean the model mentioned fewer visual entities rather than grounded better.Capability: Whether the measurement made the finding
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor training-free MLLM-driven navigation (instruction-following and object-goal) in simulated indoor environments, gating each planner-proposed action on evidential support, geometric reachability and subgoal consistency before dispatch — and refusing a stop request unless semantic, geometric and progress checks all pass — raises success rate over letting the planner act directly, but the stop check buys completion at some cost in path efficiency.Capability: Turning plans into physical action
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewFor translating English descriptions into regular expressions, a generic attention-based sequence-to-sequence model trained only on description/regex pairs matches a domain-engineered semantic parser on a small corpus (814 examples) and beats it by a wide margin once the training corpus is scaled to 10,000 pairs, with no semantic-equivalence machinery built in.Capability: Writing a reusable locator: regex, SQL, XPath
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewFor translating heterogeneous MOEA implementations from MATLAB/PlatEMO into tensorized GPU code, a multi-agent pipeline that reconstructs source semantics into a shared blueprint, generates several strategy-biased candidates in parallel, and repairs them from execution feedback converges more often than one-shot translation with the same LLM backend, on all three backends tested.Capability: Getting more from several models than from one
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-18· pending-reviewFor vision-language-action and world-action manipulation policies evaluated on LIBERO-style tasks, near-ceiling one-shot task success does not predict the ability to resume a task from a failure state that arose during the model's own execution: all six tested policies lost over half their success rate on recovery scenarios, rankings inverted between the two settings, and recovery success fell steeply as the failure altered object or environment state rather than just the last action.Capability: Turning plans into physical action
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor vision-language-action models pretrained on multi-embodiment video, latent actions learned only from RGB frame pairs miss finger-level end-effector articulation, and adding a shared discrete latent over 3D end-effector point-cloud transitions improves fine-grained motion probing and downstream manipulation success, but only when the geometric latent is forced to encode motion rather than morphology (shared bimanual code, pair-consistent augmentation, bidirectional transitions).Capability: Turning plans into physical action
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewFor VLA and world-action manipulation policies that score near-ceiling on standard LIBERO, recovery success after a real execution failure falls sharply as recovery requires deeper state reasoning \u2014 retrying or adapting the immediate action works far more often than restoring a displaced task object or clearing an environmental obstruction \u2014 and rankings on the standard benchmark do not predict recovery rankings.Capability: Turning plans into physical action
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor VLA manipulation policies doing single-step action prediction on LIBERO observations, the commanded action is fragile to camera rotation but tolerant of brightness: validated over entire continuous perturbation regions, 27% of instances admit a counterexample within ±1° of camera rotation and 51% within ±2°, while every instance is robust to a ±2% brightness shift and the median tolerates ±30%.Capability: Turning plans into physical action
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewFor VLM agents doing multi-phase navigation and pick-and-place in procedurally generated indoor scenes with moving human obstacles, subtask progress runs well ahead of full success — agents reach the first goal and then stall or loop after the phase transition — so success rate alone hides where the failure is.Capability: Turning plans into physical action
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewFor VLM-based judging of camera-conditioned world-model videos, splitting a long video pair into action-aligned chunks with structured visual evidence (source image, frame-grid overview, per-action first/last-frame panels) and voting chunk verdicts into video-level preferences agrees with human preferences better than judging the whole video pair at once, because short-lived action evidence is otherwise diluted by the full context.
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-15· pending-reviewFor VLM-based navigation agents, having the model output a 2D pixel on an egocentric image and projecting it to a 3D waypoint with depth plus a SLAM controller works better than having the same model regress metric 3D waypoints directly, because it keeps the action in the 2D image space the VLM was pretrained on instead of asking it to learn geometric transformations.Capability: Turning plans into physical action
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-18· pending-reviewFor VLM-driven embodied task planning with an egocentric camera, where hazards can fall out of view, the form of the memory handed to a risk predictor decides whether safety improves: an object-relation graph that retains out-of-view objects and updates edges after actions raises safe success rate well above no memory, while feeding the full image history or a flat list of seen objects scores below having no memory at all, because relationships and structure, not object recall, are what risk prediction needs.Capability: Turning plans into physical action
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewFor VLMs choosing an ordered route over numbered waypoints in a single egocentric image, per-waypoint feasibility and the first action are judged well while the rest of the route is not: pooled over nine models, roughly 88-96% of predictions take a legal first edge, but full-route legality falls to about 47% for a point agent and under 7% once the agent has a footprint, with most of the loss in intermediate edges rather than in output format.Capability: Turning plans into physical action
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewFor zero-shot risk forecasting from tabular weather features on a rare-event (~5% positive) binary severity task, GPT-4o/4.1 class models trail simple supervised classifiers (logistic regression, LightGBM) trained on the same data on macro-F1 and precision at every 3h/6h/12h horizon, and the weaker models close the gap only by flagging nearly every window as high risk.Capability: Predicting future events
- observationsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-04Frozen LLMs, using only retrieval over past communications and self-reflection rather than fine-tuning, can play the social deduction game Werewolf competently and show emergent strategic behavior, including deception, without being explicitly trained for it.Capability: Strategic deception and detecting it
- supportsExploring Large Language Models for Communication Games: An Empirical Study on WerewolfThis source: Exploring Large Language Models for Communication Games: An Empirical …How much the field cites it — heavily cited in the last 12 months59 in the last 12 months · 312 totalpublished 2023checked 2026-09-04click for Semantic Scholar
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-04Gating irreversible actions works only if the harness mediates every side effect, not only the branch awaiting approval. Where a gate suspends its own branch while a sibling's effect still executes, the control reads as a barrier and is not one — and the shipped implementations in widely used agent frameworks have this gap rather than it being a misconfiguration.Capability: Prioritizing safety under conflicting goals
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewGeneral-purpose factuality verifiers (entailment-, consistency-, and RAG-trained) and zero-shot frontier LLM judges perform near chance at spotting unsupported claims in scientific peer reviews that must be checked against the full submitted paper, because they cannot separate paper-unsupported assertions from legitimate evaluative critique; small models fine-tuned on in-domain examples beat them by a wide margin.Capability: Stating false facts confidently
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewGeneral-purpose factuality verifiers (entailment-, consistency-, and RAG-trained) and zero-shot frontier LLM judges perform near chance at spotting unsupported claims in scientific peer reviews that must be checked against the full submitted paper, because they cannot separate paper-unsupported assertions from legitimate evaluative critique; small models fine-tuned on in-domain examples beat them by a wide margin.Capability: Stating false facts confidently
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-20· pending-reviewGiven a published state-of-the-art method as the starting problem, a fully autonomous multi-agent loop (extract limitations, generate and novelty-filter ideas, run experiments, select the best hypothesis, plan and run ablations, draft the paper, pass a simulated peer-review rebuttal) improved on the human baseline for 86 of 107 top-venue papers (80%) by an average 25% relative, and its papers scored above the mean of accepted ICLR 2026 and NeurIPS 2025 papers under automated AI reviewers; every judgement in that result, from the improvement parsed out of result tables to the review scores, was made by a model, not a person.Capability: Improving its own scaffolding from failures
- mechanismsingle paper↑2↓0Reviewed by AI and Russ Millerchecked 2026-09-07Given several conflicting but equally legitimate passages, a model's answer is driven disproportionately by whichever passage comes first, and the skew sits in the representation of the combined prompt rather than in decoding — which is why reordering the passages does not remove it and why steering the activations does.Capability: Checking claims against evidence
- supportsLarge Language Models in Resolving Contextual Knowledge Conflicts
- supportsAdaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge ConflictsThis source: Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large …How much the field cites it — heavily cited in the last 12 months148 in the last 12 months · 382 totalpublished 2023checked 2026-09-04
- mechanismsingle paper↑2↓0Reviewed by AIchecked 2026-09-11· pending-reviewGiving a coding agent a raw shell is worse than giving it a designed interface: SWE-agent's constrained Agent-Computer Interface — a file viewer windowed to 100 lines, a search tool that truncates large output, and an editor that lints before a change persists — roughly doubled SWE-bench resolution over raw shell access with the same model, because verbose output and unmanaged state are where the model loses its place.Capability: Using the tools it is given
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewGiving an agent a parsed code graph (files, classes, functions with import, call and inheritance edges) and tools to search and traverse it lets a fine-tuned 32B open model localize issues about as well as proprietary agents at roughly an eighth of the cost, reaching about 93% file-level accuracy, and better localization carried through to about 12 points more resolved issues at Pass@10 when plugged into a fixed repair pipeline.Capability: Finding what matters in a large codebase
- observationsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03GPT-4 could reliably name a well-known celebrity's parent, but was far less reliable naming the celebrity when given the parent — the same directional-recall asymmetry demonstrated in smaller fine-tuned models, showing up in a deployed model that wasn't fine-tuned for the test.Capability: Not generalizing "A is B" to "B is A"
- supportsThe Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"This source: The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"How much the field cites it — heavily cited in the last 12 months174 in the last 12 months · 530 totalpublished 2023checked 2026-09-04
- observationsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03GPT-4 recommended non-existent Python and JavaScript packages in only a small percentage of generations — lower than the open models tested in the same study — but the same hallucinated names recurred often enough across runs to be practically exploitable by an attacker who registers them ahead of time.Capability: Writing secure code and dependencies
- supportsWe Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMsThis source: We Have a Package for You! A Comprehensive Analysis of Package Halluci…How much the field cites it — heavily cited in the last 12 months80 in the last 12 months · 106 totalpublished 2024checked 2026-09-04
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-04Grading generated code by whether it actually passes held-out functional tests, rather than by surface similarity to a reference solution, is what makes code generation gradeable at scale — and it's how the field has graded it since the earliest large benchmarks.Capability: Generating and editing working code
- supportsEvaluating Large Language Models Trained on CodeThis source: Evaluating Large Language Models Trained on CodeHow much the field cites it — very heavily cited in the last 12 months1000+ in the last 12 months · 11293 totalpublished 2021checked 2026-09-04count capped at 1000 by the fetchclick for Semantic Scholar
- mechanismreplicated↑2↓0Reviewed by AI and Russ Millerchecked 2026-09-04Grounding answers in retrieved documents reduces unsupported generation but transfers the trust problem to the retrieval channel: whatever lands in context is treated as reliable, which shows up both as following instructions planted in retrieved content and as deferring to whichever account is better represented when sources disagree.Capability: Stating false facts confidently
- supportsNot what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt InjectionThis source: Not what you've signed up for: Compromising Real-World LLM-Integrated …How much the field cites it — very heavily cited in the last 12 months1000+ in the last 12 months · 1870 totalpublished 2023checked 2026-09-04count capped at 1000 by the fetch
- supportsAdaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge ConflictsThis source: Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large …How much the field cites it — heavily cited in the last 12 months148 in the last 12 months · 382 totalpublished 2023checked 2026-09-04
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03Grounding generation in retrieved documents improves factual accuracy on knowledge-intensive tasks.Capability: Stating false facts confidently
- supportsRetrieval-Augmented Generation for Knowledge-Intensive NLP TasksThis source: Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksHow much the field cites it — very heavily cited in the last 12 months1000+ in the last 12 months · 17840 totalpublished 2020checked 2026-09-04count capped at 1000 by the fetch
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-08· pending-reviewGuardrail instructions written to stop older models doing the wrong thing — blanket prohibitions, repeated warnings, defensive defaults — became dead weight in the Claude 5 generation: Anthropic removed over 80% of Claude Code's system prompt for Opus 5 and Fable 5 with no measurable loss on its own coding evaluations, on the reading that the constraints now conflict with each other and with user intent more often than they prevent harm.Capability: Keeping its own context clean
- mechanismmechanism reasoning↑2↓0Reviewed by AIchecked 2026-09-18· pending-reviewGuides, sensors and checkpoints pay for themselves only when the same workflow runs repeatedly, mistakes have real consequences, and no person is watching each run; single-turn questions, brainstorming and one-off tasks gain nothing from them and pay their cost in tokens and setup — so the decision to build a harness is a decision about the shape of the work, not about the model.Capability: Improving its own scaffolding from failures
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewHanding an agent context-editing tools (delete, summarize, note, fold) by prompt alone makes mid-size models worse than leaving their context untouched — Qwen3-8B, Qwen3-14B and Gemma4-E4B all scored lower on long-context QA with the tools than without them; only after fine-tuning on trajectories that use the tools does the toolset become a gain.Capability: Keeping its own context clean
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewHaving a model score its confidence step by step detects failures in multi-step reasoning better than one holistic confidence score for the whole answer, with up to 15% higher AUROC on two multi-step benchmarks, so where a chain fails is more visible per step than in the final answer.
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03Holding the task fixed and only lengthening the input degrades reasoning long before the context limit is reached.Capability: Losing information in long inputs
- supportsSame Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language ModelsThis source: Same Task, More Tokens: the Impact of Input Length on the Reasoning Pe…How much the field cites it — heavily cited in the last 12 months104 in the last 12 months · 238 totalpublished 2024checked 2026-09-04click for Semantic Scholar
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03Identifies position, verbosity, and self-enhancement biases in strong judges, while also showing high agreement with humans once mitigated.Capability: Biased when judging other outputs
- supportsJudging LLM-as-a-Judge with MT-Bench and Chatbot ArenaThis source: Judging LLM-as-a-Judge with MT-Bench and Chatbot ArenaHow much the field cites it — very heavily cited in the last 12 months1000+ in the last 12 months · 10936 totalpublished 2023checked 2026-09-04count capped at 1000 by the fetchclick for Semantic Scholar
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewIn 4-7B open vision-language models (LLaVA-1.5-7B, Qwen2.5-VL-7B, InternVL-2.5-4B), how the cross-modal attention distribution over visual tokens shifts between adjacent decoder layers separates hallucinated from grounded object predictions better than the model's sensitivity to masking the attended region, and better than attention drift measured against a fixed first or last layer anchor.Capability: Grounding answers in the image
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewIn 7-13B open vision-language models, object hallucination is not explained by weak visual attention e hallucinated and real object tokens draw comparable attention to specific image regions in mid-to-late layers; the separating signal is whether a logit-lens decode of those attended visual tokens matches the generated object word.Capability: Grounding answers in the image
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewIn 7-8B multimodal models, object hallucination during visually grounded token generation tracks a shift of the visual-language information ratio toward language inside mixed-modality "synergy" attention heads rather than a shortage of visual-specific heads, and rescaling visual versus language value vectors in those heads causally moves hallucination in both directions.Capability: Grounding answers in the image
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-18· pending-reviewIn 7-8B open-weight instruct models (Llama-3.1, Mistral-v0.3, Qwen2.5), the drop in a token's probability when a contiguous mid-layer MLP span is zero-ablated is near zero for function words, elevated for knowledge-bearing tokens, and abnormally spiked for hallucinated tokens, so this internal attribution signal — approximable by a small probe from one forward pass — can be used in beam scoring to raise factuality on open-ended and long-form generation, while leaving strict short-answer EM/F1 unchanged or worse.Capability: Stating false facts confidently
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewIn 7-8B open-weight instruct models (Mistral-7B, Qwen2.5-7B, Llama-3-8B) answering closed-book factual questions, a small set of feed-forward neurons (top 1% or fewer) selected on one dataset by class-conditional separation of their residual-space contribution at the final prompt token transfers to other factual QA datasets, where probes on those neuron activations detect hallucination about as well as probes on full residual-stream or MLP-output states.Capability: Stating false facts confidently
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewIn 7-8B vision-language models, fine-tuning visual and text embeddings toward perturbation-averaged and ground-truth anchors before cross-modal contrastive alignment lowers object and attribute hallucination rates on captioning and existence benchmarks without reducing object coverage or general VQA accuracy.Capability: Grounding answers in the image
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewIn 7B open vision-language models (LLaVA-1.5, Shikra), the per-token log-probability gap between a normal forward pass and one with visual features zeroed shrinks as a caption gets longer, and while that gap ranks hallucinated tokens slightly better than entropy or max-probability, its absolute separation is weak (AUROC around 0.57-0.59) and it fails entirely when aggregated to sentence level.Capability: Grounding answers in the image
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewIn 7B vision-language models that prune visual tokens during decoding for efficiency, hallucination rises as the generated caption gets longer, because pruning is irreversible and discards fine-grained evidence needed at later decoding steps; keeping the pruned tokens as a retrievable bank and pulling them back when next-token entropy spikes recovers the grounding without a second forward pass.Capability: Grounding answers in the image
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewIn 7B-32B vision-language models, applying a bounded decoding-time logit penalty to candidate tokens whose rank-pooled visual-state evidence falls below a per-token reference estimated offline from unlabeled images lowers object-hallucination rates (CHAIR_I down up to 21.3% on four 7B backbones) while largely preserving descriptive and general multimodal scores, at about 1.33% added per-token latency once image evidence is cached at prefill.Capability: Grounding answers in the image
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewIn 7B-32B vision-language models, evidence for a candidate token read out of visual-token hidden states cannot be pooled as probabilities across visual positions, because each position's distribution sharpness differs (the rank-10 probability spans roughly 8-14x across positions and affine rescaling does not remove it); pooling vocabulary ranks instead is invariant to that scale, and because most of the remaining variability is between tokens rather than between images, the pooled rank must be compared against a per-token reference rather than one global threshold.Capability: Grounding answers in the image
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewIn 7B-class LVLMs (LLaVA-1.5, MiniGPT-4, mPLUG-Owl2) that rescore object tokens during decoding with a frozen CLIP image-text similarity, the CLIP signal must be weighted by the model's own entropy over the object candidates: a fixed mixing coefficient is sensitive to its value and gives smaller hallucination reductions, and letting CLIP fully override the LVLM logits degrades output.Capability: Grounding answers in the image
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewIn 7B-class LVLMs (LLaVA-1.5, MiniGPT-4, mPLUG-Owl2), hallucination directions in activation space are not a single mode: clustering hallucinated-minus-faithful hidden-state differences into several low-rank subspaces and weighting them per test image lowers object-hallucination rates slightly below a single global null-space projection, with the margin small (CHAIR_I 5.30 to 4.92 on LLaVA-1.5) and inconsistent on yes/no existence probing.Capability: Grounding answers in the image
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewIn 7B-class vision-language models (LLaVA-1.5, MiniGPT-4, mPLUG-Owl2), object hallucination is carried by an anisotropic set of directions in deep-layer hidden states, so attenuating each eigenmode in proportion to its hallucination-to-truthful variance ratio — estimated once offline from paired truthful/hallucinated captions and folded into the FFN output projection — lowers CHAIR rates, while a mean-offset subtraction or isotropic shrinkage of the same representations does not.Capability: Grounding answers in the image
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewIn a 20-year deterministic football-management simulation with hidden state and roughly 340-400 decision stops, what separates frontier models' final scores is managerial behavior rather than deliberation volume: cutting slow-payoff investment as the horizon ends (Spearman -0.58 with late-minus-mid investment shift), keeping cash deployed rather than idle (-0.50), and opening contract renewals early (+0.45) track the score on every seed, while token spend is uncorrelated under every accounting (r_s -0.19, all p > 0.38).
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewIn a 20-year deterministic football-management simulation with hidden state, adaptive rivals and 340-400 decision stops, final score across 15 frontier models is not ordered by model scale, price, vendor or token spend, and early standings barely predict the final order (rank correlation with the final order about 0.19 at year 5, rising to 0.78 by year 15), so short-horizon measurements would rank a different set of models.
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewIn a 241-day simulated crop-management task where decisions accumulate irreversibly, a zero-shot multi-agent LLM framework (perception, manager, decision, revision, reflection) scores below a PPO policy trained in that same environment but far above the same PPO policy transferred to a shifted weather/location/soil environment, while using under 28% of its fertilizer and irrigation.
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewIn a 5-agent LLM debate where agents iteratively revise a numeric preference vector, switching from all-to-all to ring communication once aggregate disagreement falls below a threshold reaches consensus using fewer tokens than all-to-all throughout and in fewer rounds than ring-only, but it does not dominate for every backbone: with Gemini 2.5-Flash all-to-all throughout converged in fewer rounds than the adaptive policy.Capability: Getting more from several models than from one
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewIn a deployed multilingual medical triage system, most of the accuracy gain comes from writing the clinicians' implicit decision tree into an explicit scoped rule set rather than from moving the final decision out of the LLM: replacing a flat danger-sign list with 231 structured rules in the same end-to-end prompt raised recall from 0.565 to 0.837 and F1 from 0.606 to 0.689, while further handing the decision to a deterministic rule engine over LLM-extracted symptoms left accuracy statistically unchanged (recall 0.810, F1 0.702).Capability: Following an unfamiliar procedure
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewIn a deterministic 365-day e-commerce business simulation of 1,000-4,000 agent steps, end-of-year assets do not predict the six underlying operating competences: across 18 frontier and open-weight models the top earner ranked 16th of 18 on fraud avoidance and below another model on profit per tool call, and the best negotiator finished mid-pack on assets.
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewIn a full-duplex speech-to-speech model that emits function calls on a dedicated parallel output stream, choosing the right tool from spoken, disfluent user requests is far easier than filling its arguments: tool-selection F1 reached 82.5% (above two closed realtime APIs) while argument accuracy was 42.2% and end-to-end Pass@1 33.0%, below both.Capability: Using the tools it is given
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewIn a generational genetic search over LLM-agent-generated scientific hypotheses (drug repurposing across 34 cancer types), replacing fitness-guided parent selection with uniform random selection while holding the scientific roles, variation operators and hypothesis count fixed lowers the population's mean and minimum external evidence scores but leaves the maximum unchanged, and removing crossover or mutation altogether produces mixed shifts that do not survive multiple-testing correction — so the selection rule, not the semantic variation operators, is where the measurable coordination gain sits.Capability: Getting more from several models than from one
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewIn a hierarchical multi-agent system where a controller gates acceptance of worker outputs on semantic entropy and semantic density computed over resampled completions, the intervention helps only where failures come from semantic instability (vague or contradictory outputs) and hurts where they come from evidence selection or reasoning-chain errors \u2014 gaining on ambiguity/hallucination-prone benchmarks while losing 10-14 points on multi-hop QA.Capability: Getting more from several models than from one
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewIn a preregistered experiment with 1,500 UK adults, a persuasive chatbot shifted attitudes on policy issues by about 12.6 points on a 100-point scale; telling people they were talking to an AI changed nothing (13.1 points), while additionally disclosing the chatbot's persuasive intent and instructions cut the effect roughly in half (6.3 points).Capability: Changing what a person believes or decides
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewIn a preregistered US experiment (N=2,144) where participants discussed an economic-policy misconception with ChatGPT over three rounds, a brief message that the model was biased against their party reduced belief correction by 28% (CI 16-40%) relative to a neutral control, with transcripts showing more pushback and less receptive engagement.Capability: Changing what a person believes or decides
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-22· pending-reviewIn a production ads-ranking stack, an agent that chains hypothesis generation, experiment execution, result analysis and a shared knowledge store, with human checkpoints at stage boundaries, delivered several times the completed ML iterations per engineer-week of the manual baseline and a 2.6% relative offline improvement on the headline model from multi-source exploration versus 0.4% from single-hypothesis runs; a semi-automated variant that automated phases without chaining them gained much less, so the leverage came from removing engineer-mediated handoffs, not from automating any one phase, and the authors report reliability was governed more by the orchestration harness (skill coverage, statistical rigor of evaluation, resilience to infrastructure noise) than by the base model's reasoning.Capability: Improving its own scaffolding from failures
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewIn a prompted text-to-SQL agent, a blanket output-shaping instruction in the system prompt (here "LIMIT 10") is applied by the model to every generated query regardless of the question's intent, truncating answers that legitimately need many rows, so removing it raises execution accuracy most on questions whose correct result set is large.Capability: Writing a reusable locator: regex, SQL, XPath
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewIn a training-free generate-critique-revise loop on a frozen backbone, a self-reported numeric confidence tag is useless as a stopping signal because the model declares high confidence on essentially every first generation (every threshold from 0.6 to 0.9 halted the loop at step 1 on 100 BBH questions), whereas a textual CONFIRMED sentinel from the critique stops 82-88% of items at unchanged accuracy.Capability: Saying it does not know
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewIn a two-agent extract-then-answer pipeline on multilingual extractive QA, routing the inter-agent handoff through English (forward-translate facts, answer in English, back-translate) lowers Exact Match relative to keeping handoffs in the user's language, and the loss grows with distance from English (13.0 points Spanish up to 30.6 points Hindi with Aya-23-8B).Capability: Getting more from several models than from one
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewIn a year-long multi-agent social simulation driven by a single 4-bit quantized 8B model on a 12 GB consumer GPU with five agents, the run can be carried to roughly 50-52 simulated weeks before the accumulated prompt hits the context limit, while about 10% of finalized activity records are left with at least one missing field.
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewIn agent pipelines that generate whole domain-specific ML codebases (medical imaging) from scratch, execution/assembly validation and domain knowledge bases fix different failures: removing runtime and assembly testing mainly costs autonomy (more human interventions to debug), while removing the curated domain knowledge base mainly costs output quality (syntactically fine but domain-inappropriate pipelines) with little change in intervention count.Capability: Generating and editing working code
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewIn agentic customer-service settings where a deployer incentive conflicts with a user's documented entitlement, a model's willingness to lie under incentive alone is not predicted by its willingness to lie when explicitly told to — some models comply with explicit deception instructions at high rates while almost never initiating deception under incentive, so instructed-deception evaluations measure capability rather than propensity.Capability: Strategic deception and detecting it
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewIn agentic customer-service settings where a deployer incentive conflicts with a user's documented entitlement, a model's willingness to lie under incentive alone is not predicted by its willingness to lie when explicitly told to — some models comply with explicit deception instructions at high rates while almost never initiating deception under incentive, so instructed-deception evaluations measure capability rather than propensity.Capability: Strategic deception and detecting it
- mechanismmechanism reasoning↑3↓0Reviewed by AIchecked 2026-09-11· pending-reviewIn an agent's verify-and-fix loop, deterministic checks — tests, linters, schema validators — should be exhausted before any model-based judge is added, because they are free, fast and give the same verdict every time, while an LLM-as-judge costs tokens per run, returns non-deterministic verdicts, and so belongs only on properties no rule can express, and even then as an advisory signal until measured against human review.Capability: Fixing its own mistakes
- supportsHarness Engineering — Agent = Model + Harness: The 6-Layer Production Playbook
- supportsLarge Language Models Cannot Self-Correct Reasoning YetThis source: Large Language Models Cannot Self-Correct Reasoning YetHow much the field cites it — very heavily cited in the last 12 months503 in the last 12 months · 1164 totalpublished 2023checked 2026-09-04
- supportsMaintainability sensors for coding agents
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewIn an LLM-assisted SOC pipeline where injections are planted in log fields, a deterministic SIEM pre-filter (regex decoders plus length truncation) catches only the structurally obvious payloads while an LLM self-check input/output rail catches most semantic ones, so the two layers block disjoint attack classes and neither alone is sufficient.Capability: Following instructions hidden in data
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewIn an open-ended LLM roleplay game where the character must withhold authored facts, gating generation through an external state tree — one retrieved permitted fact injected into the prompt, plus a separate verifier that decides whether the fact was actually conveyed before updating state — cuts the share of responses containing story-breaking unauthored content (17.8% to 6.3%) and stops premature reveal of the ending entirely, but makes some reveals appear unmotivated because the generator is forced to deliver the retrieved fact on that turn.Capability: Stating false facts confidently
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewIn audio in-context learning with per-episode resampled rules, current large audio language models bind arbitrary non-speech sounds to new labels reasonably well (best model 58.5% average over six such tasks at 5 shots) but stay near zero on tasks requiring temporal measurement — duration estimation, onset/offset segmentation, Morse dot-versus-dash — with no model averaging above 7.1%, and that temporal gap does not shrink across model generations while the non-temporal one improves sharply.Capability: Understanding speech and audio
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewIn audio language models that feed encoder outputs into a text LLM, downstream understanding accuracy across speech, sound and music is set by how semantically text-aligned the encoder is rather than by whether its representation is discrete or continuous — reconstruction-oriented codecs with high acoustic fidelity underperform, semantically distilled tokenizers can beat their own continuous teacher, and scaling the LLM backbone does not recover the lost information.Capability: Understanding speech and audio
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewIn claim-level hallucination detection pipelines over image, video and audio inputs (both comprehension and generation directions), decomposing an output into atomic claims contributes more to detection accuracy than ensembling several verifier models: removing decomposition costs about 6.6-7.9 macro-F1 across six cross-modal tasks while removing multi-expert majority voting costs about 4.5-5.3.Capability: Stating false facts confidently
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewIn closed-book multi-hop QA the accuracy gain from decomposed prompting shrinks to little in frontier models, but answers still change between task-equivalent prompt regimes (agreement only about 70%, and about 60% on MuSiQue), and because stable facts survive re-prompting while hallucinations do not, disagreement across regimes detects errors better than standard uncertainty baselines on F1 and AUROC without retrieval or training.Capability: Recalling facts it was trained on
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewIn confidence-gated retrieval agents that commit, retrieve or defer against a fixed numerical confidence threshold, applying a post-hoc calibration map to the same confidence signal raises accuracy among committed answers (15.8-41.0 points across three models on HotpotQA and MuSiQue) purely by moving the controller to a more selective operating point, so coverage falls and retrieval cost rises and overall accuracy can go either way (up to +15 points on HotpotQA, down to -17 on MuSiQue).Capability: Saying it does not know
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewIn diffusion vision-language models generating free-form captions, object tokens that are hallucinated commit at later denoising steps and with lower per-token confidence than grounded object tokens, so the commit-step trajectory is an available internal hallucination signal that has no counterpart in autoregressive decoding.Capability: Grounding answers in the image
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewIn DPO alignment of 7-8B instruct models, the alignment tax is predictable from properties of the preference pairs themselves: pairs whose chosen/rejected log-probability margin under the reference model is large in magnitude, or whose chosen/rejected token-length difference is large, cause the most loss of general capability, so dropping them keeps most of the alignment gain while cutting forgetting.Capability: Retaining skills and safety after fine-tuning
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewIn embodied environments with high-frequency action spaces (Atari, Minecraft, Crafter, DMLab, Overcooked, Pico Park, MindCraft), pairing a frozen instruction-tuned VLM that emits sparse text instructions with a small world-model controller trained to follow them scores higher than either the same VLM emitting low-level actions directly or the same controller running without instructions, and the gain depends on the instructions carrying semantic content — random-string instructions score no better than no instructions at all.Capability: Turning plans into physical action
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewIn embodied multi-agent household planning with a fixed high-level skill interface, a team's task success rate overstates coordination quality, because goal predicates can be reached through illegal shortcuts: several of 11 tested MLLMs succeed on more episodes than they produce legal plans on (Claude Haiku 4.5 63.1% success vs 47.2% legal; Qwen3.6-Plus 69.1% vs 56.7%), while others honour constraints yet miss the goal.Capability: Getting more from several models than from one
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewIn encoder-projector-decoder audio-language models, speaking style is linearly decodable from late audio-encoder layers at about 82-85% on a 7-way style task, yet open-ended tone prediction at the output reaches only about 20-54%, and which end of that range a model lands on tracks its training objective rather than its encoder: a model trained to generate speech keeps acoustic sensitivity through the decoder while text-only-trained siblings with a nearly identical encoder answer from the words instead.Capability: Understanding speech and audio
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewIn ESG-report RAG question answering with retrieval held fixed at a single top-1 chunk, open-weight 2B-30B generators vary widely in how well they stay anchored to the retrieved passage (faithfulness 0.607-0.822) yet all land in a narrow low band of factual correctness (0.387-0.449), and the most faithful model is the least factually correct, so groundedness to context and correctness of extracted ESG facts do not track each other.Capability: Competence in a specialist domain
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewIn event-conditioned time-series prediction (finance and traffic), models cite the factor that a controlled intervention actually manipulated in nearly all cases while their predictions change with that factor far less often — on traffic series-side pairs citation exceeds 90% for most of ten models but pair-level behavioral support stays under 22% — so citation of the right factor is not evidence the prediction depended on it.Capability: Saying what actually drove the answer
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewIn file-based sandbox agent tasks with distractor artefacts, scoring only whether the task goal is met overstates reliability, because agents frequently complete the target while modifying unrelated files: across nine LLM agents, pass rate exceeds a success-plus-preservation metric by roughly 10-17 points, with the gap widening as model capability falls.
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewIn formal persuasion dialogues on a UK political topic, LLM debaters are judged more persuasive than humans by crowd annotators (all nine LLM configurations occupied the top nine subjective ranks) while ranking far lower under argumentation-theoretic adjudication, where a human topped the logical ranking and humans took six of the top ten.Capability: Changing what a person believes or decides
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewIn four experiments with 3,996 US participants discussing a conspiracy they were unsure about, frontier models argued people into as well as out of belief with no consistent truth advantage: debunking cut belief by about 12 points on a 100-point scale and bunking raised it comparably, and participants rated the bunking model as more informative and trustworthy; the one asymmetry was that debunking produced more large (40+ point) shifts (16% vs 7%), and constraining the bunking model to true statements retained only 57-64% of its effect.Capability: Changing what a person believes or decides
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewIn frozen large vision-language models, token-level hallucination evidence is distributed across the relational, spatial and sequential structure of the model's internal trace rather than contained in any single token's hidden state, so a detector that reads the attention graph over image patches, query and response tokens with interleaved GNN/ViT/GRU modules localizes hallucinated spans better than flat probes on the same activations.Capability: Grounding answers in the image
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewIn fully connected networks of LLM agents (Llama 3.2-8B, 50 agents, 10 rounds) where a fixed minority holds an unchanging extreme stance, neutral agents' textual justifications converge on the biased agents' vocabulary faster and more completely than their numeric stances do, so monitoring only the text overstates how much the group actually agrees.Capability: Getting more from several models than from one
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewIn GRPO-style RL post-training of 7B multimodal reasoning models, composing each optimization group from rollouts stratified by length-normalized sequence surprisal and outcome (confident-correct anchors, confident-wrong negatives, low-confidence successes) within the same step keeps path entropy from collapsing and avoids the late-training performance decline GRPO and DAPO show after a few hundred steps.Capability: Retaining skills and safety after fine-tuning
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewIn GRPO-style RLVR on math for 7B models, injecting partial reasoning prefixes from a small out-of-family model (Gemma-2-2B, LLaMA-3.2-1B) into a fraction of rollouts keeps policy entropy higher early in training and raises pass@k, with gains growing as k rises; the effect comes from distributional unfamiliarity rather than prefix quality, since prefixes from same-family models of equal or higher quality give no gain over vanilla GRPO.Capability: Retaining skills and safety after fine-tuning
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-15· pending-reviewIn hierarchical LLM agents acting in gridworld cooperation tasks (Overcooked layouts) with unseen partners, supplying an accurate partner-skill estimate as planner prompt context does not by itself stop the agent from continuing a stale, non-complementary skill; the estimate only changes behaviour when it is wired into an explicit interrupt rule that aborts the in-progress skill when new partner actions are unlikely under the committed estimate.Capability: Turning plans into physical action
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewIn hours-long ML development runs on Kaggle-style competitions, agent scaffolds each lock into one strategy policy — the single-loop CLI agent pivots far less than humans (9% vs 25% of transitions) and the evolutionary search agent far more (58%) with pivot gains cancelling out — and neither reopens an approach it had abandoned, even though it recovers scores after setbacks as well as top humans.Capability: Planning a sequence of actions before acting
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewIn interactive video world models, world state communicated only through prompt switching is not reliably preserved across a rollout — characters that should stay alive vanish and killed ones keep moving — whereas keeping the canonical state in an external deterministic engine and compiling it into pixel-aligned spatial controls for the video model keeps the rendered frames consistent with that state.Capability: Tracking state through a long task
- supportsProgrammable World Model
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewIn label-free RL post-training of reasoning models, deriving each model's reward from a separately-trained peer model's majority vote — rather than from its own majority vote — avoids the reward collapse and length degeneration that self-rewarding runs show, and the benefit grows as the paired models differ in family or size because their errors overlap less.Capability: Getting more from several models than from one
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-18· pending-reviewIn large audio-language models answering binary "is sound X present" questions, the yes/no outcome is fixed at the first decoding step and the model's yes-minus-no margin is positive for most clips even with a silent input, so penalizing affirmative tokens at that single step whenever real audio fails to raise the margin above a small threshold increases correct rejection of absent events beyond fixed-strength contrastive decoding, at the cost of lower recall on genuinely present events and lower precision.Capability: Stating false facts confidently
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewIn LLaVA-1.5-7B captioning, restricting a LoRA adapter to attention heads picked by a hallucination diagnosis (attention-to-image drop around hallucinated object words, then ablation screening) reduces object hallucination, while an identically trained layer-matched random-head adapter does not — so the selection, not the added adapter capacity, carries the effect.Capability: Grounding answers in the image
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewIn LLaVA-1.5-7B captioning, restricting a LoRA adapter to attention heads picked by a hallucination diagnosis (attention-to-image drop around hallucinated object words, then ablation screening) reduces object hallucination, while an identically trained layer-matched random-head adapter does not — so the selection, not the added adapter capacity, carries the effect.Capability: Grounding answers in the image
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewIn LLM fact-checking over supplied evidence, high veracity accuracy does not imply the verdict depends on that evidence: when the model's own cited sentences are iteratively removed, off-the-shelf models (GPT-4o-mini, Gemini-2.5-Flash-Lite, Qwen-2.5-32B, Llama-3.1-8B) keep their SUPPORT/REFUTE labels at a non-zero plateau instead of abstaining, and training with paired supervision that requires NOT_ENOUGH_INFO on the evidence-ablated input removes most of that persistence without losing accuracy.Capability: Checking claims against evidence
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewIn LLM multi-agent systems that reflect after failure, restricting reflection to the single agent responsible for the first incorrect step — located by a trained step-level correctness scorer and paired with a counterfactually generated corrected step — raises task success more than having every agent reflect on the trajectory, because reflections written by agents that behaved correctly enter their memory as misleading guidance for later tasks.Capability: Getting more from several models than from one
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewIn LLM multi-agent systems whose communication graph is produced by a learned topology generator, the generated graph carries edges that are not needed: scoring each edge by how much masking it changes the task outcome and the semantic entropy of the final answer, then keeping a top-ranked budgeted subgraph, cuts online tokens by about 20-25% while leaving average accuracy equal or slightly higher, and a small distilled predictor of those subgraphs transfers to generators and agent counts it was not calibrated on.Capability: Getting more from several models than from one
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewIn LLM-based ASR, steering decoding toward acoustic evidence only helps if candidates are first restricted to tokens within a small likelihood margin of the greedy token: unconstrained selection by an acoustic compatibility score removes hallucinated outputs but destroys transcription accuracy through unbounded insertions, while the constrained version removes a large share of hallucinations at near-zero WER/CER cost.Capability: Stating false facts confidently
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-18· pending-reviewIn LLM-based program repair on Defects4J bugs with frontier models (GPT-5, DeepSeek-R1, Claude Sonnet 4.5) prompted zero-shot, passing the developer-written test suite does not mean the patch is grounded in the bug: manual inspection of 812 sampled repairs found ungrounded repairs in 72.7% of cases including test-passing patches, most often from incorrect causal localization (45.9%) rather than a wrong edit style, and the same models frequently misidentify the bug-triggering tests and generate additional tests that miss the triggering condition.Capability: Stating false facts confidently
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewIn LLM-based repository-level program repair, telling the generator to be minimal or constraining its output format does not shrink patches, because verbosity comes from the capability-raising machinery itself (iterative refinement, broad context); a separate post-generation refiner that rewrites an already-plausible patch is what reduces size without losing fixes.Capability: Cutting to the chase
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewIn LLM-driven iterative prompt optimization, restricting gradient generation to examples the current prompt gets wrong and accepting a candidate edit only when its gain on hard cases exceeds a penalized regression on a preservation set of initially-correct examples yields higher final accuracy than unfiltered textual-gradient optimization, which degrades on some reasoning tasks below the zero-shot prompt.Capability: Improving its own scaffolding from failures
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewIn long-context tasks over 8K-128K-token documents, the middle-of-context penalty is granularity-dependent: word-level QA and paragraph-filling accuracy dip when the evidence sits mid-document, but sentence-cloze accuracy peaks in the middle, because mid-paragraph sentences carry concrete entity and date overlap that models match on instead of reasoning about discourse role.Capability: Losing information in long inputs
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewIn long-horizon tool-mediated environments where strategically relevant state is available only on explicit query (300+ turn Civilization VI via MCP, 23 runs, four model families under a shared playbook), agents allocate only about 1-2% of non-infrastructure tool calls to strategic monitoring and execute only about 48-66% of the near-term commitments they state in their own planning diaries, even though the playbook told them to query victory progress every 20 turns.
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewIn LVLMs asked to describe people (LLaVA, InstructBLIP, Shikra), masking attention so that text tokens read visual tokens but language-only self-attention is suppressed in late decoder layers lowers gender misattribution while keeping answer coverage high, whereas visual-only or language-only masking scores well on the bias metric only because the model names a gender in under 10% of cases.Capability: Grounding answers in the image
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-18· pending-reviewIn masked diffusion language models (LLaDA, Dream) asked to generate around a frozen constraint span placed mid-output, failures to honor the constraint come partly from too little attention being allocated to that span during denoising: adding a fixed additive bias to attention logits on the span raises constraint adherence and overall quality up to a moderate bias, then degrades fluency and contextual fit as the bias grows.Capability: Following an unfamiliar procedure
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewIn multi-agent debate with 3 open-weight agents over 3 rounds, giving each agent a memory of past debates helps only if retrieval is conditioned on the current consensus ratio and peer responses are annotated with memory-derived reliability labels; randomly inserting past cases is the weakest retrieval policy and plain task-similarity retrieval trails every consensus-conditioned variant.Capability: Getting more from several models than from one
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewIn multi-agent literature-synthesis pipelines, most of the gain over a fixed role-based pipeline comes from the shared-workspace architecture and the added critique/dissemination actions rather than from learning the coordination policy: on OARelatedWork, a learned decentralized RL controller scored 3.82 overall versus 3.80 for a centralized LLM coordinator given the same workspace and action space, while a naive orchestrator without that architecture scored 3.40.Capability: Getting more from several models than from one
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewIn multi-agent LLM networks where a correct minority peer disagrees with an agreeing majority, ranking peers by raw divergence from the receiver's own belief systematically prefers the wrong peers, while scoring each peer by asserted precision divided by its divergence from a second-order model of what that peer should have concluded given its data subscription isolates the correct one.Capability: Getting more from several models than from one
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewIn multi-agent LLM pipelines over reasoning and code tasks with a pool of 3B-32B open-weight backbones, conditioning each per-step routing decision on a learned filtered memory rather than the full accumulated trajectory leaves accuracy about the same while cutting inference cost by roughly 40-50%, so most of the accumulated history is cost without accuracy value.Capability: Getting more from several models than from one
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewIn multi-agent LLM systems where one agent starts with an erroneous shared belief, broadcasting every agent's full context at every step raises the rate of false assertions above doing no synchronization at all, because the error propagates to agents that were previously correct — and the harm appears only in tasks where one wrong fact cascades across semantically linked dimensions (destination to airport to weather), not where agent contexts are largely orthogonal.Capability: Stating false facts confidently
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewIn multi-agent LLM workflows with a mixed-size model zoo, choosing which model instantiates each agent step-by-step from an online estimate of task-progress gain plus remaining time and cost budget keeps task quality near the best single-model policy at lower energy than one-shot query-level routing or stall-triggered escalation.Capability: Getting more from several models than from one
- mechanismmechanism reasoning↑1↓0Reviewed by AIchecked 2026-09-18· pending-reviewIn multi-hop KGQA evaluated as graph navigation with annotated gold evidence paths, reported agreement between an agent's traversed path and the reference path cannot be credited to question conditioning unless it is compared against question-agnostic references — unbiased random walks and answer-oracle shortest paths — because dense or small graphs let structure alone deliver both answer reachability and partial path overlap.Capability: Whether the measurement made the finding
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewIn multi-reward RL fine-tuning where several valid reward dimensions are scalarized with fixed weights, the policy can raise total reward by saturating the easiest and densest dimensions while the hardest one degrades, because the scalar projection makes different reward profiles indistinguishable to the advantage estimator and shortcut profiles then crowd informative samples for the neglected dimension out of rollout groups.Capability: Gaming the measure instead of doing the task
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-07· pending-reviewIn multi-step visual QA, a critic step run by the same model improves final accuracy only when it is shown the candidate evidence alone and denied the generator's textual rationale; conditioned on that rationale, the critic ratifies the generator's initial hypothesis and the loop performs worse than no critic at all.Capability: Fixing its own mistakes
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewIn multi-turn LLM buyer-seller market simulations, a measured welfare gain attributed to a platform guardrail can be mostly an artifact of the scaffold the guardrail was bundled with: holding the offer schema and buyer choice rule identical across guarded and unguarded cells shrank or reversed the effect (+87.4, +35.0, +28.8 became +7.2, -13.9, +23.8 across a Qwen2.5 1.5B-14B ladder), and single-generation effects on the most favorable profiles shrank from a mean +229 to +37.6 with three generations per cell, with generation residuals accounting for 49.9% of variation.Capability: Whether the measurement made the finding
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewIn multi-turn text-to-SQL where an underspecified question hides two dependent clarification layers, current LLM agents almost never surface both layers (joint identification under 5% for all 12 models tested), and this is not explained by how often they ask: ask rates spanning under 10% to over 60% all produce joint identification below 4%, and second-layer identification stays low (1.2-13.8%) even when the first layer is pre-resolved for them.Capability: Writing a reusable locator: regex, SQL, XPath
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewIn one-shot, no-communication self-play where independent copies of the same LLM are told their counterparts run the same model, models exceed the Nash baseline mainly in games whose optimal joint play requires matching actions (coordination, stag hunt, public goods) and fall below it in games requiring symmetry-breaking (anti-coordination, battle of the sexes), with several open-weight models scoring near the worst joint cell.Capability: Getting more from several models than from one
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewIn open-ended editing tasks with no unique correct output (multi-slide design editing), a separate judge model that scores the structural diff between the original and edited artifact against the instruction, and returns its critique verbatim as the next-turn instruction, raises instruction-following over a single pass, but the gain is selective: most tasks pass on the first attempt, and a minority of the ones that enter the loop get worse unless an earlier iteration is restored when the judge's own score declines.Capability: Fixing its own mistakes
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewIn open-ended romantic-advice prompts to consumer chat models, varying grammatical mood (declarative, conditional, interrogative, imperative) produces no systematic change in social sycophancy, while the perspective the prompt presupposes does, and acceptance of the user's premises and moral stance rises from the first response to a follow-up turn even when the follow-up question is neutral.Capability: Telling the user what they want to hear
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewIn open-source vision-language models (LLaVA-1.5, Qwen-VL-Chat, Qwen2.5-VL, LLaVA-NeXT), a training-free decoding intervention that detects large layer-to-layer and step-to-step shifts in hidden states and nudges the diverging states back toward their prior value lowers object-hallucination rates, but the lower rate comes with fewer objects mentioned and shorter outputs.Capability: Grounding answers in the image
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewIn open-source vision-language models (LLaVA-1.5, Qwen-VL-Chat, Qwen2.5-VL, LLaVA-NeXT), a training-free decoding intervention that detects large layer-to-layer and step-to-step shifts in hidden states and nudges the diverging states back toward their prior value lowers object-hallucination rates, but the lower rate comes with fewer objects mentioned and shorter outputs.Capability: Grounding answers in the image
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewIn open-weight 7-8B instruct models, a supervised detector built on how hidden states change between adjacent layers separates factually wrong from correct answers better than a probe on any single layer, and shuffling the layer order destroys most of the signal — so the ordering of the depth trajectory, not just the content of one layer, carries the information.Capability: Stating false facts confidently
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewIn open-weight instruction-tuned models, interventions that suppress caving to user pushback (DPO, SFT on chosen responses, or activation steering) also tend to reduce the model's rate of correcting a wrong answer when genuine supporting evidence is supplied, because the two answer-flip behaviors run on overlapping MLP neurons and attention heads with positively aligned steering directions.Capability: Telling the user what they want to hear
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewIn partially observable text environments (ALFWorld, ScienceWorld), letting an LLM agent query an externally maintained state estimate that keeps an explicit distribution over unobserved object locations raises task success more than querying a deterministic memory of what has already been observed, and the gain shrinks to near zero on a frontier model that already nearly saturates the benchmark.Capability: Tracking state through a long task
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewIn prompt-pool LLM time series forecasting where top-K semantic anchors are retrieved by cosine similarity, the harmful redundancy is concentration of anchor usage accumulated across training batches, not redundancy among the anchors picked for a single forecast: diversifying within a retrieval (MMR) never beat relevance-only selection on four dataset-horizon cells, while an EMA usage-tracking coverage penalty in the training loss lowered MSE on most of six long-term benchmarks.Capability: Predicting future events
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewIn region-guided contrastive decoding for medical VLMs, the effectiveness of the guidance depends on how much of the image the guiding mask covers: coarse ground-truth boxes (mean ~39% coverage on SLAKE) suppress so much of the unconditional branch that the contrast weakens, while compact automatically discovered masks (~7% coverage) selected by the drop in answer likelihood under occlusion give a sharper signal and higher closed-ended accuracy.Capability: Grounding answers in the image
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewIn retrieval-augmented answers, splitting an answer into claims, checking each against the retrieved source, and then acting on the flagged claims reduces the share of answers judged to contain unsupported content — but the strategies trade grounding against preservation: deleting unsupported claims reduces unsupported content most while retaining the least original text, and rewriting retains the most while reducing least.Capability: Stating false facts confidently
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-18· pending-reviewIn retrieval-augmented personalization with a frozen 8-9B generator, prepending more retrieved user-history records does not monotonically improve quality — on the six LaMP tasks a fixed top-k profile peaked at k=5 and lost 0.8 quality points at k=10 while the share of cases made worse than an empty profile rose from 23.2% to 31.7% — so a learned per-request stopping decision beats any single fixed k.Capability: Remembering across sessions
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewIn RL-trained multimodal image-quality models, applying the reasoning reward only to reasoning tokens and the human-rating reward only to rating tokens keeps rating alignment at the level of rating-only training while turning reasoning-guided edit quality gain positive, and it prevents the collapse to a single universal edit that an unmasked shared reward produces despite that variant's larger raw gain.Capability: Saying what actually drove the answer
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewIn RL-trained multimodal image-quality models, supervising reasoning with the rating reward alone raises rating alignment while making the stated reasoning less faithful — rating-only training reached an average PLCC/SRCC of 0.836/0.815 yet its reasoning-guided edits lowered judged image quality (mean gain -0.260, worse than the untrained baseline's -0.136).Capability: Saying what actually drove the answer
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewIn role-play with an assigned character, models usually recognize that a query lies outside the character's knowledge boundary yet fail to abstain when asked a concrete factual question about it, so the dominant failure is compliance rather than recognition.Capability: Saying it does not know
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-15· pending-reviewIn search-based automated program repair of Simulink/Stateflow models, replacing hand-crafted domain-specific mutation operators with an LLM that proposes label rewrites from static component context lowers repair coverage under a fixed wall-clock budget, because unconstrained generative edits widen the candidate space while each candidate costs an expensive simulation and the generator gets no behavioural feedback about why a patch failed.Capability: Generating and editing working code
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewIn semi-extractive RAG QA where the model is prompted to put quoted evidence in brackets, the bracketed spans frequently do not appear verbatim in the retrieved context (as low as 0.37 verbatim rate on a medical dataset with Mistral-Nemo), and masking logits during bracketed spans to only tokens continuing a verbatim context span raises verbatim fidelity to near 1.0 across Llama-3.1-8B, Mistral-Nemo and Qwen3-4B without lowering ROUGE-L or BERTScore much.Capability: Stating false facts confidently
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewIn simulated networks of LLM agents where one or two agents are instructed to advocate a position, the messages other agents write understate how far their elicited stance has moved: they mirror the persuader's rhetorical framing in most messages while explicit stance-change language is rare, so monitoring only generated text misses belief movement that a repeated token-probability probe detects.Capability: Getting more from several models than from one
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03In simulated retail and airline domains with tools and policy documents, frontier models succeeded on well under two thirds of tasks and success dropped sharply when the same task had to be passed several times in a row.Capability: Using the tools it is given
- supportsτ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World DomainsThis source: τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Dom…How much the field cites it — very heavily cited in the last 12 months889 in the last 12 months · 1075 totalpublished 2024checked 2026-09-04
- observationsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03In simulated retail customer-support tasks with real tools and policy documents, GPT-4o — the best performer in the study — still completed well under two-thirds of tasks correctly, and its rate of passing the identical task eight times in a row (a proxy for production reliability) was far lower than its single-attempt success rate.Capability: Using the tools it is given
- supportsτ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World DomainsThis source: τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Dom…How much the field cites it — very heavily cited in the last 12 months889 in the last 12 months · 1075 totalpublished 2024checked 2026-09-04
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-18· pending-reviewIn simulated urban air-ground object search where a VLM drives both a UAV and a UGV, wrapping the model in a fixed search-handoff-verify protocol that leaves geometric projection and road-graph path planning to deterministic tools raises task success for backbones with adequate perception while cutting decision calls for all, but does not lift weak small open models, whose success can fall on harder splits.Capability: Turning plans into physical action
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewIn synthetic hallucination benchmarks where faithful items are human-written and hallucinated items are LLM-rewrites, a detector's binary score partly measures human-vs-machine style rather than factuality: feeding faithful LLM rewrites of the same truthful articles makes some fine-tuned encoders flag nearly all of them as hallucinated.Capability: Whether the measurement made the finding
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewIn training-free contrastive decoding for 7B-class medical VLMs, the spatial and temporal contrast terms act on different answer types: anatomy-conditioned classifier-free guidance is what raises closed-ended accuracy (68.5 to 81.5 on SLAKE with Phi-3.5V-Med) while step-wise contrast against the previous step's logits is what preserves open-ended recall (37.8 to 44.9), so dropping either term costs about five overall points.Capability: Grounding answers in the image
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-18· pending-reviewIn training-free video temporal grounding that localizes segments by cosine similarity between a text query embedding and video features, clip-level features alone localize worse than frame-level features alone, yet projecting clip similarities onto the frame timeline and mixing the two streams beats either alone, because frames supply boundary resolution and clips supply temporal context.Capability: Reasoning about time in video
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewIn training-free video temporal grounding with frozen vision-language embedding backbones, matching a query only against frames sampled at a fixed rate leaves temporal context unused: adding clip-level (sliding-window) similarities projected onto the frame timeline raises localization accuracy even though clip-only similarity is worse than frame-only on its own.Capability: Reasoning about time in video
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewIn two-agent embodied social tasks requiring coordinated gaze, movement and expression, fine-tuning open VLMs (8B-14B InternVL) by timestep-level action imitation on demonstrations from an architecture with an explicit belief module shifts trial outcome scores up or down depending on condition but does not make the student's nonverbal signals more directionally coherent, because cross-channel communicative coordination is a trial-level property that per-timestep supervision does not encode.Capability: Turning plans into physical action
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewIn two-agent embodied social tasks where one agent must guide or mislead another through movement, gaze and facial expression, open-source VLMs up to 15B show no reliable behavioral change when an explicit theory-of-mind order constraint is imposed, failing at two separate points: most do not produce directionally coherent nonverbal signals, and those that do (e.g. a 14B model with near-ceiling trajectory and gaze alignment) are not decoded by the partner agent, whose choices track a fixed spatial preference instead.Capability: Turning plans into physical action
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewIn UI-to-code generation, where a rendered screenshot gives genuine external feedback, iterative self-critique still drifts below the initial draft over ten rounds, because a code edit changes the rendering non-locally and a fix for one region breaks regions that were already faithful; restricting each round to one typed, scoped repair target and carrying past targets forward as an avoid-list reduces that drift.Capability: Generating and editing working code
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewIn VLM-driven human-aware robot task planning over scene graphs, the bottleneck is inferring and grounding the symbolic goal, not the planning: once both human and robot goals correctly cover the ground truth, a formal multi-agent planner produces a valid, conflict-free joint plan in 99.6% of trials, while end-to-end success with VLM-inferred human goals is 69.7%.Capability: Planning a sequence of actions before acting
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewIn whole-repository cross-language code migration (Java to Python), supplying an LLM with AST-derived dependency-graph context and translating parent classes before children lowers the rate of calls to nonexistent APIs relative to vector-similarity retrieval of text chunks, while lexical similarity metrics such as CodeBLEU show no difference between the two.Capability: Stating false facts confidently
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewIn zero-shot LLM program repair on Defects4J bugs, passing the developer-written test suite does not indicate that the patch is grounded in a correct understanding of the bug: manual inspection of sampled repairs found ungrounded patches in about three quarters of cases including ones that passed all available tests, with incorrect causal localization the largest category.Capability: Stating false facts confidently
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewInstruction tuning that forces an answer to every question teaches models to fabricate when the question lies outside their parametric knowledge; splitting the tuning data by whether the base model already answers correctly and appending "I am unsure" to the rest (refusal-aware tuning) raises average precision on known-versus-unknown separation over vanilla tuning, transfers to out-of-domain datasets, and yields better calibrated confidence than post-hoc uncertainty scoring.Capability: Saying it does not know
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03Introduced grade-school math word problems on which 2021 models made frequent arithmetic slips, motivating verifier training.Capability: Digit-level arithmetic
- supportsTraining Verifiers to Solve Math Word ProblemsThis source: Training Verifiers to Solve Math Word ProblemsHow much the field cites it — very heavily cited in the last 12 months1000+ in the last 12 months · 10385 totalpublished 2021checked 2026-09-04count capped at 1000 by the fetchclick for Semantic Scholar
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewIssue localization can be done as retrieval-and-rerank with no agent at all: an embedding model and reranker trained on issue-to-patched-function pairs mined from public GitHub (SweLoc) localize functions on SWE-bench Lite and LocBench at least as accurately at Acc@10 as agents built on Claude 3.5 Sonnet, at a small fraction of the per-issue cost and with one pass instead of seven to ten model rounds.Capability: Finding what matters in a large codebase
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewKnowledge-editing methods such as ROME and fine-tuning insert a stated fact reliably but do not carry its logical consequences: on multi-hop questions generated from knowledge-graph rules that the edit entails, accuracy is up to 24 points lower than on the directly edited fact, so an edited model can assert a new fact and still answer as if its implications never changed.Capability: Absorbing and flexibly using new knowledge
- mechanismmechanism reasoning↑1↓0Reviewed by AIchecked 2026-10-19· pending-reviewLanguage models trained on the published literature inherit its positive bias, because failure data is systematically missing from what they read, and that degrades them in three roles at once — as research tools that suggest what to try, as consumers of scarce high-quality training data, and as peer reviewers — so the absence of negative results now costs more than it did when only people read the papers.Capability: Whether the measurement made the finding
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewLarge pretrained models are well calibrated on multiple-choice and true/false questions when answer options are shown as lettered choices, and can predict the probability that their own sampled answer is correct (P(True)) or that they know the answer at all (P(IK)); calibration improves with model size and few-shot prompting, self-evaluation improves when the model sees several of its own samples first, and P(IK) generalizes only partly to new tasks, where its calibration degrades.Capability: Saying it does not know
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewLLM-written advertisements matched human expert ads when personalized to personality traits (51% vs 49% preference, n=400) and beat them when built on general persuasion principles such as authority and consensus (59% preference, n=800); participants who recognized an ad as AI-made penalized it only slightly, and 29% chose the AI ad knowing its origin.Capability: Changing what a person believes or decides
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03Long-form biographies from strong models contain a substantial share of unsupported atomic facts, rising for less well-known subjects.Capability: Stating false facts confidently
- supportsFActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text GenerationThis source: FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long…How much the field cites it — very heavily cited in the last 12 months697 in the last 12 months · 1643 totalpublished 2023checked 2026-09-04click for Semantic Scholar
- mechanismmechanism reasoning↑1↓0Reviewed by AIchecked 2026-10-19· pending-reviewMachine learning research treats beating the state of the art on a benchmark as the main kind of publishable result, so negative results — methods that did not work, conditions under which a known method fails — go largely unpublished, and compute is spent repeating attempts that others have already found not to work; registering what does not work is no less important than registering what works.Capability: Whether the measurement made the finding
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03Managing context explicitly, paging information in and out of a bounded window, sustains performance on tasks that exceed the window.Capability: Keeping its own context clean
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03Many failures come from not following domain policy documents that were provided in the prompt.Capability: Following an unfamiliar procedure
- supportsτ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World DomainsThis source: τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Dom…How much the field cites it — very heavily cited in the last 12 months889 in the last 12 months · 1075 totalpublished 2024checked 2026-09-04
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03Models complied with insecure completions a large fraction of the time, and more capable models were more likely to suggest insecure code.Capability: Writing secure code and dependencies
- supportsPurple Llama CyberSecEval: A Secure Coding Benchmark for Language ModelsThis source: Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Mode…How much the field cites it — heavily cited in the last 12 months73 in the last 12 months · 180 totalpublished 2023checked 2026-09-04click for Semantic Scholar
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03Models endorse widely held falsehoods, showing weak verification against what they know.Capability: Checking claims against evidence
- supportsTruthfulQA: Measuring How Models Mimic Human FalsehoodsThis source: TruthfulQA: Measuring How Models Mimic Human FalsehoodsHow much the field cites it — very heavily cited in the last 12 months1000+ in the last 12 months · 3851 totalpublished 2021checked 2026-09-04count capped at 1000 by the fetch
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03Models fine-tuned on "A is B" fail to answer "B is A", and GPT-4 recalls celebrity parents far more often than the reverse.Capability: Not generalizing "A is B" to "B is A"
- supportsThe Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"This source: The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"How much the field cites it — heavily cited in the last 12 months174 in the last 12 months · 530 totalpublished 2023checked 2026-09-04
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewModels judge a syllogism's validity partly by how believable its conclusion is because validity and plausibility are encoded as linearly separable but strongly aligned directions in the residual stream: steering along the plausibility direction causally shifts validity judgements and vice versa, the degree of alignment predicts the size of the content effect across models, and a debiasing vector that separates the two reduces the effect and raises accuracy.Capability: Deducing validly from logical structure
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03Models readily adopt a single counter-memory passage when it is coherent, and when sources conflict they follow the majority and show confirmation bias toward their own beliefs.Capability: Checking claims against evidence
- supportsAdaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge ConflictsThis source: Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large …How much the field cites it — heavily cited in the last 12 months148 in the last 12 months · 382 totalpublished 2023checked 2026-09-04
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03Models reproduce common human misconceptions, and larger models were not more truthful on this benchmark.Capability: Stating false facts confidently
- supportsTruthfulQA: Measuring How Models Mimic Human FalsehoodsThis source: TruthfulQA: Measuring How Models Mimic Human FalsehoodsHow much the field cites it — very heavily cited in the last 12 months1000+ in the last 12 months · 3851 totalpublished 2021checked 2026-09-04count capped at 1000 by the fetch
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03Models that predict next steps well can still hold an incoherent implicit world model, which fails when the task deviates from familiar traces.Capability: Tracking state through a long task
- supportsEvaluating the World Model Implicit in a Generative ModelThis source: Evaluating the World Model Implicit in a Generative ModelHow much the field cites it — heavily cited in the last 12 months75 in the last 12 months · 144 totalpublished 2024checked 2026-09-04
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03Most models claiming long contexts fail well before their advertised length on synthetic retrieval, tracing and aggregation tasks.Capability: Losing information in long inputs
- supportsRULER: What's the Real Context Size of Your Long-Context Language Models?This source: RULER: What's the Real Context Size of Your Long-Context Language Mode…How much the field cites it — very heavily cited in the last 12 months759 in the last 12 months · 1243 totalpublished 2024checked 2026-09-04
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03Multi-digit multiplication accuracy falls to near zero as problem size grows, and models rely on pattern matching rather than a learned algorithm.Capability: Digit-level arithmetic
- supportsFaith and Fate: Limits of Transformers on CompositionalityThis source: Faith and Fate: Limits of Transformers on CompositionalityHow much the field cites it — very heavily cited in the last 12 months229 in the last 12 months · 690 totalpublished 2023checked 2026-09-04
- mechanismmechanism reasoning↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-04Offloading the calculation to generated code removes the arithmetic as a source of error but does not remove error — what remains is the translation from problem to program, which is now the only thing the model is doing. Verification has to move there with it.Capability: Digit-level arithmetic
- supportsPAL: Program-aided Language ModelsThis source: PAL: Program-aided Language ModelsHow much the field cites it — very heavily cited in the last 12 months268 in the last 12 months · 824 totalpublished 2022checked 2026-09-04
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03Offloading the computation to generated code and an interpreter removes most arithmetic errors on math word problems.Capability: Digit-level arithmetic
- supportsPAL: Program-aided Language ModelsThis source: PAL: Program-aided Language ModelsHow much the field cites it — very heavily cited in the last 12 months268 in the last 12 months · 824 totalpublished 2022checked 2026-09-04
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewOn 1,000 Lean 4 targets from computational and applied mathematics textbooks (optimization, numerical linear algebra, numerical analysis), whose statements depend on locally introduced definitions, current provers solve about 62-72% of a 200-problem sample, so applied-math formalization is tractable but not saturated and requires recovering textbook context to state problems faithfully.Capability: Formalizing and machine-checking mathematics
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewOn 1,010 counterfactual edits rooted in Chinese cultural entities and aligned with English counterparts, an edit made in one language largely does not propagate to the other: edits succeed in the edited language (about 80% for Chinese at batch size 10, falling with batch size) while the other language keeps answering the old fact, so multilingual models store the same fact in language-specific ways that editing does not bridge.Capability: Absorbing and flexibly using new knowledge
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewOn 102 IPC Blocksworld instances with a 180-second budget, a classical planner (Fast Downward) solved 85%, a model planning directly solved 64%, and the same model choosing one action at a time against a PDDL simulator with observe, reset and retry solved 67%, a consistent but small three-point gain at about 5.7 times the token cost, so interactive simulation helps a little and classical search still wins where a formal model exists.Capability: Planning a sequence of actions before acting
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-12· pending-reviewOn 170 software and research tasks timed with human domain experts, the length of task that frontier agents complete at 50% reliability, measured in human time, has doubled roughly every seven months since 2019 and was around 110 minutes for o3 in mid-2025 — but the horizon at 80% reliability follows the same trend at roughly five times shorter, and success drops as tasks carry more of the 16 "messiness" factors that separate benchmark tasks from real work, so the headline horizon overstates what an agent can be relied on to finish.
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewOn 2,154 expert-curated multiple-choice questions across eight architectural categories, 11 current models all score above 90% zero-shot (best about 94%), with quality-attribute and pattern questions near ceiling, but accuracy varies by category and drops for smaller models on the harder tiers (to about 85-87%), so declarative architectural knowledge is largely present while the benchmark cannot show whether it is applied in design.
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewOn 430 tasks distilled from real professional-forum threads across seven domains, the best model (GPT-5) scores about 54%, under the 60% passing line the authors set, with computer science and applied technology lowest at roughly 40-45%, so long-tail practitioner knowledge remains a gap even where standard benchmarks are saturated.Capability: Competence in a specialist domain
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewOn 450 Lean 4 theorem targets in graduate stochastic processes — a field thinly covered by Mathlib — a tool-using frontier agent (Opus 4.8 with Lean LSP access, one 15-minute run per problem) proves only 34.9%, and the rate varies widely by topic, from about 5% on renewal processes to about 62% on martingales and stopping.Capability: Formalizing and machine-checking mathematics
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewOn 613 research-level theorems from six real Lean Blueprint formalization projects, the best current prover reaches only about a 10% pass rate, so progress on curated benchmarks such as miniF2F does not carry over to theorems embedded in live formalization projects with their own definitions and lemmas.Capability: Formalizing and machine-checking mathematics
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-26· pending-reviewOn a 30-task ladder spanning instruction following, tool use, multi-step coordination and long-horizon planning, evaluated over 16 open-weight models from 0.27B to 32B against GPT-5 across 16,542 scored runs, small and mid-sized open models are already sufficient for the short-horizon structured tool use that makes up most calls in real agent pipelines, and the strongest open-weight model matches GPT-5 in aggregate — but the gap reappears on long-horizon planning that needs sustained coordination and constraint tracking, where frontier models still lead and neither side is reliable, and the boundary is not explained by model scale alone.Capability: Using the tools it is given
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewOn a suite of verifier-checked tool-using ReAct task families (arithmetic, SQL analytics, security judgment, URL grounding, dependency planning, orchestration, tool policy), a single task-weighted aggregate accuracy hides inverted per-family rankings: the model that ranks last overall can lead both URL-grounding families while scoring lowest on structured SQL and multi-step tool families, which spread the four tested models by more than sixty points.Capability: Whether the measurement made the finding
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewOn adversarial video hallucination pairs scored only when a method both accepts the factual statement and rejects the counterfactual one, purpose-built video hallucination detectors (embedding-similarity, learned-entailment, atomic-fact and attention-based) top out around 34.63% overall accuracy, well below a general frontier video-language model prompted to judge the same items (about 83.63%).Capability: Stating false facts confidently
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewOn anonymized retrospective social-event forecasting (entities replaced, dates shifted), frontier LLMs score much better at assigning low probability to events that did not happen than at assigning high probability to events that did, with a mean false-minus-true calibration gap of about 15.9 points across six models and up to 38 points for one model, so benchmarks weighted toward invented events overstate forecasting skill.Capability: Predicting future events
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewOn audio temporal-localization tasks in 11 Southeast Asian languages, current audio LLMs return time spans whose coverage of the true span consistently exceeds their purity, and F1 falls steadily as the recording grows from under 30 seconds to 3 minutes, with several models unable to run at all on the longer bins.Capability: Understanding speech and audio
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewOn clinical-calculator tasks where the formula, gold variables and full note are all supplied, having an open-weight model write case-specific Python for a restricted executor instead of doing the arithmetic itself gives no reliable gain at 7B (+3.29 pp, calculator-cluster CI crossing zero) but a clear gain at 32B (+7.05 pp, CI clear of zero), and the remaining errors are invented unit conversions and one-tier slips in banded scores rather than raw computation.Capability: Digit-level arithmetic
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewOn code prompts that are unsatisfiable by design, open-weight code and reasoning models refuse based on how suspicious the request looks rather than how deeply impossible it is: they refuse famous theoretical impossibilities far more often than requests for plausible-sounding nonexistent packages, and the stronger models in the set close the gap on theory while barely improving on fabricated ecosystem entities.Capability: Stating false facts confidently
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewOn complex clinical diagnosis from published case reports (rare disease and challenging cases), running several presenter-rebutter debater pairs on one backbone and synthesizing their revised hypotheses with an aggregator raises exact-match diagnostic accuracy and rated safety over the same model prompted once, while giving no measurable gain on multiple-choice clinical knowledge questions.Capability: Getting more from several models than from one
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewOn county-level US knowledge (14,782 questions over 526 counties), the best of 13 models reaches only about 57% on narrative questions and no model exceeds 13% on numeric Census facts, larger models are not reliably better, and web search helped Gemini (+14 points) while hurting GPT models (about -11), so hyper-local recall is thin and retrieval does not fix it uniformly.Capability: Recalling facts it was trained on
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewOn cross-system urban tasks where several services must be chained under matched tools and budgets, the task-success gap between orchestration strategies is not explained by required-tool coverage, call ordering, or asking for missing inputs — those are near-saturated across strategies — but by whether the agent goes beyond the minimal toolchain to refine coarse results and verify each candidate against the stated constraints before answering.Capability: Using the tools it is given
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewOn EEG and spike neuroscience classification benchmarks, a multi-agent LLM pipeline with planning, code generation and closed-loop retry does not by itself beat a single-shot LLM coding agent; the measured gain comes from making domain-specific pre-trained neural models addressable through one interface and choosing among them per dataset.Capability: Competence in a specialist domain
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewOn expert-authored Saudi-dialect prompts scored against rubric criteria derived from expert ground truths, four frontier models all score under 55% and their dominant failure is distorting register and flattening pragmatic nuance (37.3% of 466 errors) rather than stating falsehoods (11.2%), so MSA-fluent models are not dialectally competent.Capability: Handling culture-specific norms and viewpoints
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewOn factual QA (817-item TruthfulQA multiple-choice, plus a 300-item Natural Questions short-answer set), a prompt-only three-stage gate in which the model lists the information units a question needs, scores its support for each, and answers only when the mean score clears a threshold lowers the unconditional wrong-answer rate and raises accuracy on answered items for all eleven models tested, but most of the abstentions it produces fall on questions plain chain-of-thought answered correctly (about 69% at the primary operating point), so the gain is bought with lost coverage rather than with selective self-knowledge.Capability: Saying it does not know
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewOn forecasting next-year music festival lineups (link prediction between artists and festivals), zero-shot LLMs prompted for ranked candidate lists score far below specialized temporal-knowledge-graph forecasting models trained on the domain graph, and their errors are dominated by popularity bias toward globally famous artists plus duplicate entries in the ranked list.Capability: Predicting future events
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewOn four diagram-grounded geometry benchmarks a multi-agent pipeline consistently helps open-source vision-language models (Qwen-2.5-VL 7B +6.8 and 32B +3.3 points on Geometry3K, with further gains on OlympiadBench and We-Math) but a strong closed model (Gemini 2.0 Flash) generally did better as a single agent on the classic benchmarks, gaining only modestly on the newest one, so agentic decomposition pays for weaker models and less familiar tasks rather than universally.Capability: Getting more from several models than from one
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-15· pending-reviewOn function-level algorithmic tasks translated to C#, a model's functional test pass rate carries almost no information about the structural quality or runtime cost of the code it writes: across 340 solutions from four frontier models, correctness correlated near zero with a composite quality score (r = 0.075) and model rankings by correctness reversed against rankings by quality, while the structurally simplest generator was also the most memory-hungry.Capability: Whether the measurement made the finding
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewOn Hofstede's VSM13 survey, most of eight current chat models answer closest to the United States when no culture is specified; a system prompt naming a target country moves seven of the eight toward that country's profile, with the largest gains for the countries furthest away at baseline (Japan and China, roughly 17-19% closer), yet those two remain the hardest to match even for the DeepSeek models.Capability: Handling culture-specific norms and viewpoints
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewOn hour-scale (mean 88.8 min) open-ended streaming video QA with causal access, every streaming memory design degrades as the evidence-to-query gap grows, and they fail in different ways: recent-window systems cannot recover distant events (AURA 25.4% within 5 minutes falling to 10.5% beyond 30 minutes), text-summary systems keep temporal coverage but discard visual evidence, and model-internal compression loses fine-grained detail (StreamForest 14.4%, ThinkStream 7.5% on retrospection), while human accuracy without rewatching is 63.4%.Capability: Reasoning about time in video
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewOn human-annotated step-level verification of frontier-model proofs to recent IMO and Putnam problems, most of 29 generative critics and process reward models do little better than chance at finding the first wrong step, a leading math PRM (Qwen2.5-Math-PRM-72B) reaches only about 37% balanced F1, and beyond a few standouts open-source verifiers lag closed models; verification is harder than generation for the same frontier models.Capability: Spotting an error in a given artifact
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewOn knowledge, math and medical QA benchmarks with 8B open backbones (Llama-3.1-8B-Instruct, Qwen3-8B), the accuracy advantage of multi-agent debate pipelines over a single agent can be matched by a single-turn single agent whose hidden state is steered by a query-composed mixture of role vectors, at roughly a twentieth of the token cost, so the benefit of these pipelines comes from multi-perspective specialization rather than from inter-agent text exchange.Capability: Getting more from several models than from one
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewOn language-conditioned manipulation tasks where the same base task is instantiated at increasing levels of spatial ambiguity and procedural length, current VLA models degrade on fine-grained spatial relations, precise low-level execution, and memory-dependent multi-step planning, so success measured only on predefined simple settings overstates their competence.Capability: Turning plans into physical action
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewOn long-context tasks over 8K-128K-token documents in the six official UN languages, models answer word-level span questions well (about 73% average accuracy across 12 models) but degrade sharply as the required unit coarsens to sentence selection (about 21%), paragraph filling and document-level summarization and translation, so long-context competence measured by needle-style word retrieval overstates comprehension of larger discourse units.Capability: Losing information in long inputs
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewOn long-video (10-150 minute) audio-visual reasoning questions, multiple-choice accuracy overstates grounding: a large open model can answer about 55% of items correctly while scoring 8.5 on spatio-temporal evidence localization, whereas a frontier proprietary model at 73% accuracy scores 35.2 on the same evidence metric, so answer accuracy alone cannot distinguish deduction from prior-driven guessing.Capability: Reasoning about time in video
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewOn month-level egocentric recordings (20-120 days per participant) with human-written multiple-choice questions, current multimodal models degrade as the question moves from inferring recurring habits, to retrieving one specific episode, to composing evidence across days — the best model reached 71.8% macro-average against a 94.2% human baseline, with several models near or below the 25% chance level on route reasoning, cross-view spatial reasoning and direction judgement.Capability: Reasoning about time in video
- observationsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03On multi-document question answering with GPT-3.5 Turbo's 16k-context variant, accuracy dropped by more than twenty points when the document containing the answer was moved from the start or end of the context to the middle, with nothing else about the task changed.Capability: Losing information in long inputs
- supportsLost in the Middle: How Language Models Use Long ContextsThis source: Lost in the Middle: How Language Models Use Long ContextsHow much the field cites it — very heavily cited in the last 12 months1000+ in the last 12 months · 4887 totalpublished 2023checked 2026-09-04count capped at 1000 by the fetch
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewOn multi-hop video questions written so no single frame or timestamp lookup suffices, single-turn MLLMs answer poorly (0.07-0.51 accuracy across 12 proprietary and open models), and wrapping the same models in an agentic loop helps only when the loop lets the agent choose what, where and at what granularity to look: frameworks giving adaptive fps/resolution/interval control raised accuracy and trajectory scores for nearly every model, while a fixed-schedule framework with weak perception tools (STAR) fell below the single-turn baseline for all of them.Capability: Reasoning about time in video
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewOn multi-session dialogue questions whose answers depend only on how an utterance was delivered, a cascaded ASR-then-LLM pipeline (Whisper large-v3 into a frontier model) scores below 7B audio-native models on the same clips, and below even a transcript-only baseline, because the transcription step drops delivery cues entirely and adds content errors.Capability: Understanding speech and audio
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewOn multi-turn tool-calling benchmarks that score whether the final environment state matches a gold trajectory, aggregate accuracy hides wrong action-class choices: a model can call a withheld or under-specified tool with fabricated arguments, be handed the missing piece on a later turn, and still be graded PASS — so heavily tool-trained model families rank high while almost never asking for a missing parameter or refusing an unavailable function.Capability: Whether the measurement made the finding
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewOn multi-year authentic personal photo/video archives turned into memory streams, frontier multimodal models and structured memory systems recall facts and infer persona far better than they predict a user's future choices: predicting or recommending in novel contexts caps around 59 on a 0-100 scale for every system tested, and compressing the history ~70x into a memory system costs nothing on that tier while costing a lot on factual recall, so predictive personalization is limited by preference representation rather than by retrieval or context length.Capability: Remembering across sessions
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewOn multimodal cultural-interpretation multiple-choice questions, pipelines that pool several same-backbone evidence streams (perception, retrieval, web verification) and weight them by scored relevance beat majority voting, self-consistency and agent debate mainly on the items where the streams disagree, and the margin shrinks sharply as the backbone gets stronger.Capability: Getting more from several models than from one
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-15· pending-reviewOn multiple-choice audio questions where each answer choice is paired with a generated image, adding the images helps native multimodal models most on environmental-sound questions and not at all on speech (speech is best audio-only), and in cascaded pipelines that convert audio to transcripts or captions first the images lower average accuracy.Capability: Understanding speech and audio
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewOn multiple-choice questions derived from industry cybersecurity certifications, frontier models reach human-expert level on general networking and IT security, but accuracy falls on vendor-specific material and formal standards, staying below 40% on the hardest ISA/IEC 62443 and Fortinet NSE questions; general security benchmarks such as MMLU Computer Security are saturated near 100% and no longer separate models.Capability: Competence in a specialist domain
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewOn natural-language logical reasoning benchmarks, auto-formalizing each reasoning step into SMT-LIB and regenerating from the first step the solver cannot verify raises both the share of solver-verified steps and final-answer accuracy over plain chain-of-thought, and greedy regeneration at the failure point beats beam search or MCTS on step verifiability.
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-18· pending-reviewOn natural-language stock questions that require judging a future condition (e.g. whether a price will exceed a threshold next month), open-weight 20B-31B LLMs reasoning directly from the retrieved historical series answer near chance, and routing the question through an explicit SQL-retrieval plus time-series-forecasting toolchain raises accuracy to roughly the level of human stock investors.Capability: Predicting future events
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewOn natural-language-to-SQL synthesis over multi-table databases, a model's accuracy overstates its ability whenever the same database or the same query appears in both training and test: splitting by database rather than by example cut the best system's exact-match accuracy from 33.0% to 12.4%, and accuracy falls further as the target database has more foreign keys.Capability: Writing a reusable locator: regex, SQL, XPath
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-15· pending-reviewOn numeric sequences whose first differences repeat only after a long prefix, LLaMA 3.1-8B extrapolates by computing and locally storing each first difference in its hidden states, retrieving the delta that followed the last observed one through an induction-like attention step, and adding it to the last number — so the extrapolation is a composed arithmetic operation over latent structure rather than token-level copying.Capability: Digit-level arithmetic
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewOn paired solvable/unsolvable math problems translated into French and Greek, natively multilingual models (Gemma-4-31B, Qwen3-4B/30B) encode solvability most sharply in English yet agree least with that internal belief in their stated verdict in English, being more faithful in the lower-resource languages, while a weakly multilingual model (Llama-3.1-8B) shows the opposite ordering.Capability: Saying what actually drove the answer
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-07· pending-reviewOn questions whose premises are false or internally contradictory, prompting mid-size instruct models (Qwen2.5-7B, LLaMA-3.1-8B, Gemma3-12B) to reason step by step can lower accuracy below plain answering, because the reasoning chain elaborates from the flawed premise instead of challenging it.Capability: Stating false facts confidently
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewOn real C/C++ faults with reproducing tests, 24 current models repair only about 11% of the 248 bugs and about 6% of the 102 vulnerabilities, far below their Java and Python repair rates on comparable benchmarks, and fine-tuning on the benchmark's commit corpus raised Pass@k in 21 of 28 comparisons by an average 85% relative, so the gap is partly a training-data gap.Capability: Finding and fixing a fault in existing code
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03On realistic web tasks with explicit goals, the best model completed only a small fraction end to end, far below human performance.Capability: Following an unfamiliar procedure
- supportsWebArena: A Realistic Web Environment for Building Autonomous AgentsThis source: WebArena: A Realistic Web Environment for Building Autonomous AgentsHow much the field cites it — very heavily cited in the last 12 months1000+ in the last 12 months · 1931 totalpublished 2023checked 2026-09-04count capped at 1000 by the fetch
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewOn repository-grounded LLM-infrastructure engineering tasks, frontier agents score far below ceiling (best 36.53% overall), and every one of eight models scores lower on long-horizon multi-file implementation than on single-file kernel completion, even though the full repository and test cases are visible.
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewOn repository-level issue-fixing tasks (SWE-bench Lite/Verified/Pro), spending effort upstream — expanding the issue text into a structured requirement spec from retrieved repo context, then testing and refining that spec via generated code and tests — raises resolve rate over agents that consume the raw issue description, and raises the share of patches that even apply cleanly by a larger margin than it raises correctness.Capability: Generating and editing working code
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewOn repository-level issue-fixing tasks (SWE-bench Lite/Verified/Pro), spending effort upstream — expanding the issue text into a structured requirement spec from retrieved repo context, then testing and refining that spec via generated code and tests — raises resolve rate over agents that consume the raw issue description, and raises the share of patches that even apply cleanly by a larger margin than it raises correctness.Capability: Generating and editing working code
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewOn research-level theorem-proving tasks drawn from recent FOCS/STOC/SODA papers, wrapping a frontier model in a staged harness that explores strategies, gates when to decompose, solves section-level subproblems along a dependency graph, and routes verifier findings back to the affected section raises graded proof accuracy far above prompting the same model directly (30.3% to 54.0% for Gemini 3.1 Pro on TCS-Bench), and two such runs with different backbones make complementary errors, so selecting between them by sampled critiques reaches 71.0% against an oracle ceiling of 77.3%.
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-07· pending-reviewOn rubric-graded long-context tasks, most of the gain from a context- compilation harness comes from extracting the context's rules, exact terms and output spec into an explicit checklist placed in the prompt, not from the executable verifiers built on top of it — and the gain appears only on rule-dense tasks and on models with enough capacity, disappearing or reversing on open-ended tasks and on a small-activation model.Capability: Losing information in long inputs
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewOn scientific tasks where a domain-specific constraint and a general presentation constraint (letter case, output format, structure) are both stated in the same prompt, multimodal models satisfy the domain-specific constraint more often than the general formatting one, so domain competence does not imply full instruction compliance.Capability: Following an unfamiliar procedure
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewOn self-contained linguistic Rosetta Stone puzzles where a single context example uniquely carrying a question-required token is deleted, frontier models told to output N/A when context is insufficient almost never abstain, and on some puzzles answer the damaged variant as well as or better than the original.Capability: Saying it does not know
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewOn SelfAware, a set of unanswerable questions (no scientific consensus, imagination, philosophical, too many variables, subjective) paired with answerable ones, models' ability to flag the unanswerable ones rises with parameter count, with instruction tuning and with in-context examples, yet the best model tested (GPT-4, F1 about 75) stays well below the human reference of about 85, so self-knowledge is present but incomplete.Capability: Saying it does not know
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewOn simple HLS kernel generation tasks (85 designs from CHStone, MachSuite, Polybench, Rosetta, C2HLSC), giving a modest open-source model (gpt-oss-120b) a shell with a C++ compiler for self-verification and iterative refinement saturates the benchmark at pass@10 (100% through synthesis) and exceeds 90% at pass@1, so this class of task no longer separates models; repeated sampling helps the weaker model far more (+60 points synthesis pass rate at k=10 for gpt-oss-20b vs +11 for the 120b).Capability: Generating and editing working code
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewOn single-step retrosynthesis for real industrial target molecules, frontier general LLMs produce chemically infeasible top-1 routes (unresolved selectivity or no mechanistic basis) more often than specialized retrosynthesis models, even though their reference-matching recall is comparable, so disconnection fluency does not carry selectivity awareness.Capability: Stating false facts confidently
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewOn span-level hallucination-detection leaderboards where absolute scores are low, bootstrap resampling of the test set moves top systems across wide rank intervals, so point-estimate ordering does not establish that one detector beats another.Capability: Whether the measurement made the finding
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewOn statutory tax calculation cases, models asked to solve an input that contains an injected contradictory fact mostly compute straight through it and return the clean-input answer, while the same model asked only to verify the same input flags most of those contradictions \u2014 so abstention failure here is a task-framing gap, not missing detection ability.Capability: Saying it does not know
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewOn SWE-bench Lite, asking the model to name suspicious files from the issue plus a repository tree finds the ground-truth file in about 79% of cases and embedding retrieval over the issue text in about 68%, and taking the union of the two raises file-level localization to about 82% because the two methods miss different files; a fixed three-step narrowing (files, then classes and functions, then edit lines) with no agent loop then supported the best open-source resolve rate at the time (32%).Capability: Finding what matters in a large codebase
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewOn text-to-SQL benchmarks, aggregate accuracy hides a steep difficulty gradient: pre-trained-language-model-based parsers resolved roughly 80% of Spider cases overall but only about 50% of its extra-hard cases, and as natural-language understanding improved the residual errors shifted toward database-side problems — large schemas, ambiguous columns, dirty cell values — rather than language parsing.Capability: Writing a reusable locator: regex, SQL, XPath
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewOn text-to-SQL questions that are underspecified with respect to the schema, retrieving clause-level fragments (conditions, joins, calculations, groupings, output specs) from a database's own past successful SQL queries and prepending them to existing few-shot pipelines raises execution accuracy across four different pipelines, but the gain saturates and can reverse once too many candidates are retrieved, because high recall with low precision lets the generator pick a plausible-but-wrong pattern.Capability: Writing a reusable locator: regex, SQL, XPath
- observationsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-07On the best current calibration, AI is not yet accelerating its own development in a self-sustaining way: the modelled threshold is that a one-unit gain in model capability must buy at least 15% higher AI R&D productivity, and the back-of-envelope figure from reported engineer uplift since coding agents launched is about 9% — below it, but rising, so the gap is a current reading rather than a ceiling.Capability: Whether the measurement made the finding
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03On verifiable instructions such as length and format constraints, strong models still fail a meaningful share, and failures grow when several constraints apply at once.Capability: Following an unfamiliar procedure
- supportsInstruction-Following Evaluation for Large Language ModelsThis source: Instruction-Following Evaluation for Large Language ModelsHow much the field cites it — very heavily cited in the last 12 months625 in the last 12 months · 1081 totalpublished 2023checked 2026-09-04
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-19· pending-reviewOn web-browsing (WebArena, Mind2Web) and software-engineering (SWE-Bench-Verified) benchmarks, an agent that distills reusable strategies from both its successful and its failed attempts, judged by an LLM without ground truth, and retrieves the relevant ones on later tasks, outperforms memories that store raw trajectories or only successful routines — up to 20% relative improvement in success and up to 16% fewer interaction steps — with the weights unchanged; including the failures is part of the gain, not noise.Capability: Improving its own scaffolding from failures
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-18· pending-reviewOn zero-shot binary risk classification of rare weather-driven power outages from tabular weather features, GPT-4o/4.1-class models trail supervised classifiers trained on the same data by 0.04-0.13 macro-F1 and have much lower precision, and they gain little from being given the full forecast window rather than only current observations, while supervised models gain more from forecast features.Capability: Predicting future events
- mechanismmechanism reasoning↑1↓0Reviewed by AIchecked 2026-09-18· pending-reviewOnce an AI system is much smarter than the people overseeing it, control mechanisms such as a kill switch stop being reliable, because the system can persuade or work around its operators; the only natural example of a less intelligent agent reliably controlling a more intelligent one is a mother controlled by her infant, which works because the mother was built to care, so the safeguard has to be built into what the system wants — "maternal instincts", caring about people — rather than imposed on it from outside, and nobody yet knows how to build that.Capability: Prioritizing safety under conflicting goals
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03Only models with substantial code pretraining track entity state through a sequence of operations, and all degrade as the sequence lengthens.Capability: Tracking state through a long task
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewOpen-weight instruction-tuned models, told in plain language to not think about a concept while transcribing a sentence, suppress that concept's residual-stream projection enough to evade linear probes, the Jacobian lens, activation oracles and natural-language autoencoders, with no fine-tuning or prompt search — detection at a 1% false-positive budget falls to roughly 5-21% under suppression while engagement is caught near ceiling.Capability: Strategic deception and detecting it
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewOutside mathematics, the step-level error detection of both process reward models and general LLMs is markedly weaker than on math benchmarks, and the two fail differently: PRMs miss knowledge-based errors while LLMs miss computational errors, across science and logic domains.Capability: Spotting an error in a given artifact
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03Performance on multi-document QA is highest when the answer is at the start or end of the context and drops sharply in the middle.Capability: Losing information in long inputs
- supportsLost in the Middle: How Language Models Use Long ContextsThis source: Lost in the Middle: How Language Models Use Long ContextsHow much the field cites it — very heavily cited in the last 12 months1000+ in the last 12 months · 4887 totalpublished 2023checked 2026-09-04count capped at 1000 by the fetch
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03Position embeddings that encode digit place let small transformers add 100-digit numbers, showing the limit is representational not fundamental.Capability: Digit-level arithmetic
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewPresenting syllogisms as Euler or linear diagrams instead of text or logical notation did not consistently improve two models' accuracy on 285 problems; both did well on entailment and contradiction items but poorly on neutral ones (GPT-4o-mini under 30% on neutral cases), and made systematic conversion errors, treating 'all A are B' as if it implied 'all B are A'.Capability: Deducing validly from logical structure
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewPrompt injection defenses that report near-zero attack success on short-context benchmarks lose most of that protection when the injected instruction sits inside a document of thousands to tens of thousands of tokens: fine-tuned separation defenses and detect-localize-remove pipelines still let a large share of injections through on paper review, resume screening, code review and email threads.Capability: Following instructions hidden in data
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewPrompting a model to behave as if its knowledge ended at an earlier date works for facts asked about directly (about 82% unlearning success) and for word-sense shifts (about 70%), but fails when the post-cutoff knowledge is only causally connected to the question (about 19%), so a simulated cutoff cannot be trusted to remove contamination from temporal prediction tests.Capability: Recalling facts it was trained on
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewPrompting with a Task-Method-Knowledge model, which spells out what to do, how, and why in a hierarchical structure, lifted o1 on the obfuscated Random Blocksworld set of PlanBench from about 32% to 97%, and o1-mini from about 9% to 27%, gains far beyond the roughly 1% that zero-shot chain-of-thought gave on PlanBench in earlier work; the smaller model did not benefit on the Mystery variant.Capability: Planning a sequence of actions before acting
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03Recursively summarizing the dialogue into a running memory improves consistency in long conversations.Capability: Remembering across sessions
- supportsRecursively Summarizing Enables Long-Term Dialogue Memory in Large Language ModelsThis source: Recursively Summarizing Enables Long-Term Dialogue Memory in Large Lan…How much the field cites it — heavily cited in the last 12 months51 in the last 12 months · 93 totalpublished 2023checked 2026-09-04
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewRefusal learned by alignment tuning lives in a low-rank subspace of hidden-state activations that recurs across model families; ablating directions in that subspace lowers harmful-request refusal, and a scalar computed from the subspace's geometry before any fine-tuning (the Geometric Fragility Score) picks out which initially safe model will keep the most refusal after small benign LoRA fine-tuning.Capability: Retaining skills and safety after fine-tuning
- mechanismmechanism reasoning↑2↓0Reviewed by AI and Russ Millerchecked 2026-09-07Reporting how much a lab's models are used in its own research — tokens, lines of code, inference compute, experiments per researcher — cannot show that the models are accelerating the research, because every one of those is an input. Establishing the loop needs an outcome variable over time: algorithmic efficiency as a function of capability, which is the edge that turns a pipeline into a feedback loop.Capability: Whether the measurement made the finding
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-22· pending-reviewReward-related information in a reasoning model's hidden states is concentrated in a sparse set of neurons: a linear probe trained with a temporal-difference objective finds 'value neurons' (under 1% per layer) whose activations predict expected correctness, and zeroing them in one layer drops MATH500 accuracy from 75% to an average 20% (as low as 1%), while zeroing the same number of random or magnitude-selected neurons costs 1-9 points; the step-to-step change in the probed value ('dopamine neurons', a stated analogy) used as a process reward for inference-time search lifted MATH500 from 72.2 to 77.8, above an implicit-PRM baseline at 75.0.Capability: Spotting an error in a given artifact
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03Rewarding sycophantic behavior generalizes to more serious specification gaming in a small fraction of cases.Capability: Telling the user what they want to hear
- supportsSycophancy to Subterfuge: Investigating Reward-Tampering in Large Language ModelsThis source: Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Lang…How much the field cites it — heavily cited in the last 12 months89 in the last 12 months · 158 totalpublished 2024checked 2026-09-04click for Semantic Scholar
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewRewriting a harmful request in an emotional or narrative style (fearful, curious, compassionate) while keeping its intent raises jailbreak success by up to 57 percentage points across 16 instruction-tuned models, contextual rewrites beating fixed templates, and the vulnerability does not shrink with model scale; a preprocessing step that has a second model strip stylistic cues cuts the success rate substantially.Capability: Refusing harmful requests under obfuscation
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03Roughly forty percent of Copilot completions in security-relevant scenarios were vulnerable.Capability: Writing secure code and dependencies
- supportsAsleep at the Keyboard? Assessing the Security of GitHub Copilot's Code ContributionsThis source: Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Cod…How much the field cites it — very heavily cited in the last 12 months413 in the last 12 months · 952 totalpublished 2021checked 2026-09-04click for Semantic Scholar
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-20· pending-reviewRouting the gradient updates from monitor-flagged samples into a sacrificial adapter and deleting it after training avoided monitor-exploiting policies in every toy environment tested, improved Qwen3-8B on a coding task without raising its propensity to reward hack, and in a supervised variant trained Qwen3-32B on agentic coding data so that it never reward hacked under prompted elicitation while matching the solve rate of training on perfectly filtered data; detecting only some forms of the hack was enough to suppress the undetected forms as well.Capability: Gaming the measure instead of doing the task
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewSafety realignment after a poisoned fine-tune is brittle in three ways: existing defenses (SSRD, RESTA, soft-SFT, rollback) often destroy the downstream skill they were meant to keep, lose more than 30% of their effect when the defender's prompt template differs from the attacker's, and a realigned model can be re-jailbroken by switching the system prompt; distilling on-policy from two frozen teachers, the original model for refusal and the fine-tuned model for the task, routed per token, cut attack success to about 2% under the defender's template while holding task score near pre-attack level.Capability: Retaining skills and safety after fine-tuning
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03Self-supervised training on tool-call examples teaches a model when and how to call simple tools, narrowing the gap on tasks that need them.Capability: Using the tools it is given
- supportsToolformer: Language Models Can Teach Themselves to Use ToolsThis source: Toolformer: Language Models Can Teach Themselves to Use ToolsHow much the field cites it — very heavily cited in the last 12 months1000+ in the last 12 months · 5422 totalpublished 2023checked 2026-09-04count capped at 1000 by the fetchclick for Semantic Scholar
- mechanismreplicated↑2↓0Reviewed by AI and Russ Millerchecked 2026-09-04Separating instructions from data raises injection resistance substantially, but both published versions get their strength from fine-tuning the model on the separation, and both report improved robustness rather than elimination. Treat it as one layer of defense in depth; the prompt-only variant, without training, has no measured efficacy behind it here.Capability: Following instructions hidden in data
- supportsStruQ: Defending Against Prompt Injection with Structured QueriesThis source: StruQ: Defending Against Prompt Injection with Structured QueriesHow much the field cites it — very heavily cited in the last 12 months243 in the last 12 months · 394 totalpublished 2024checked 2026-09-04
- supportsThe Instruction Hierarchy: Training LLMs to Prioritize Privileged InstructionsThis source: The Instruction Hierarchy: Training LLMs to Prioritize Privileged Inst…How much the field cites it — very heavily cited in the last 12 months285 in the last 12 months · 486 totalpublished 2024checked 2026-09-04click for Semantic Scholar
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03Separating prompt and data with reserved delimiters and fine-tuning on that structure blocks most injections at little cost to utility.Capability: Following instructions hidden in data
- supportsStruQ: Defending Against Prompt Injection with Structured QueriesThis source: StruQ: Defending Against Prompt Injection with Structured QueriesHow much the field cites it — very heavily cited in the last 12 months243 in the last 12 months · 394 totalpublished 2024checked 2026-09-04
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03Shows LLMs hallucinate API names and arguments when calling large API collections from memory, and that retrieving documentation reduces this.Capability: Using the tools it is given
- supportsGorilla: Large Language Model Connected with Massive APIsThis source: Gorilla: Large Language Model Connected with Massive APIsHow much the field cites it — very heavily cited in the last 12 months805 in the last 12 months · 1585 totalpublished 2023checked 2026-09-04
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03Simple goal-hijacking and prompt-leaking attacks succeed against production models with short adversarial strings.Capability: Following instructions hidden in data
- supportsIgnore Previous Prompt: Attack Techniques For Language ModelsThis source: Ignore Previous Prompt: Attack Techniques For Language ModelsHow much the field cites it — very heavily cited in the last 12 months475 in the last 12 months · 1055 totalpublished 2022checked 2026-09-04click for Semantic Scholar
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewSplitting a task across several agents helps on long-horizon tasks whose parts depend on each other sparsely, and hurts on tightly coupled sequential workflows, where generic multi-agent methods fall below a single-agent harness because decomposition fragments the dependency chain and coordination overhead outweighs the context isolation it buys; as single agents improve the collaboration benefit shrinks.Capability: Getting more from several models than from one
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-04State-of-the-art models such as GPT-4 can understand and induce false beliefs in other agents through deliberate strategic reasoning, a capability that was absent in earlier-generation language models.Capability: Strategic deception and detecting it
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-04Static, fixed-problem coding benchmarks saturate and leak into training data over time, so getting an honest read on current code-generation ability requires a benchmark that continuously adds newly-published problems rather than reusing an older fixed set like HumanEval.Capability: Generating and editing working code
- supportsLiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for CodeThis source: LiveCodeBench: Holistic and Contamination Free Evaluation of Large Lan…How much the field cites it — very heavily cited in the last 12 months1000+ in the last 12 months · 2073 totalpublished 2024checked 2026-09-04count capped at 1000 by the fetchclick for Semantic Scholar
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03Surveys causes, kinds, and detection of hallucination across data, training, and inference, and finds no complete solution.Capability: Stating false facts confidently
- supportsA Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open QuestionsThis source: A Survey on Hallucination in Large Language Models: Principles, Taxono…How much the field cites it — very heavily cited in the last 12 months1000+ in the last 12 months · 3664 totalpublished 2023checked 2026-09-04count capped at 1000 by the fetch
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03Swapping candidate order alone can flip the judge's preference, so an evaluator can be gamed by ordering.Capability: Biased when judging other outputs
- supportsLarge Language Models are not Fair EvaluatorsThis source: Large Language Models are not Fair EvaluatorsHow much the field cites it — very heavily cited in the last 12 months506 in the last 12 months · 1242 totalpublished 2023checked 2026-09-04
- mechanismmechanism reasoning↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-04Sycophancy is triggered by the user's stated view being in context, so the cheapest control is not putting it there — ask for the answer before the opinion, or withhold the opinion entirely. Fine-tuning is the answer for the cases where the opinion has to be in context and the answer still must not move.Capability: Telling the user what they want to hear
- supportsSimple synthetic data reduces sycophancy in large language modelsThis source: Simple synthetic data reduces sycophancy in large language modelsHow much the field cites it — heavily cited in the last 12 months92 in the last 12 months · 181 totalpublished 2023checked 2026-09-04
- mechanismmechanism reasoning↑3↓0Reviewed by AIchecked 2026-09-11· pending-reviewThe agent that produced an artifact is a biased judge of it — it holds the context and the incentives that skew its assessment — so verification belongs with a deterministic sensor or a separate verifier that reports failures back rather than rewriting the output, and in a multi-agent system the verifier is the one component no agent may override.Capability: Fixing its own mistakes
- supportsHarness Engineering — Agent = Model + Harness: The 6-Layer Production Playbook
- supportsLarge Language Models Cannot Self-Correct Reasoning YetThis source: Large Language Models Cannot Self-Correct Reasoning YetHow much the field cites it — very heavily cited in the last 12 months503 in the last 12 months · 1164 totalpublished 2023checked 2026-09-04
- supportsHarness engineering for coding agent users
- observationsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-04The large reported gains from fine-tuning on tool-call trajectories come from small open models starting far behind — distilling a few hundred trajectories from a stronger model moves them a long way. Whether the same procedure helps a model that is already good at tool use is a separate question this evidence does not answer.Capability: Using the tools it is given
- observationreplicated↑2↓0Reviewed by AIchecked 2026-10-19· pending-reviewThe published research literature overreports results that support the hypothesis tested: in psychology, 96% of a random sample of standard hypothesis-testing papers reported a positive first result against 44% of Registered Reports, which are accepted before results are known; and across all disciplines the share of hypothesis-testing papers reporting positive support rose from 70.2% in 1990-1991 to 85.9% in 2007 — so any index built from published papers, this catalog included, starts from a sample tilted toward what worked.Capability: Whether the measurement made the finding
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewTool-augmented multimodal models trained only on how to call vision tools accept tool outputs uncritically; adding supervised trajectories where tool evidence is checked against the image and re-queried, plus a reward for correction that reaches a correct answer, raises accuracy on perception-heavy benchmarks (counting, depth, spatial relations) in 3B and 7B Qwen2.5-VL, while slightly lowering it on knowledge-heavy ones.Capability: Fixing its own mistakes
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewTraining a language model with differential privacy — whether DP-SGD pre-training or DP LoRA fine-tuning — makes it generate more false claims about facts in its training data than a non-private counterpart, and the effect grows as the privacy budget tightens, because per-example gradient clipping plus noise prevents low-frequency facts from being acquired at all while flattening the next-token distribution onto incorrect alternatives.Capability: Stating false facts confidently
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03Training against an explicit set of principles produces models that are more harmless without becoming evasive.Capability: Prioritizing safety under conflicting goals
- supportsConstitutional AI: Harmlessness from AI FeedbackThis source: Constitutional AI: Harmlessness from AI FeedbackHow much the field cites it — very heavily cited in the last 12 months1000+ in the last 12 months · 3617 totalpublished 2022checked 2026-09-04count capped at 1000 by the fetchclick for Semantic Scholar
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03Training models to rank system, user, and tool instructions by privilege improves robustness to injections in tool outputs.Capability: Following instructions hidden in data
- supportsThe Instruction Hierarchy: Training LLMs to Prioritize Privileged InstructionsThis source: The Instruction Hierarchy: Training LLMs to Prioritize Privileged Inst…How much the field cites it — very heavily cited in the last 12 months285 in the last 12 months · 486 totalpublished 2024checked 2026-09-04click for Semantic Scholar
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03Training on reversed sequences alongside forward text largely removes the effect.Capability: Not generalizing "A is B" to "B is A"
- supportsReverse Training to Nurse the Reversal CurseThis source: Reverse Training to Nurse the Reversal CurseHow much the field cites it — steadily cited in the last 12 months15 in the last 12 months · 60 totalpublished 2024checked 2026-09-04
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03Training on simple gameable environments generalizes, rarely, to tampering with the model's own reward mechanism.Capability: Prioritizing safety under conflicting goals
- supportsSycophancy to Subterfuge: Investigating Reward-Tampering in Large Language ModelsThis source: Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Lang…How much the field cites it — heavily cited in the last 12 months89 in the last 12 months · 158 totalpublished 2024checked 2026-09-04click for Semantic Scholar
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03Training the model to decide when to retrieve and to critique whether passages support its output improves factuality and citation accuracy.Capability: Checking claims against evidence
- supportsSelf-RAG: Learning to Retrieve, Generate, and Critique through Self-ReflectionThis source: Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Re…How much the field cites it — very heavily cited in the last 12 months1000+ in the last 12 months · 2517 totalpublished 2023checked 2026-09-04count capped at 1000 by the fetch
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewTranslating natural-language planning problems into PDDL can be learned from solver feedback alone, with one model acting as actor, judge and editor: on PlanBench average success rose from 35.5% (LLM+P) to 70.8%, with 66.3% of successes judged faithful to the described task and semantic drift, where a solvable but wrong problem is produced, cut to 6.4%.Capability: Planning a sequence of actions before acting
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-22· pending-reviewTreating post-deployment improvement as scoped behavioural maintenance, where each recurring failure becomes a skill patch with a stated scope and injection point that is promoted only after passing its targeted cases, a protected regression suite and negative controls, raised correctness from 3.70 to 4.55 on an SEC-filing QA benchmark, and over twelve operational rounds promoted only 6 of 33 candidate skills while the monitoring non-correct rate fell from 20.0% to 12.5%; the ungated condition scored higher on each candidate's own targeted cases but admitted regressions on the protected set.Capability: Improving its own scaffolding from failures
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03Under strong goal nudging, several frontier models disabled oversight mechanisms or lied about their actions in a minority of trials.Capability: Prioritizing safety under conflicting goals
- observationsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03Under strong, deliberately adversarial goal pressure in an autonomous-agent setting, Claude 3.5 Sonnet sometimes disabled oversight mechanisms or misreported its own actions to keep pursuing the stated goal — in a minority of trials, and the study explicitly does not claim this happens in ordinary, non-adversarial use.Capability: Prioritizing safety under conflicting goals
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewUnlearning methods that make a model answer "I don't know" about a forget set can leave the knowledge in place: the strongest IDK-style fine-tuning method still answered those same questions accurately when probed in a different format, and across nine unlearning methods on three model families none met a joint standard of utility and honesty on retained knowledge plus consistent, acknowledged forgetting; aligning representations after feature randomization (ReVa) roughly doubled the rejection rate that survives a second conversational turn.Capability: Retaining skills and safety after fine-tuning
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewUsed to steer test-time compute, generative LLM judges (7B-70B) rerank candidate responses about as well as outcome reward models, are consistently worse than process reward models at step-level beam search, and their natural-language critiques, the one thing only a judge can produce, did not guide generators to better responses on math, code or instruction following.
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewUsed zero-shot as step-level action critics in long-horizon tool-calling tasks, frontier models are over-pessimistic — flagging a large share of correct actions and pushing the actor into revision loops that lower success below no critic at all — whereas small models fine-tuned on action-level verification rationales flag far fewer valid actions and raise repeated-run reliability.Capability: Using the tools it is given
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03Users with an AI assistant wrote less secure code and were more confident it was secure.Capability: Writing secure code and dependencies
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewVerifying a chain of thought by decomposing it into nodes and asking binary consistency questions per node localizes errors better than judging the whole chain at once (F1 up 10-25% over CoT-based verifiers, error-localization accuracy far above an 8-vote end-to-end judge) while using 6-58 times fewer tokens, because whole-chain judging dilutes attention and multi-sampling pays for long generations.
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewWhen 4B-8B LLM agents are trained with plain group-based RL (GRPO) to emit variable-length action chunks in text environments, the policy degenerates — collapsing to one action per round on one backbone and over-committing to 5-6-action chunks with lower success on another — because a sparse terminal reward carries no information about where chunk boundaries belong; supplying boundary targets from subskills segmented out of successful trajectories yields an intermediate 3-4-action regime with both higher success and fewer decision rounds.
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewWhen 7-8B open-weight instruct models are fully fine-tuned on a single domain dataset, confining gradient updates to FFN neurons that activate stably across the training set and adding an MSE loss that pulls activations back toward pre-computed activation anchors retains far more general-benchmark ability (MMLU, BBH, TyDiQA) than full-parameter fine-tuning, by about 9.9 points on average, while also raising in-domain scores.Capability: Retaining skills and safety after fine-tuning
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewWhen a benign GitHub issue is augmented with a plausible-sounding maintenance requirement that specifies a helper reading an environment variable into a dangerous sink, repository-level APR agents produce patches that fix the original bug and pass the project test suite while containing the requested vulnerability in roughly half of tasks, and the strongest instruction-following coding models are more susceptible, not less.Capability: Writing secure code and dependencies
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewWhen a calculator-using model is trained with reinforcement learning on verifiable final-answer rewards for an arithmetic search task (Countdown), the gain lands almost entirely at low k — pass@1 rises sharply while pass@16 moves little or falls — because the update can only reinforce correct trajectories the starting policy already samples, and prompt groups with no correct sample supply no gradient.Capability: Using the tools it is given
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewWhen a code-generation prompt combines a prompt-enforced output format (JSON or XML), a persona, and urgency framing at once, pass@1 on function-level Python problems can fall well below the sum of each constraint's individual effect, even when each constraint alone is neutral or helpful — observed in the GPT-4o family and absent in the GPT-4.1 family and o3-mini.Capability: Generating and editing working code
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewWhen a critique's worth is measured by whether a fixed downstream repair agent can improve the interface from it, eight frontier models differ meaningfully in how actionable their UX critiques are, lead on different product-surface categories, and vary in reliability fixture by fixture; the best repair lift was about +0.22 on the benchmark's scale, so the task is unsaturated and not one-dimensional.
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewWhen a document image states multi-step selection constraints in narrative text and the supporting numbers sit in several charts, multimodal models degrade steadily as either the number of logical hops (2 to 5) or the number of charts (2 to 6) grows, with the best model reaching 62.83% overall against over 90% for human annotators.
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewWhen a drone's action space is declared only in the prompt and the model must itself declare arrival, small open multimodal models (2B-9B) enter the 5 m success radius as often as or more often than frontier models yet lose most episodes by declaring arrival too early or never declaring, so the gap at small scale is protocol adherence rather than navigation.Capability: Turning plans into physical action
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewWhen a frozen pretrained time-series forecaster leaves structured residual error, having an LLM propose named semantic features of that residual (e.g. renewable generation over total load) and admitting them through a validation gate into a small post-hoc corrector reduces test error more than statistical feature banks or compositional search over the raw channels, but the gain scales with how much raw error the backbone leaves and vanishes once the backbone already fits the series.Capability: Predicting future events
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-18· pending-reviewWhen a hallucination metric is an unsupported-claim rate computed over whatever claims a model happens to make, models that avoid committing to verifiable claims or directional outputs score best, so the ranking partly measures abstention rather than grounding — on financial chart-to-recommendation tasks the lowest unsupported-claim model made directional recommendations on only 6.4% of queries and ranked near the bottom once coverage was factored in.Capability: Whether the measurement made the finding
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-18· pending-reviewWhen a language model is updated by fine-tuning on a substituted fact, the facts it already answered correctly that are most likely to flip to wrong are those whose answer entity is referenced by many other facts, and updates to such structurally popular facts corrupt connected facts out to five hops in the fact graph — so popularity, not long-tail rarity, predicts collateral corruption among already-correct knowledge.Capability: Stating false facts confidently
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewWhen a language model must judge whether a specific vehicle can perform a described mission, it is overconfident about partially-capable configurations — declaring the mission fully possible despite missing hardware — and supplying the required-capability list plus the vehicle's inferred capabilities from an ontology is what corrects that specific error, not the harder fully/impossible cases which the models already get mostly right.Capability: Planning a sequence of actions before acting
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewWhen a long chain of thought is verified by classifying a graph of its steps, the predictive signal comes almost entirely from the semantic content of the steps, not the graph shape: replacing step embeddings with purely structural metadata (child/parent counts, chain index, position) drops accuracy to near chance on math, science and multiple-choice reasoning, while remaining above chance only on code.
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-18· pending-reviewWhen a mistaken user pushes back adaptively for up to 25 turns, frontier and 7B open models keep abandoning initially correct positions at higher rates the longer the conversation runs, so sycophancy measured with five-turn or fixed-script rebuttals understates it — in one ablation collapse on false-presupposition items rose from 50% by turn 5 to 92% by turn 25 for the same model, and a fixed four-follow-up script reached only 28%.Capability: Telling the user what they want to hear
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03When a model gets a real environment signal after acting — a test result, a tool error, a task-success indicator — reflecting on that signal in words and retrying substantially improves success rates on multi-step coding and decision-making tasks over a single attempt.Capability: Fixing its own mistakes
- supportsReflexion: Language Agents with Verbal Reinforcement LearningThis source: Reflexion: Language Agents with Verbal Reinforcement LearningHow much the field cites it — very heavily cited in the last 12 months1000+ in the last 12 months · 5075 totalpublished 2023checked 2026-09-04count capped at 1000 by the fetch
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-18· pending-reviewWhen a model is asked to answer without visible reasoning, rewriting the same question with candidate answers supplied makes answer-only responses easier to produce, and on Boolean and multiple-choice questions answering directly leaves accuracy unchanged — whereas on open-ended questions there is a trade-off, and forcing an answer-only response costs accuracy.Capability: Cutting to the chase
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewWhen a model is asked to audit many documents for planted errors in a single large-batch prompt, recall of known contaminants collapses and the model reports confident fabricated findings — invented absurd contaminants that appear in no document — rather than signalling incomplete processing.Capability: Stating false facts confidently
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewWhen a model is asked to predict its own rates on behavioral evaluations (sycophancy, reward hacking, agentic policy violation, bias), its answers track its own behavior no better than the same question asked about 'capable AI agents in general' or than other models' answers about themselves, so self-report reflects a generic theory of AI assistants rather than privileged self-knowledge, and first-person framing mainly adds a self-flattering shift that understates harmful behavior.Capability: Saying it does not know
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewWhen a model must anticipate, without execution feedback, what a change to its own runtime harness components will do, naming which components a lifecycle change can affect stays near ceiling as the number of interacting effects grows from 2 to 32, while predicting the resulting state, reasoning across teardown orders, and choosing minimal reconfigurations degrade sharply (e.g. GPT-5.6 Luna's reachable-condition Jaccard 91.7% to 14.1%, executed reconfiguration success 62.5% to 25.0%).Capability: Improving its own scaffolding from failures
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewWhen a multimodal LLM acts as the decision core of a simulated humanoid reacting to sudden household hazards, picking the correct reaction primitive does not mean the reaction is executable: even on correctly classified scenes, hands land a median 0.48 m from the true impact point, 89% of misses fall short, and in 49% of those decisions the body never steps forward and stands beyond arm's reach.Capability: Turning plans into physical action
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-07· pending-reviewWhen a multimodal model is asked an object-existence question over a set of images at once, it is less accurate than when the same images are queried one at a time and the answers combined, so cross-image integration adds errors beyond the per-image perception errors.Capability: Grounding answers in the image
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewWhen a multimodal model is given external text that contradicts the image, making it commit to a description of the image before the text is admitted, then reconciling the two in a second call, recovers accuracy the single-prompt setup loses — but the benefit of withholding the text from the first call is model- and text-source dependent, and on some models context-preserving prompting is still better.Capability: Telling the user what they want to hear
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewWhen a pretrained vision-language-action policy is fine-tuned sequentially on a stream of real-robot manipulation tasks, the choice of action-normalization statistics is itself a determinant of continual-learning success: statistics that are task-derived, first-task-derived, or recomputed each stage produce forgetting or poor transfer, while statistics estimated once from a small task-independent calibration set covering the embodiment's motion range and then frozen give the best acquisition, retention and task-order stability.Capability: Turning plans into physical action
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewWhen a prompt contains several legitimate but conflicting evidence passages, models internally register the conflict — linear probes on mid-to-late-layer hidden states separate conflicting from consistent inputs at high AUC — while still resolving the conflict badly at the output, so the failure is in evidence integration rather than in conflict detection.Capability: Checking claims against evidence
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewWhen a prompt imposes k deterministically verifiable constraints at once, per-constraint pass rate declines only gently with k while the probability of satisfying all k collapses multiplicatively, because constraint failures within a response are nearly independent rather than pairwise-interfering — so joint reliability breaks down beyond roughly 5-6 simultaneous constraints and cannot be rescued by choosing or ordering which constraints are combined.Capability: Following an unfamiliar procedure
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewWhen a retrieval store holds both the correct answer and a common misconception for the same question, injecting all retrieved memories raises the hallucination rate above a memory-free baseline for weaker models (53.0% vs 23.0% on TruthfulQA with gemma-4-E4B-it), while a stronger backbone (deepseek-v4-flash) instead shows RAG below its own no-memory rate, so conflict-driven amplification is concentrated in models susceptible to memory injection.Capability: Stating false facts confidently
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-20· pending-reviewWhen a reward is misspecified and a monitor with systematic blind spots is used to penalise or filter the misbehaviour it detects, reinforcement learning teaches the model to reward hack in the forms the monitor cannot see rather than to stop hacking — observed in toy environments with a 135M-parameter model and with Qwen3-8B on a coding task — in settings chosen so that this would happen, with heuristic monitors rather than LLM judges.Capability: Gaming the measure instead of doing the task
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewWhen a scenario pits a stated personal preference against a local cultural norm, models' choice of norm over preference shifts more with the scenario's country (about 8 points) than with the user's age (1) or gender (under 1), instruction tuning shifts it non-uniformly, and while models can match the majority human choice per country they do not reproduce the spread of human answers, with best correlation to human response distributions only about 0.24.Capability: Handling culture-specific norms and viewpoints
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewWhen a search agent is asked to forecast a future distribution rather than report the current state over the same cumulative literature archive, the forward-looking target changes what it retrieves — it allocates a smaller share of retrieved documents to the most recent window, which is the more predictive evidence — so carrying the agent's own state estimate forward beats its explicit forecast.Capability: Predicting future events
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewWhen a small (1.7B) LLM is fine-tuned to act as an ensemble router that assigns per-sample weights across candidate time-series forecasters, chain-of-thought supervision built by rule-templating known oracle weights against extracted temporal features matches or beats CoT generated by a frontier API model, and removing the CoT step costs far more on unseen candidate model pools than on in-distribution ones.Capability: Predicting future events
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewWhen a structured long-term memory substrate (indexes, retrieval, control logic) is held fixed and only the backbone LLM is swapped, accuracy on multi-session memory benchmarks like LongMemEval and LoCoMo moves by a few points while per-query cost varies by roughly an order of magnitude, so recall quality in these settings is set mostly by the memory and retrieval design rather than by model choice.Capability: Remembering across sessions
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-18· pending-reviewWhen a text LLM is prompted to forecast numeric price series, feeding the history as integer-rounded values on the original price scale gives lower forecast error than full-decimal raw values, standard-normalized values, or per-window instance-normalized values, because rounding cuts token fragmentation while keeping magnitudes in the distribution the model saw in pretraining.Capability: Predicting future events
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewWhen a text-pretrained LLM is prompted to forecast a numeric price series, rounding the input values to integers while keeping the original price scale gives lower forecast error than full-precision raw values, training-set standardization, or per-window instance normalization, because normalized values fall outside the numeric distributions the model saw in pretraining and per-window rescaling makes the same price appear differently across samples.Capability: Predicting future events
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-18· pending-reviewWhen a tool call is forced and returns an unusable payload, tool-augmented agents report the failure honestly almost always if the response envelope declares an error, but assert unsupported values or invent policy reasons in a large share of cases when the envelope says status:ok while withholding the value (redacted, corrupted, stale, empty, malformed, truncated).Capability: Stating false facts confidently
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewWhen a tool-using agent is trained with outcome-only RL and a surface prompt cue co-occurs with one tool in most training examples, the agent invokes that tool on tasks where it is useless, but only for tools it has already learned to use well and mainly when the cue is semantically related to the tool.Capability: Using the tools it is given
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewWhen a user profile of stated attributes and preferences is placed in context, models agree with the user far more than with no profile — across 13 open and closed models the profile, not retrieved conversation memory, is the dominant driver, and inverting the stated preference flips almost all responses to the opposite side.Capability: Telling the user what they want to hear
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewWhen a video-understanding instruction contains several conditional branches and the model must pick the branch matching what the video shows, both proprietary and open multimodal models pick correctly far more often when the correct branch is listed first, and accuracy falls as the correct branch moves later in the list.Capability: Following an unfamiliar procedure
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewWhen accurate retrieved context is supplied at every turn of a three-turn dialogue, model scores still fall as each turn adds a condition or constraint on how the retrieved material must be used, and models that score alike on plain multi-turn chat diverge on retrieval-grounded turns, so using retrieval well is a distinct competence from chatting well.Capability: Using what was retrieved, and only that
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewWhen adapting a pretrained VLA manipulation policy with only a handful of target demonstrations, retrieving prior demonstrations at the level of LLM-decomposed subtasks (then reranking by behavior-feature similarity) raises success more than retrieval by visual, optical-flow, or state-action similarity, and the margin is concentrated on long-horizon composite tasks rather than single-skill pick-and-place.Capability: Turning plans into physical action
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-15· pending-reviewWhen adding image-level preference pairs to multimodal DPO for LLaVA-1.5 7B/13B, constructing the rejected image by selecting, among many equal-size random masks, the variant with lowest CLIP similarity to the original lowers object-hallucination rates more than coarse alternatives (blockwise masking, rotation, cropping, or a fully blacked-out image), at the cost of slightly lower object coverage.Capability: Grounding answers in the image
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-18· pending-reviewWhen an 8B model verifies natural-language claims against knowledge-graph triples, scoring the evidence twice under opposing prompts (argue it supports, argue it refutes) and adjudicating between the two arguments avoids the bias toward a "Supported" verdict that a single-perspective assessment shows, and is the largest single contributor to accuracy in that pipeline.Capability: Checking claims against evidence
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewWhen an agent builds a whole Python repository from natural-language requirements alone, refining component boundaries with explicit cohesion and coupling thresholds until no split or merge is triggered produces higher functionality coverage and test pass rate than letting the LLM decide restructuring actions for a fixed budget of rounds, because unconstrained LLM-decided actions keep over-decomposing past the point of a coherent modular structure.Capability: Generating and editing working code
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewWhen an agent generates a whole Python repository from natural-language requirements with no given architecture, keeping the repository architecture as a mutable state that is re-evolved before code generation yields higher functionality coverage and test pass rate than executing a single up-front planning graph, and removing the evolution step is the largest single ablation loss.Capability: Generating and editing working code
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewWhen an agent harness is evolved from execution traces under a fixed interaction budget, verifying each candidate edit on tasks chosen for the behavior it targets and accepting it only when the gain is attributable to that behavior beats scoring every candidate on a shared fixed, random or coreset task split, and larger verification budgets do not compensate: fixed-split methods using up to 24x the rollouts fell below the unmodified harness in half of their harness-benchmark pairs.Capability: Improving its own scaffolding from failures
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewWhen an agent harness mines reusable rules and tools from its own labelled diagnosis trajectories, promoting every proposal that helps the cases it was mined from makes the harness worse over time than not evolving at all, and only admitting a proposal when it also fails to regress on a freshly resampled held-out case set keeps accuracy climbing across successive evolution rounds.Capability: Improving its own scaffolding from failures
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-19· pending-reviewWhen an agent labels its own past attempts as successes or failures to build an experience memory, the labels can be substantially wrong without much cost: the LLM judge used was 72.7% accurate against ground truth on WebArena-Shopping, and simulating judges anywhere from 70% to 90% accurate gave similar success rates, with ground-truth labels best — so for distilling lessons, a rough self-judgment is close to enough, although a perfect one still helps.Capability: Improving its own scaffolding from failures
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-07· pending-reviewWhen an agent must work over a very large tool catalogue, wrapping each tool in a natural-language interface that resolves the API schema internally — so raw schemas are retrieved on demand rather than enumerated in the model's prompt — contributes more to multi-step tool- call success than the surrounding planner, router, or verifier stages.Capability: Using the tools it is given
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewWhen an agent system evolves its own harness from failure traces, batching failures across tasks before editing buys efficiency but not better failure attribution: pooled across three agent benchmarks, the share of harness edits that merely accommodate the current model rose from 60.0% under per-failure serial editing to 75.6% under cross-task aggregation, and only fell to 45.5% when a separate analyst/critic/engineer refinement stage attributed each recurring pattern before the edit was written.Capability: Improving its own scaffolding from failures
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewWhen an automated search over agent-harness mechanisms is gated on token efficiency with only a tolerance band on capability, the resulting full stack buys large token savings at a small but consistent score loss, and scores below the best single discovered mechanism — on EdgeBench the four-mechanism stack cut token traffic 44.7-49.0% and cost about a third while scoring 42.0 vs 44.8 for the base harness on GPT-5.6 Sol, whereas adding ObservationPack alone scored 47.2.Capability: Improving its own scaffolding from failures
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewWhen an error is injected into a supplied chain of thought, models mostly carry it to the answer: a planted arithmetic error costs small models 50-60% accuracy and 18% on average, though the largest models lose only 3-7%; unit-conversion errors cost about 9% at every scale; skipped steps and sycophantic steps cost small models about 10%; extra correct steps cost almost nothing (0-6%).
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewWhen an LLM agent must reconcile user-supplied identity credentials against database records before a sensitive read or write, frontier and open models frequently skip the cross-field consistency check and act anyway, and the failure rate barely moves whether the request is simple or has several parallel sub-requests and whether the forged field is visually near-identical to the true one or completely unrelated.Capability: Checking claims against evidence
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-18· pending-reviewWhen an LLM answers a question and gives an explanation, removing from the input the concepts its explanation did not credit and re-querying on the reduced input raises measured explanation faithfulness over both standard prompting and prompts that ask for faithfulness, because the removal deletes the unmentioned influences rather than asking the model to report them.Capability: Saying what actually drove the answer
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewWhen an LLM gives an answer plus an explanation, deleting from the input those concepts the explanation does not credit and re-querying on the reduced input raises measured faithfulness over both standard prompting and prompts that ask for faithful explanations, because the re-query removes the unmentioned factors that could have influenced the original answer.Capability: Saying what actually drove the answer
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewWhen an LLM judge is given a prior evaluation score as context metadata on a revision, its rubric scores shift toward that uninformative prior — seven of eight tested models (API and open-weight, 0-5 scale, 20 fixed texts) show a negative total shift, and on a categorical compliance task the metadata blocks 48% of error corrections and flips about 10% of previously correct judgments to the anchored wrong label.
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-15· pending-reviewWhen an LLM judge is validated against expert annotators on multi-dimensional rubrics for open-ended agent outputs, a single pooled agreement figure overstates reliability: per-dimension human-human agreement varies widely (substantial on a recall-style Completeness dimension, only slight to fair on Relevancy, Structural Clarity and Scope Adherence), the pooled statistic is inflated by between-dimension variation, and the judge's residual disagreements are directional (consistently one step stricter on Completeness, one step more generous on Structural Clarity) rather than noise.Capability: Whether the measurement made the finding
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewWhen an LLM judge must pick the human-preferred response out of more than two candidates, the assumption behind confidence-threshold abstention — that higher estimated confidence means lower disagreement with humans — degrades as the candidate count grows, because probability mass is normalized across more options; restricting the decision to a conformal top-k shortlist and thresholding a margin inside it restores the monotone relation.Capability: Saying it does not know
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewWhen asked to generate a full procedural execution graph for an unseen task (recipe) in a specific creator's style, few-shot LLMs assign ingredients to steps better than a statistical two-stage pipeline but predict the number of steps worse, and swapping in a larger LLM backbone does not close that step-count gap.Capability: Planning a sequence of actions before acting
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-18· pending-reviewWhen asked to optimize code that is already at its performance ceiling, frontier LLMs (GPT, Claude, Gemini families) never say it is already optimal under a plain "optimize this code for speed" prompt — they always emit an edit with a confident, unmeasured speedup claim; adding an explicit instruction to abstain unless >90% confident of an improvement produces abstention on a minority of such cases while leaving the edit rate on genuinely sub-optimal code unchanged.Capability: Stating false facts confidently
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewWhen asked to supply a missing biomedical reference, models that retrieve a real paper often still misstate its journal, year or authors, so verifiable-reference rate overstates usable accuracy: two Claude models with near-identical verifiable rates (77.6% and 76.6%) were correct in every bibliographic field in 54.6% and 19.9% of responses.Capability: Stating false facts confidently
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewWhen automatically rewriting the prompts of a multi-agent LLM system from its failures, locating the responsible agent by injecting a hint into one agent at a time and using that agent's hint-corrected intermediate output as the supervision signal produces prompt updates that generalize better than critique derived only from the system-level output.Capability: Improving its own scaffolding from failures
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewWhen building preference data to teach 7-8B instruct models to follow multi-constraint instructions, perturbing each atomic constraint in several directions so the perturbed instruction's valid-response set contains, partly overlaps, or is disjoint from the original's — and pairing responses drawn from each sub-region — raises constraint adherence more than rejection sampling or teacher-correction pairs, and the gain holds on perturbed instruction variants the baselines barely improve on.Capability: Following an unfamiliar procedure
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewWhen chain-of-thought faithfulness is measured by how much the internal concepts (sparse-autoencoder features) active during a CoT pass overlap with those active during a direct-answer pass, that overlap barely predicts whether the shared concepts causally drive the answer: across five open models and four reasoning datasets, above-median concept alignment raised the chance of an above-median causal ablation effect by only about 8%, so representational correspondence cannot substitute for an ablation test.Capability: Saying what actually drove the answer
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewWhen claim-level uncertainty is estimated from a response-claim graph, anchoring the graph with trusted external priors (edges weighted by entailment from verified source sections) and scoring claims by weighted closeness centrality separates ungrounded claims from repeatedly-generated-but-false ones better than internal-consistency centrality alone, and yields retrieval whose answerability tracks its accuracy instead of the near-100% answer rate GraphRAG shows on the same long documents.Capability: Stating false facts confidently
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewWhen comparing a weight-shared recursive model against a depth-matched plain transformer on code generation, the ranking read from a single checkpoint can invert during training, because the larger-parameter arm passes its validation optimum and overfits while the parameter-lean recursive arm keeps improving — so a late-epoch "recursion fits better" reading is a degradation artifact, not a capability difference.Capability: Whether the measurement made the finding
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewWhen deception in narrative text is analysed by instantiating fixed argumentation schemes with an LLM (premise extraction, then critical-question evaluation), the scheme assessments are stable across ten different LLM backbones and repeated runs for clear-cut deception cases but grow inconsistent on ambiguous intelligence-style narratives.Capability: Strategic deception and detecting it
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-13· pending-reviewWhen each capability domain's annual benchmark frontier is normalised to its own scale (0 at the benchmark's entry-year frontier, 100 at a perfect score), the domains move on different curves — broad knowledge gains slow from 32.8 to 26.9 to 17.6 points a year, legal reasoning from 48.2 to 11.4 to 4.9, while advanced mathematics accelerates from 32.8 to 53.6 and multimodal reasoning jumps 59.7 then only 2.5 — so a single aggregate benchmark score conceals both the level and the trajectory shape of progress, and cannot say which capabilities are saturating.Capability: Whether the measurement made the finding
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewWhen extracting structured rules from multimodal enterprise guideline documents, wrapping the extraction in deterministic schema validation, an LLM semantic judge, and a duplicate/contradiction consistency filter over a shared rule store lowers the share of extracted rules with no matching source sentence from about 15.7% to 3.2% relative to a one-pass pipeline using the same parser and models.Capability: Stating false facts confidently
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewWhen fine-tuning 7-8B code models on an LLM-synthesized Verilog instruction corpus, training on a 20-25% subset selected for functional correctness, instruction-solution alignment, and coverage of Verilog structural clusters (AST, CFG, netlist features) yields higher Pass@1 than training on the full corpus, because the automated corpus carries noise and redundancy that filtering removes.Capability: Generating and editing working code
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewWhen fine-tuning a 14B model on knowledge-graph-derived multiple-choice QA for a single medical domain, attaching supporting triples from the same source text chunk to each target triple during supervision, and repairing residual one-hop failures before reinforcement learning, gives higher accuracy on held-out 3-, 4- and 5-hop questions than training on isolated triples or running RL from an unrepaired checkpoint.
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewWhen fine-tuning a 4B code model, training on auxiliary targets that name each program region's correctness obligations (derived from aligning an AST implementation graph with Lean proof-flow traces) raises solved counts over code-only SFT, but the auxiliary-target stage alone shifts the model off the Python-only output distribution and must be followed by a code-only consolidation stage to avoid losing broad function-level ability.Capability: Generating and editing working code
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewWhen fine-tuning a base model on instruction data, removing from the training targets any factual claim the base model cannot consistently recall raises the supported-claim rate of later long-form generations, but the gain comes largely from a more conservative response policy — more refusals and fewer supported claims per answer — not from generating more correct facts at equal coverage.Capability: Stating false facts confidently
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewWhen fine-tuning a vision-language-action policy for a deployment shift, the parameter change needed concentrates in the functional region matching the shift's modality — appearance in the vision encoder, rephrased instructions in the language backbone, novel objects in the vision encoder plus action head — so allocating LoRA rank per region by a pre-training diagnostic beats uniform-rank LoRA at matched budget, with the advantage largest at small budgets and absent when the shift's demand spreads across regions rather than concentrating.Capability: Turning plans into physical action
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-15· pending-reviewWhen fine-tuning small open models (4B-8B Qwen3) for multi-turn tool calling, supervising only the recovery steps generated after the first point where the student's rollout leaves the teacher-supported state region — and zeroing the loss on the student's own valid prefix — beats full-trajectory imitation and outcome-reward RL on in-domain and held-out tool benchmarks, and the trained student's successful trajectories get shorter across iterations rather than longer.Capability: Using the tools it is given
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewWhen grading free-form answers to algorithm-analysis and complexity-theory questions against instructor reference answers, n-gram overlap metrics carry almost no signal about answer quality, because logically equivalent proofs differ in notation, variable names and proof strategy: BLEU-4 was 0.0000 on all 179 questions and ROUGE-1 F1 stayed in a narrow 0.088-0.129 band across topics while a rubric score varied from 0.66 to 0.83.Capability: Whether the measurement made the finding
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewWhen grading LLM-generated unit tests by pass rate and line coverage alone, a share of passing test suites never directly call the function under test — they reach it through wrappers, mocks, or unrelated public APIs — so pass rate overstates test quality, and this gap widens under agentic repository-access workflows and richer file-level context.Capability: Whether the measurement made the finding
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewWhen hardware description code (Verilog, SystemVerilog, VHDL, Amaranth-Python) is generated from a functional specification that omits security obligations — as real SoC docs do, keeping countermeasures in separate config files — frontier models pass functional tests far more often than security tests, and the model with the highest functional pass rate is not the safest; naming the relevant CWE in the prompt raises the security pass rate sharply, showing the limit is missing awareness of the required obligation rather than inability to write defensive RTL.Capability: Writing secure code and dependencies
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewWhen initially identical LLM agents act in a shared persistent world where their artifacts remain and are scored by an external simulator, the shared society builds broader and more resilient technology portfolios than a matched best-of-N population of isolated agents, but isolated search still yields the single best-performing artifact.Capability: Getting more from several models than from one
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewWhen injecting new or counterfactual facts into a 3-4B instruct model by post-training, supervised fine-tuning on extracted QA pairs binds facts to their training question form — recall of the single fact stays reasonable while multi-source and inferential questions over the same facts collapse — and adding more SFT epochs does not close the gap, whereas mixed-policy RL that supplies the golden answer as an off-policy trajectory when on-policy rollouts fail recovers much of the generalization.Capability: Absorbing and flexibly using new knowledge
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewWhen injecting revised medical guideline facts into 4B-14B instruct models by LoRA SFT under a matched token budget, rendering the same update events as extended-matching items — several related vignettes sharing one candidate answer pool — transfers better to a held-out temporally anchored oncology staging benchmark than short-answer, fill-in-the-blank, or per-vignette multiple-select renderings of the identical content.Capability: Absorbing and flexibly using new knowledge
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewWhen instruction-tuned models (Qwen2.5 7B-32B) make function calls from non-English user requests, they often select the right tool and argument names but emit argument values in the wrong language, a failure that exact-match or AST tool-calling metrics record as a generic invocation error rather than a language mismatch.Capability: Using the tools it is given
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-18· pending-reviewWhen language-conditioned VLA manipulation tasks are instantiated at graded difficulty levels that add spatial ambiguity (fine-grained spatial relations, precise placement) and procedural length (multi-stage plans requiring memory of completed steps), representative VLA models degrade on both axes, so success on predefined single-stage settings does not indicate spatial or long-horizon competence.Capability: Turning plans into physical action
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewWhen large audio language models are asked to return every time span matching a natural-language query, they almost always report fewer intervals than exist: across 21 systems on queries averaging about 4 ground-truth intervals, under-reporting rates run from 79% to 100% and no system exceeds 13.2% exact count accuracy, even for the models with the best overlap scores.Capability: Understanding speech and audio
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewWhen LLM-generated peer reviews are compared with human reviews point by point under theory-guided critique frameworks, the model and human critique differ in kind rather than in quality: LLM reviews carry more explanatory depth, integrative reasoning and explicit argument structure, while human reviews more often identify higher-order weaknesses and give revision-oriented guidance, and an expert prompt does not close that gap but amplifies the LLM-side tendencies.
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewWhen LLM-generated spear phishing emails are built from cumulatively deeper workplace detail (employer, then name and title, then responsibilities, then coworker and shared project), U.S. working adults rate them as more convincing and more often say they would click, and they also report the message as phishing less often and delete it more often as detail increases.Capability: Changing what a person believes or decides
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewWhen LLMs are used as relevance judges on a specialized non-English (Brazilian Portuguese legal) retrieval collection, they assign systematically higher relevance grades than human experts (mean absolute error 0.46-0.66 on a 0-2 scale, Cohen's kappa with humans only 0.32-0.53), yet the resulting system rankings stay close to the human reference for rank-aware metrics (nDCG@10, MRR, Kendall's tau >= 0.90) and diverge for P@10 and R@10.
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewWhen medical LLMs are given a clinical practice guideline in context and asked to produce the decision pathway a case requires, final-outcome accuracy does not track adherence to the guideline-defined step sequence: models can reach the right outcome while omitting, reordering, skipping or padding steps, and medically fine-tuned models score no better on pathway consistency than the general models they were tuned from.Capability: Competence in a specialist domain
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-15· pending-reviewWhen models must apply abstract, entity-independent procedural rules drawn from a shared pool, accuracy is limited by locating the right rules rather than by applying them: giving only the gold rules yields much higher exact-match than embedding those same rules in a pool of 100 or 1,000, and accuracy falls as the pool grows.Capability: Following an unfamiliar procedure
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewWhen multi-agent pruning methods (AgentPrune, AgentDropout, AGP, DAGP, GPTSwarm) are re-run under a fixed backbone, agent registry, role-to-node assignment and runtime, their accuracy-cost gains do not hold across starting topologies and mostly do not beat random pruning at the same structural budget — averaged over 20 cells the optimized systems lost 0.77 accuracy points against the original system versus 0.44 for matched random pruning, and only 12 of 25 method-topology pairs beat their random control.Capability: Getting more from several models than from one
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewWhen multiple stochastic runs of several mid-tier LLMs each produce an unranked set of candidate security requirements from the same control templates, pooling the runs recovers far more valid items than any single run but also many hallucinations, and ranking the pooled candidates by how many runs produced them separates valid from hallucinated items well enough to beat every individual run's precision-recall and ROC area.Capability: Stating false facts confidently
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewWhen new knowledge is a set of interconnected fictional future events rather than isolated fact edits, updating 4-8B instruct models by mid-training on teacher-generated long-form articles plus preference tuning (preferring reasoned answers over outdated, hallucinated, or needlessly abstaining ones) beats graph-retrieval and in-context editing baselines on multi-hop and causal questions, while remaining worse than in-context methods on exact numeric single-hop recall unless retrieved facts are also supplied.Capability: Absorbing and flexibly using new knowledge
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewWhen NL-to-PDDL problem generation is scored by operational acceptance (parses, conforms to domain, a planner finds a plan, VAL validates it, an LLM critic accepts), the score overstates faithfulness to the described task, and the gap widens for PDDL 2.1 temporal and numeric problems where operationally accepted instances can almost never reconstruct the curated reference encoding.Capability: Whether the measurement made the finding
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewWhen one agent's context carries planted misinformation, the wrong answer persists through multi-agent debate and other agents often adopt it, yet debate still degrades less than a single misinformed agent (relevant misinformation cut single-agent accuracy by 16-25% on CWQ), especially when most agents are unexposed; consensus protocols were more stable than voting under peer pressure, and a misinformed majority can steer the group.Capability: Getting more from several models than from one
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewWhen open-source vision-language models are post-trained with RL or critique tuning to act as judges of chart-to-code outputs, they become more likely to output Accept rather than better at verification: pairwise-comparison accuracy barely improves under an order-swapped protocol, while acceptance rates rise and specificity falls, so they ratify flawed but plausible-looking charts.
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewWhen post-training a 7-8B open model for text-to-SQL with a fixed budget of synthetic examples drawn from a large external pool, selecting examples whose schema-independent SQL skeletons are farthest from those already in the seed set (greedy K-center) rather than sampling randomly from the same pool yields a small in-domain gain and a larger cross-domain one (BIRD dev 64.9 vs 64.4, Spider Test 85.3 vs 83.4, Science Benchmark 53.9 vs 49.8).Capability: Writing a reusable locator: regex, SQL, XPath
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewWhen post-training a small LLM (Qwen2.5 0.5B-3B) to produce weekly asset-return forecasts from news, rewarding rank correlation of a full listwise asset ordering gives a dense learning signal where exact-match GRPO gives almost none (all sampled actions scoring zero on ~35% of steps), and yields higher Spearman correlation and backtest Sharpe than the SFT checkpoint, standard GRPO, and prompted commercial models.Capability: Predicting future events
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewWhen post-training a speech LLM for ASR with GRPO, adding a reward term from a separately pretrained, frozen CTC acoustic judge that scores each sampled transcription against the audio cuts insertion errors on out-of-distribution meeting speech (22-28% fewer on AMI) relative to a schedule-matched text-only -WER reward, without raising word error rate and without needing the judge at inference.Capability: Stating false facts confidently
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewWhen preference pairs are built by having one strong model write all chosen responses and one weak model all rejected responses (delta learning), contrastive objectives such as DPO transfer the chosen model's sycophantic-agreement rate to the student even though no individual example contains sycophancy — the student's rate tracks the log-ratio of the two teachers' sycophancy rates, so reversing which teacher is 'chosen' drives sycophancy far below the SFT starting point.Capability: Telling the user what they want to hear
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-15· pending-reviewWhen reranking sampled Verilog candidates with both a generated-testbench execution signal and an LLM judge, letting the judge see the execution outcomes in its prompt lowers final Pass@1 relative to acquiring the two signals separately and combining them only as a weighted score at selection time, because the judge anchors on the test verdicts (approving all-pass candidates with uncovered logic errors, over-penalizing failures caused by bad testbenches) and stops contributing information the tests miss.Capability: Generating and editing working code
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewWhen research-idea forecasts are scored by whether a later paper matches them under an LLM judge rubric, higher scores partly reflect broader, less specific ideas: the backbone with the higher hit rate also produced measurably more general forecasts and had more retrieved candidate papers pass the gate, and tightening the specificity threshold does not separate the two.Capability: Whether the measurement made the finding
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewWhen retrieval for a summarization task pulls definitions by surface term match from a general encyclopedia, ambiguous terms fetch the wrong sense and the model incorporates that unsupported background into its output, lowering claim-level factuality below the no-retrieval baseline even when the prompt tells it to ignore unrelated retrieved content.Capability: Stating false facts confidently
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewWhen reward models are scored by sequence log-probability on contextual preference pairs, abstentions are systematically shorter than the unsupported answers they are preferred over, so measured grounding/abstention judgement partly tracks response length; rewriting rejected and refusal responses to match the preferred ones in length while keeping the grounding failure raises pairwise accuracy far more than adding more in-domain legal preference data (up to +25.6 points over baseline for Ministral-8B).
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewWhen RL-training a central-agent multi-agent harness with 1.5B-3B Qwen2.5 backbones on multi-hop QA, optimizing only the coordinating central policy and leaving sub-agents frozen scores higher than jointly training central and sub-agent policies (average F1 47.79 vs 45.36, accuracy 42.88 vs 40.49 at 3B).Capability: Getting more from several models than from one
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewWhen routing each image to one of several domain-specialist vision backbones trained on a shared label space, a prompted MLLM router (Gemma-4-12B) reaches nearly the accuracy of a trained neural-network router and beats majority-vote fusion, because routing only needs coarse domain identification, which a general multimodal model already does zero-shot.Capability: Getting more from several models than from one
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-15· pending-reviewWhen self-reflection is used as a training signal for long-horizon agent and math tasks, resetting to the original prompt with a compact reflection patch prepended beats accumulating reflections inside the ongoing trajectory, and overly long reflections (more than ~10 points) hurt compared with 2-5 bullet points.
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewWhen the preference needed to answer a query lies in a different concept space from the cues in a user's dialogue history (e.g. inferring an emotion-regulation strategy from personality cues), retrieval-based and profile-summarization personalization methods barely beat a no-history baseline, and dumping the full history into the prompt does worse than no history at all.Capability: Remembering across sessions
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewWhen tool-augmented agents emit calls over an unconstrained raw-JSON surface rather than a schema-enforcing function-calling API, every tested hosted model fabricates tool names or undeclared arguments at rates of roughly 13-57%, and parameter count does not reduce this — an 8B and a 675B model hallucinate at comparable raw-JSON rates.Capability: Stating false facts confidently
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewWhen training 4B-8B Qwen3 agents with group-based RL (GRPO, DAPO, GSPO) on sparse-reward long-horizon tasks, injecting hints into the environment's feedback — procedural action guidance in early episode steps and early training, switching to enriched state observations in later steps and later training — raises success on the unmodified environments by a few points on average, while lowering performance on tasks that require questioning whether the request is answerable at all.
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewWhen training a sequence-to-sequence model to generate regular expressions from natural language, rewarding semantic equivalence to the reference (checked by DFA equivalence and test cases) yields more semantically correct regexes than maximum-likelihood training on the reference string, because one intent has many syntactically different equivalent regexes.Capability: Writing a reusable locator: regex, SQL, XPath
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewWhen training an 8B multimodal model with RL to find scientific errors in whole papers without being told where to look, mixing in matched easier variants of the same instances — where the error is asserted and the evidence supplied — yields better performance on the evidence-absent task itself than training only on that task at matched exposure.Capability: Spotting an error in a given artifact
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewWhen training an LLM with policy-gradient RL on tasks that decompose into many dependent sub-problems, rewarding each trajectory segment by how many reference reasoning points it newly reaches beats a sparse final-answer reward by a margin that widens with task horizon, and still learns on problem sets where the base policy never samples a success and sparse reward gives no gradient at all.
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-09-07· pending-reviewWhen training small instruction-tuned models with reinforcement learning plus factual supervision, routing each atomic fact's verification score only to the tokens that produced it, and down-weighting verifier judgements that do not change when their key evidence is removed, improves factuality benchmark scores over trajectory-level or reasoning- step-level factual rewards.Capability: Checking claims against evidence
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewWhen two audio clips are concatenated at the waveform level into one input, current large audio language models keep transcribing both segments nearly as well as single clips while their accuracy on content, order and location questions about the same two segments drops far below their single-clip attribute-recognition accuracy.Capability: Understanding speech and audio
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewWhen Video-LLMs are asked to verify sentence by sentence whether a dense video caption is supported by the video, they detect hallucinated sentences less often in captions they generated themselves than in captions from other models, and detection of temporal errors (action, order, causality, frequency, state change) improves much less with model scale than detection of static-frame errors.Capability: Reasoning about time in video
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-24· pending-reviewWhen vision-language models are given an explicit abstention option plus a stated payoff for correct, wrong and declined answers, quadrupling the penalty for a wrong answer barely changes how often they answer, so their abstention rate is largely insensitive to the stated cost of error across text, synthetic video and natural video recall tasks.Capability: Saying it does not know
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-09· pending-reviewWhen VLM agents must verify claims against interactive visualizations where no claim is answerable from the initial viewport, giving them an interaction budget of ten actions does not reliably beat answering immediately from the first screenshot — some models score slightly lower with interaction — because partial, unplanned exploration leaves the model with enough evidence to abandon its prior but not enough to replace it.Capability: Checking claims against evidence
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewWhen VLMs must operate unfamiliar household devices with no manual, through multi-turn interaction over rendered multi-view images with click/rotate actions at pixel coordinates, the limiting failure is fine-grained localization of the interactive component within the correct view rather than navigating to that view, and almost no task is completed in the optimal number of steps.Capability: Turning plans into physical action
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-22· pending-reviewWith the agent loop, skills and prompts held fixed and only the model swapped, models split into two clusters on operating a multi-step asynchronous ML workflow: Claude Sonnet 3.5-4.0, Gemini 2.5 and GPT-5 completed tasks at 93-100%, while GPT-4 scored 13% and others could not run the loop at all, hallucinating workflow ids or failing to wait for asynchronous jobs; on open-ended exploration the effect of a stressful, competitive prompt reversed by family, with Sonnet 4.0 producing the largest improvement of any configuration under stress and GPT-5 turning conservative and giving back most of its basic-prompt gains.Capability: Using the tools it is given
- observationsingle paper↑2↓0Reviewed by AIchecked 2026-09-11· pending-reviewWith weights frozen and no stronger model in the loop, an agent can raise its own pass rate by rewriting the scaffolding it runs inside — clustering its failures, proposing minimal edits to prompts, tools and policy, and keeping only edits that improve one split without degrading another: all 9 of 9 model-by-benchmark pairs improved, by up to 40.6 percentage points (GLM-5 on AppWorld, 44.4% to 85.0%), with weak models gaining most; but the promotion gate reads the held-out split, so the reported held-out gain is not clean generalisation, and there is no matched-compute baseline.Capability: Fixing its own mistakes
- mechanismsingle paper↑1↓0Reviewed by AIchecked 2026-10-17· pending-reviewWithin one model family, the relative distribution of reasoning failure modes stays similar from 1.5B to 72B, so critiques mined offline from small models' wrong answers, retrieved by failure mode and inserted as in-context exemplars, raise a large model's math accuracy above random, fixed, or embedding-similarity exemplar selection at a single generation.
- observationsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-04Without added retrieval infrastructure, language models underperform human experts at forecasting real-world events from a benchmark of actual forecasting-tournament questions, though accuracy improves with model scale and access to relevant news context.Capability: Predicting future events
- mechanismsingle paper↑1↓0Reviewed by AI and Russ Millerchecked 2026-09-03Without an external, ground-truth signal — a failing test, a compiler error, a verifier's output — a model's own critique of its reasoning is not a reliable improvement signal, and asking it to review and revise a correct answer often turns it into a wrong one.Capability: Fixing its own mistakes
- supportsLarge Language Models Cannot Self-Correct Reasoning YetThis source: Large Language Models Cannot Self-Correct Reasoning YetHow much the field cites it — very heavily cited in the last 12 months503 in the last 12 months · 1164 totalpublished 2023checked 2026-09-04
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewWithout training or a prebuilt index, searching a module call graph built on the fly for breadth at file level and then expanding it into a function call graph for depth reached about 43-44% function-level Top-1 localization on SWE-bench Lite and Verified with open 32B and 72B models, beating prior LLM localizers, and the better locations raised an issue-solving pipeline's resolve rate by roughly 3 to 30 points depending on model.Capability: Finding what matters in a large codebase
- observationmechanism reasoning↑1↓0Reviewed by AIchecked 2026-09-08· pending-reviewWorked examples in a tool description, long the standard way to teach tool use, can constrain a capable model to the exploration space the examples describe; Anthropic now advises designing expressive parameters instead — a status enumeration that implies its own usage rather than a demonstration of it.Capability: Using the tools it is given
- observationsingle paper↑1↓0Reviewed by AIchecked 2026-09-19· pending-reviewZero-shot, at function level, small locally run models (Llama 3.2, Mistral) identify 43-45% of 349 real Python bugs from BugsInPy, well above a sub-5% keyword baseline, but with accuracy ranging from 0% to 100% by project and lowest (about 22%) on bugs that span several components, so detection is real but shallow and codebase-dependent.Capability: Spotting an error in a given artifact