Open questions
These are the places where we still don’t have a good answer.
Some are weaknesses that have been documented, but no one has shown that a particular technique actually works. Others are techniques people use in practice, but we couldn’t find evidence that they do what people claim they do. Those are the gaps most worth investigating.
Not all gaps mean the same thing. If we searched the literature and found nothing, that tells us something about the state of the research. If we simply haven’t investigated a technique yet, that only tells us we haven’t looked at it. We keep those two cases separate.
30 open
Documented, but nothing measured to fix it 12
These are weaknesses we know exist, but we couldn’t find a technique that has actually been tested and shown to work.
That makes these especially interesting: evidence that a technique works would add something genuinely new, rather than just confirming an existing result.
No technique 5see in Capabilities →
No technique has been catalogued for these yet, so there is nothing to measure.
- Grounding answers in the image9 claims documenting it
Saying only what the image supports, rather than describing plausible objects that are not there.
No technique here at all.
- Generating and editing working code7 claims documenting it
Whether a model can write correct code from a specification, and go beyond isolated functions to real repository-scale editing and debugging.
No technique here at all.
- Strategic deception and detecting it3 claims documenting it
Whether a model can deliberately induce false beliefs in others when a task calls for it, and detect deception from others.
No technique here at all.
- Predicting future events2 claims documenting it
Whether a model can make well-calibrated predictions about events that haven't happened yet, not just recall or verify existing facts.
No technique here at all.
- Reasoning about time in video1 claim documenting it
Tracking motion, ordering, causality, and change across video frames rather than treating video as isolated stills.
No technique here at all.
Untested techniques 7see in Capabilities →
These capabilities have techniques, but nothing here measures whether any of them works. If a study already exists, finding it closes the gap — no new experiment needed.
- Telling the user what they want to hear11 claims documenting it
Models shift answers toward the user's stated view, cave to pushback, and mirror mistaken premises.
Catalogued but unmeasured: Fine-tune on opinion-irrelevant examples
- Whether the measurement made the finding10 claims documenting it
Whether a reported result describes the model or the way it was scored — the metric, the prompt format, the contamination, the baseline.
Catalogued but unmeasured: Deterministic checks before LLM judges
- Remembering across sessions8 claims documenting it
Assistants fail to recall and apply facts, preferences, and decisions from earlier sessions.
Catalogued but unmeasured: Checkpoint task state to a file, Maintain a running summary, Tiered memory managed by the model
- Keeping its own context clean4 claims documenting it
Agents fill their working context with re-read files, restated notes, and excess documentation until quality degrades.
Catalogued but unmeasured: Keep a guide file of past failures, Tiered memory managed by the model
- Biased when judging other outputs3 claims documenting it
A model used as a judge favors whichever answer is shown first, longer answers, and its own outputs.
Catalogued but unmeasured: Judge both orders and reconcile, Separate the verifier from the producer
- Not generalizing "A is B" to "B is A"3 claims documenting it
A model that learned a fact in one direction often cannot recall it when asked in the reverse direction.
Catalogued but unmeasured: Train on reversed text
- Tracking state through a long task3 claims documenting it
Models lose track of the current state of entities, files, or an environment after many steps.
Catalogued but unmeasured: Checkpoint task state to a file, Re-read before editing
Techniques nothing measures 18
Searched, still open 1see in Techniques →
We went looking for a study that isolates the technique, and did not find one. Each entry records what was searched for. Where there was a nearest paper, it names that paper and why it does not actually support the technique.
That makes each of these a research brief: the question, the dead end already walked, and what a clean experiment would have to separate.
- Re-read before editingprocess
Force the agent to read the current version of a file immediately before changing it.
Would bear on Tracking state through a long task
Looked for a controlled study isolating the harness rule itself: rejecting an edit unless the file was read after its last modification, with exact-match replacement so a changed file fails loudly. Found nothing that separates this from the surrounding agent scaffold. The adjacent literature is about context efficiency and repository exploration, which is a different intervention — it changes what the agent reads, not whether a stale read can be acted on. The measurable question is narrow and cheap to run: rate of silently wrong edits with and without the read-recency precondition, on the same task set.
Nearest missFastContext: Training Efficient Repository Explorer for Coding AgentsDeploys a separate exploration subagent to cut token spend and keep the solver's context clean, and reports end-to-end resolution gains. It never isolates stale-read edits, and its intervention is about what gets read rather than about refusing to act on an outdated read. Citing it here would look like support without being any.
searched 2026-09-04
Not yet searched 14see in Techniques →
No efficacy claim is filed for these, and no search is recorded either. That only tells us we have not looked yet. It says nothing about whether the research exists.
The first useful move is to look, and to record the result either way.
An explicit allow/ask/deny policy plus rate, cost and time ceilings, enforced by the harness rather than requested of the model.
Would bear on Prioritizing safety under conflicting goals, Using the tools it is given, Following instructions hidden in data
Split a long answer into atomic claims and verify each against search or a corpus.
Would bear on Stating false facts confidently, Checking claims against evidence
Write task id, status, completed steps and artifact paths to a small file after every meaningful step, and read it at session start.
Would bear on Tracking state through a long task, Remembering across sessions
Wire linters, test suites and schema validators into the agent's loop before adding any model-based judgment.
Would bear on Fixing its own mistakes, Whether the measurement made the finding
- Encode digit position in the modelarchitecture
Add positional embeddings that tell the model which place each digit occupies.
Would bear on Digit-level arithmetic
Run the judge with candidates in both orders and count a preference only when it holds in both.
Would bear on Biased when judging other outputs
A versioned instruction file the agent reads before acting, where every line is a past failure converted into a permanent rule.
Would bear on Following an unfamiliar procedure, Keeping its own context clean
Cluster failed traces into signatures, have the same fixed model propose a few minimal harness edits, and promote only edits that improve one split without degrading another.
Would bear on Fixing its own mistakes
Put the exact documentation for candidate tools into the prompt instead of relying on memory.
Would bear on Using the tools it is given
Run a static insecure-pattern scanner on every generated change before it is accepted.
Would bear on Writing secure code and dependencies
The agent that produced an artifact does not judge it; a deterministic sensor or a separate verifier agent does, and reports rather than rewrites.
Would bear on Fixing its own mistakes, Biased when judging other outputs
- Tiered memory managed by the modelarchitecture
Keep a small working context and let the model page facts in and out of external storage.
Would bear on Remembering across sessions, Keeping its own context clean
Use a written set of principles to generate critiques and preferences, then train on them.
Would bear on Prioritizing safety under conflicting goals
- Train on reversed texttraining
Include reversed word or entity sequences in pretraining so facts are learned in both directions.
Would bear on Not generalizing "A is B" to "B is A"
Argued, not measured 3see in Techniques →
These are backed only by reasoning about how the technique works. No source measures the effect. The mechanism may well be right; nobody has put a number on it.
That is a weaker opening than silence, but a real one. It is also the easiest kind to mistake for settled, which is why we keep it separate.
Fine-tune on synthetic prompts where a stated user opinion must not change the answer.
Would bear on Telling the user what they want to hear
- Maintain a running summaryprompting
Periodically fold the conversation into a summary that replaces older turns.
Would bear on Remembering across sessions
- Offload arithmetic to codetooling
Have the model write a short program for the numeric part and run it, instead of computing in text.
Would bear on Digit-level arithmetic
Every gap on this page is here because nobody has closed it yet. That is a statement about effort so far, not about what is possible. Given time, iteration and work, there is a way — and where there isn’t, the job is to prove that rather than assume it.