repository-grounded-llm-infrastructure-engineering-tasks-frontier-agents-score-far-below CLM-0474On repository-grounded LLM-infrastructure engineering tasks, frontier agents score far below ceiling (best 36.53% overall), and every one of eight models scores lower on long-horizon multi-file implementation than on single-file kernel completion, even though the full repository and test cases are visible.
Capability: Completing tasks that take many steps or many hours
Observed on
85 tasks built from systems papers and public LLM-infra repositories, three formats (single-file kernel completion, multi-file feature implementation, end-to-end optimization), one.
Sources
- Measured across 8 frontier models on 85 tasks (55 KFC, 20 LHI, 10 E2EO) with correctness-gated rewards and anti-cheating proctoring. LHI < KFC holds for all 8 models in the reported table. E2EO scores are higher than LHI, so the difficulty ordering is not monotone in openness; scores are not comparable across formats because metrics differ (binary implementation vs log-normalized speedup). Single benchmark, no human baseline.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- When a small (1.7B) LLM is fine-tuned to act as an ensemble router that assigns per-sample weights across candidate time-series forecasters, chain-of-thought supervision built by rule-templating known oracle weights against extracted temporal features matches or beats CoT generated by a frontier API model, and removing the CoT step costs far more on unseen candidate model pools than on in-distribution ones.Predicting future events · reviewed by AI
- For small open-weight backbones (Qwen3-4B) driving long-horizon remote-sensing tool chains, inserting a step-level verifier that checks each intermediate tool output against schema, sub-goal and domain constraints and triggers replanning is the single largest architectural gain, while hierarchical decomposition without domain-aligned tool grouping adds little and can lower final accuracy relative to a flat agent with the same verifier.Completing tasks that take many steps or many hours · reviewed by AI
- For multi-hop QA over 28k-896k token haystacks with frozen Qwen2.5 backbones (7B-72B), summarizing the document in a parallel hierarchical tree — overlapping windows scanned ~5x, raw segments scored 'critical' passed straight to the final answer, and one summarization channel per decomposed sub-question — scores higher than sequential memory-agent and BM25 retrieval baselines without training, and beats an RL-trained memory agent on the out-of-distribution dataset while losing to it in-distribution.Losing information in long inputs · reviewed by AI
- Giving an agent a parsed code graph (files, classes, functions with import, call and inheritance edges) and tools to search and traverse it lets a fine-tuned 32B open model localize issues about as well as proprietary agents at roughly an eighth of the cost, reaching about 93% file-level accuracy, and better localization carried through to about 12 points more resolved issues at Pass@10 when plugged into a fixed repair pipeline.Finding what matters in a large codebase · reviewed by AI
- Adding a repository-wide line-level graph of definitions and references, queried for the ego-graph around names the model is looking at, raised SWE-bench resolve rates for all four frameworks it was plugged into (two agentic, two procedural) by an average of about 33% relative, at a modest cost increase that was larger for the agentic ones; the same plug-in also helped on CrossCodeEval.Finding what matters in a large codebase · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A model scoring higher on the multi-file long-horizon format than on single-file kernel completion under the same harness, or a model exceeding roughly 70% overall on this benchmark.