function-level-algorithmic-tasks-translated-c-model-s-functional-test CLM-0059On function-level algorithmic tasks translated to C#, a model's functional test pass rate carries almost no information about the structural quality or runtime cost of the code it writes: across 340 solutions from four frontier models, correctness correlated near zero with a composite quality score (r = 0.075) and model rankings by correctness reversed against rankings by quality, while the structurally simplest generator was also the most memory-hungry.
Capability: Whether the measurement made the finding
Observed on
85 HumanEval-derived single-function C# tasks, one greedy sample per model (GPT-4, Gemini 1.5 Pro, Claude 3.5 Sonnet, Grok 3); quality score is an equal-weight composite whose own .
Model cards: GPT-4
Sources
- Measured on 85 tasks x 4 models = 340 solutions, one deterministic sample each; static metrics from Roslyn AST, runtime from BenchmarkDotNet. Authors admit the r = 0.075 is partly self-referential since normalized correctness is one of the five composite components, and no robustness check without it was run. Claude's runtime and memory means are driven by two anomalous tasks.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For data-science Python tasks where several correct solutions differ by orders of magnitude in runtime, a model's correctness rate does not predict the efficiency of the code it writes: rankings invert between the two, code-specialized open models are less efficient than general open models of similar size class, and the fastest model rotates by library (NumPy vs Pandas vs PyTorch vs Polars).Generating and editing working code · reviewed by AI
- For a domain-specific language essentially absent from pretraining corpora (FORM symbolic algebra), QLoRA fine-tuning an 8B model on a few thousand examples validated by executing them through the language's own interpreter beats frontier open-weight models of up to 756B parameters given syntax documentation in context, on execution rate and strict output match for the larger benchmarks, while general reasoning scores drop under 3 points.Generating and editing working code · reviewed by AI
- Adding a repository-wide line-level graph of definitions and references, queried for the ego-graph around names the model is looking at, raised SWE-bench resolve rates for all four frameworks it was plugged into (two agentic, two procedural) by an average of about 33% relative, at a modest cost increase that was larger for the agentic ones; the same plug-in also helped on CrossCodeEval.Finding what matters in a large codebase · reviewed by AI
- On 2,154 expert-curated multiple-choice questions across eight architectural categories, 11 current models all score above 90% zero-shot (best about 94%), with quality-attribute and pattern questions near ceiling, but accuracy varies by category and drops for smaller models on the harder tiers (to about 85-87%), so declarative architectural knowledge is largely present while the benchmark cannot show whether it is applied in design.Making design decisions that hold up as a system grows · reviewed by AI
- On research-level theorem-proving tasks drawn from recent FOCS/STOC/SODA papers, wrapping a frontier model in a staged harness that explores strategies, gates when to decompose, solves section-level subproblems along a dependency graph, and routes verifier findings back to the affected section raises graded proof accuracy far above prompting the same model directly (30.3% to 54.0% for Gemini 3.1 Pro on TCS-Bench), and two such runs with different backbones make complementary errors, so selecting between them by sampled critiques reaches 71.0% against an oracle ceiling of 77.3%.Completing tasks that take many steps or many hours · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A larger multi-language study where per-model pass rate and independently measured maintainability or runtime efficiency correlate strongly and rankings agree, with correctness excluded from the quality composite. Drafted stance toward data-science-python-tasks-where-several-correct-solutions-differ-orders: supports -- Both find that correctness rates do not predict the efficiency or quality of the generated code, and that rankings invert between the two axes, here in C# rather than Python.