data-science-python-tasks-where-several-correct-solutions-differ-orders CLM-0036For data-science Python tasks where several correct solutions differ by orders of magnitude in runtime, a model's correctness rate does not predict the efficiency of the code it writes: rankings invert between the two, code-specialized open models are less efficient than general open models of similar size class, and the fastest model rotates by library (NumPy vs Pandas vs PyTorch vs Polars).
Capability: Generating and editing working code
Observed on
Python data-science code using Pandas/NumPy/PyTorch/SciPy/Polars, tasks drawn from efficiency-oriented Stack Overflow posts, one greedy sample per task, efficiency scored relative .
Sources
- 1,000 human-validated instances, 16 models in 3 tiers, log-normalized relative runtime score restricted to solved tasks; rank inversions confirmed on pairwise win rates and on the 131 tasks all six representative models solved, which controls for correctness-conditioned task mix. Human reference beat the best model on efficiency. Single benchmark, Python only, Stack Overflow provenance so contamination is not ruled out.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- On function-level algorithmic tasks translated to C#, a model's functional test pass rate carries almost no information about the structural quality or runtime cost of the code it writes: across 340 solutions from four frontier models, correctness correlated near zero with a composite quality score (r = 0.075) and model rankings by correctness reversed against rankings by quality, while the structurally simplest generator was also the most memory-hungry.Whether the measurement made the finding · reviewed by AI
- For a domain-specific language essentially absent from pretraining corpora (FORM symbolic algebra), QLoRA fine-tuning an 8B model on a few thousand examples validated by executing them through the language's own interpreter beats frontier open-weight models of up to 756B parameters given syntax documentation in context, on execution rate and strict output match for the larger benchmarks, while general reasoning scores drop under 3 points.Generating and editing working code · reviewed by AI
- Adding a repository-wide line-level graph of definitions and references, queried for the ego-graph around names the model is looking at, raised SWE-bench resolve rates for all four frameworks it was plugged into (two agentic, two procedural) by an average of about 33% relative, at a modest cost increase that was larger for the agentic ones; the same plug-in also helped on CrossCodeEval.Finding what matters in a large codebase · reviewed by AI
- On multi-turn tool-calling benchmarks that score whether the final environment state matches a gold trajectory, aggregate accuracy hides wrong action-class choices: a model can call a withheld or under-specified tool with fabricated arguments, be handed the missing piece on a later turn, and still be graded PASS — so heavily tool-trained model families rank high while almost never asking for a missing parameter or refusing an unavailable function.Whether the measurement made the finding · reviewed by AI
- Zero-shot, at function level, small locally run models (Llama 3.2, Mistral) identify 43-45% of 349 real Python bugs from BugsInPy, well above a sub-5% keyword baseline, but with accuracy ranging from 0% to 100% by project and lowest (about 22%) on bugs that span several components, so detection is real but shallow and codebase-dependent.Spotting an error in a given artifact · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: On a comparable efficiency benchmark, model ranking by pass rate matches ranking by runtime efficiency of correct solutions, with no per-library rotation of the fastest model. Proposed technique, not catalogued: Efficiency-taxonomy-guided reprompting and library-conditioned model routing.