software-architecture CAP-0045 · activeMaking design decisions that hold up as a system grows
Whether a model can choose structure — module boundaries, interfaces, data flow — that stays maintainable, as opposed to code that passes the tests today; the failure is locally correct code that raises the cost of every later change.
Tags: coding-agent
Filed from Russ's capability map (2026-09-12). Almost every coding benchmark measures whether tests pass, which says nothing about whether the change was made in the right place or made the next change harder. Lehman's law, filed as an adage, predicts the cost. Evidence here is scarce because the property is slow to measure; that is the research brief.
What counts as this capability
Scope boundary used when deciding whether a paper is really about this capability, rather than merely mentioning it.
In scope when the paper measures maintainability, structural quality or the cost of later changes in model-written code, or compares model design choices to expert ones. NOT in scope for test-pass rates alone or for style and formatting.
Claims
- observationsingle paper↑1↓0On 2,154 expert-curated multiple-choice questions across eight architectural categories, 11 current models all score above 90% zero-shot (best about 94%), with quality-attribute and pattern questions near ceiling, but accuracy varies by category and drops for smaller models on the harder tiers (to about 85-87%), so declarative architectural knowledge is largely present while the benchmark cannot show whether it is applied in design.
- observationsingle paper↑1↓0Among 6,540 code comments referencing ChatGPT, Copilot, Claude or Gemini in public Python and JavaScript repositories (2022-2025), 81 also admit technical debt, most often postponed testing of the generated code, incomplete adaptation to the codebase, and limited understanding of what the code does, with a lower share of design-related debt than in ordinary self-admitted debt.
- observationsingle paper↑1↓0Auditing AI-generated code with static smell detection at code, structure and architecture level, from single-file tasks up to agent-built systems, more capable models produced more bloated and more coupled code, code volume was a near-perfect predictor of structural decay, and neither functional correctness nor more detailed prompting reduced it, so passing tests does not indicate maintainable design.
Techniques
None yet.
Sources
Every paper, post or observation behind this capability. Ones cited by a claim show the claim and its stance; ones filed directly are about the capability but have no claim drafted from them yet. Open one to add support or contest it.
- ↑ supports a claim here
- AI-Generated Smells: An Analysis of Code and Architecture in LLM and Agent-Driven Development2026-05-04↑ supports a claim here
- ↑ supports a claim here
Capabilities are a way of carving up the subject, and carvings are arguable. Say so if this one is wrong — especially a proposed one, which a pipeline added because several papers used the same framing, not because anyone decided it was right.