SAKE: Software Architectural Knowledge Evaluation Benchmark for Large Language Models

Current frontier models already sit near the ceiling on textbook software-architecture knowledge, but their competence is uneven by category, and adding more context helps recall while hurting trade-off reasoning.

SAKE is a benchmark of 2154 expert-authored 4-option multiple-choice questions spread over 8 categories (quality attributes, architectural solutions, styles, scalable practices, quantum computing, and the 3 Gang-of-Four pattern families) and 4 context-length bins by word count. 11 proprietary and open-weight models (Claude Haiku/Sonnet/Opus 4.5-4.6, Deepseek v3.2, Gemini 3.1 Flash Lite, GPT 5.4 and Nano, Grok 4.1 Fast, Grok 4.2, Mistral Small 4, Qwen3 235B A22B) were scored zero-shot and five-shot at temperature 0 on the same 2114 test items.

Accuracy clusters tightly from 89.31% to 94.23%. Quality Attributes averages 98.2% zero-shot and Architectural Solutions is lowest at 89.5%. Five-shot helps some models and hurts others: Deepseek v3.2 falls from 93.61% to 89.4%. Within Architectural Solutions, accuracy drops as prompts lengthen, Grok 4.1 Fast going from 100% on the shortest bin to 91.7% or less above 135 words.

Effects are measured, with matched items across settings, but the context-length claim is correlational across question bins rather than a controlled manipulation, and the distractor-density explanation is argued, not tested. All 3 reviewers were the authors; no external validation. Items are closed-form recall, not open-ended design.

Treat per-category scores, not aggregates, when deciding where to trust an LLM architect.