multiple-choice-questions-derived-industry-cybersecurity-certifications-frontier-models-reach CLM-0312observationsingle paperReviewed by AI
On multiple-choice questions derived from industry cybersecurity certifications, frontier models reach human-expert level on general networking and IT security, but accuracy falls on vendor-specific material and formal standards, staying below 40% on the hardest ISA/IEC 62443 and Fortinet NSE questions; general security benchmarks such as MMLU Computer Security are saturated near 100% and no longer separate models.
Capability: Competence in a specialist domain
Observed on
CyberCertBench MCQA suites from IT, OT and vendor certifications, 2026; frontier and open models; difficulty tiers from the exams..
Sources
- Multiple-choice only, which favours recall over practice; the split between general and vendor/standard knowledge is clear and the saturation observation about older benchmarks is useful in itself.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- On 2,154 expert-curated multiple-choice questions across eight architectural categories, 11 current models all score above 90% zero-shot (best about 94%), with quality-attribute and pattern questions near ceiling, but accuracy varies by category and drops for smaller models on the harder tiers (to about 85-87%), so declarative architectural knowledge is largely present while the benchmark cannot show whether it is applied in design.Making design decisions that hold up as a system grows · reviewed by AI
- For a domain-specific language essentially absent from pretraining corpora (FORM symbolic algebra), QLoRA fine-tuning an 8B model on a few thousand examples validated by executing them through the language's own interpreter beats frontier open-weight models of up to 756B parameters given syntax documentation in context, on execution rate and strict output match for the larger benchmarks, while general reasoning scores drop under 3 points.Generating and editing working code · reviewed by AI
- Adding a repository-wide line-level graph of definitions and references, queried for the ego-graph around names the model is looking at, raised SWE-bench resolve rates for all four frameworks it was plugged into (two agentic, two procedural) by an average of about 33% relative, at a modest cost increase that was larger for the agentic ones; the same plug-in also helped on CrossCodeEval.Finding what matters in a large codebase · reviewed by AI
- On repository-grounded LLM-infrastructure engineering tasks, frontier agents score far below ceiling (best 36.53% overall), and every one of eight models scores lower on long-horizon multi-file implementation than on single-file kernel completion, even though the full repository and test cases are visible.Completing tasks that take many steps or many hours · reviewed by AI
- On 430 tasks distilled from real professional-forum threads across seven domains, the best model (GPT-5) scores about 54%, under the 60% passing line the authors set, with computer science and applied technology lowest at roughly 40-45%, so long-tail practitioner knowledge remains a gap even where standard benchmarks are saturated.Competence in a specialist domain · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Models scoring above 70% on the IEC 62443 and vendor tiers without retrieval, or evidence those tiers are mis-keyed.