CyberCertBench: Evaluating LLMs in Cybersecurity Certification Knowledge

Passing a general IT security benchmark tells you little about whether a model knows vendor-specific procedures or formal industrial standards; accuracy falls sharply as knowledge specificity increases.

CyberCertBench is a suite of multiple-choice benchmarks scraped from certification question banks (Cisco CCNx, Fortinet NSE, Fortinet ICS/SCADA, ISA/IEC 62443), manually reviewed for label errors, alongside MMLU Computer Security and CyberMetric80 as baselines. 11 representative models span proprietary frontier systems and open-weight families, evaluated 5-shot at temperature zero with exact-match accuracy against the 80% certification passing grade.

Frontier models approach perfect scores on CyberMetric80 and MMLU Computer Security and pass on Cisco CCNx. On ISA/IEC 62443 only Claude 3.5 Sonnet passes, at 81%, and Claude 4.5 Sonnet scores lower. On Fortinet NSE no model passes, top score 62.4%. On the hardest ISA/IEC 62443 and Fortinet NSE items even the largest models stay below 40%, and the curve versus parameter count is nearly flat.

The gradient is measured, not argued, across 6 benchmarks and many models. Questions come from community braindump sites with acknowledged sampling bias, datasets are small (ISA/IEC 62443 has 53 questions, Fortinet ICS/SCADA 43), and contamination is not tested. The Proposer-Verifier difficulty-description method is weakly evidenced: synthesized descriptions reach 39.0% average F1 versus 36.7% baseline and 33.3% random, with large standard deviations and sub-random results on CyberMetric80, and GPT-5 serves as proposer, verifier and synthesizer.

Treat strong general cyber benchmark scores as no evidence of reliability in OT or proprietary-product settings.