arxiv-2511-06346 SRC-0332 · paper

LPFQA: A Long-Tail Professional Forum-based Benchmark for LLM Evaluation

Created: 2025-11-09 · Ingested: 2026-09-19

Large Language Models (LLMs) perform well on standard reasoning and question-answering benchmarks, yet such evaluations often fail to capture their ability to handle long-tail, expertise-intensive knowledge in real-world professional scenarios. We introduce LPFQA, a long-tail knowledge benchmark derived from authentic professional forum discussions, covering 7 academic and industrial domains with 430 curated tasks grounded in practical expertise. LPFQA evaluates specialized reasoning, domain-specific terminology understanding, and contextual interpretation, and adopts a hierarchical difficulty

In brief

Frontier models answer only about half of real questions pulled from specialist professional forums, and giving them a code interpreter or web search makes them worse.

LPFQA contains 430 items (249 multiple-choice, 181 short-answer) scraped as screenshots from over 50 vetted forums, covering 7 primary and 21 secondary domains, built through VLM extraction, LLM question rewriting, expert verification, and difficulty stratification. 11 LLMs were scored by an LLM judge on binary answer equivalence.

All model averages fell in a 45%–55% band; GPT-5-High led at 54.42%, with the all-model average 48.96%. Computer Science was the weakest domain (41.26% average). Error attribution on 50 sampled failures per domain put knowledge gaps plus question misinterpretation at over 73% of errors and reasoning errors at 16.3%. Adding a Jupyter interpreter dropped average accuracy from 50.00 to 35.62 (28.76% relative), and search dropped it to 34.87 (30.26%). ROUGE-L overlap was lower for LPFQA (24.02–33.73) than MMLU-devset (38.05–46.17), arguing against contamination.

The tool ablation has a matched tool-free baseline on the same 5 models, which isolates the tool effect; scoring rests on 1 unaudited LLM judge, and difficulty tiers were calibrated using model accuracy, so the benchmark is partly defined by what models already fail.

Use this if you assume retrieval fixes long-tail knowledge gaps.

Written from the abstract by claude-opus-5 on 2026-10-17, with every figure checked against it. Not a substitute for the paper.

Referenced by

Claims in this catalog that draw on this source, and whether as support or counterpoint.

supportsOn 430 tasks distilled from real professional-forum threads across seven domains, the best model (GPT-5) scores about 54%, under the 60% passing line the authors set, with computer science and applied technology lowest at roughly 40-45%, so long-tail practitioner knowledge remains a gap even where standard benchmarks are saturated.