arxiv-2511-10459 SRC-0334 · paper

LocalBench: Benchmarking LLMs on County-Level Local Knowledge and Reasoning

Created: 2025-11-13 · Ingested: 2026-09-19

https://arxiv.org/abs/2511.10459(opens in a new tab)

Large language models (LLMs) have been widely evaluated on macro-scale geographic tasks, such as global factual recall, event summarization, and regional reasoning. Yet, their ability to handle hyper-local knowledge remains poorly understood. This gap is increasingly consequential as real-world applications, from civic platforms to community journalism, demand AI systems that can reason about neighborhood-specific dynamics, cultural narratives, and local governance. Existing benchmarks fall short in capturing this complexity, often relying on coarse-grained data or isolated references. We pres

In brief

County-level local knowledge is a place where neither scale nor web search reliably helps. On numerical questions about U.S. counties, no model tested exceeds 15.5% accuracy, and larger models often do worse than smaller ones from the same family.

LocalBench contains 14,782 QA pairs over 526 counties in 49 states, built from Census/USDA indicators (34 localness indicators), local subreddit threads, and the NELA-Local news corpus, with QA pairs generated by o3, filtered by a DPO-tuned GPT-4o-mini, and labelled against a physical/cognitive/relational localness framework. 13 models were evaluated, closed-book and web-augmented, at temperature 0.0, 3 seeds.

Best non-numerical accuracy is 56.8% (Gemini-2.5-Pro+Grounding, GPT Judge). Grounding adds +13.6% for Gemini but costs GPT-4.1 −11.4%; on numerical items both gain (+9.3% and +5.3%). Qwen3-235B-A22B (6.6%) trails Qwen3-32B (4.9%)? in some metrics and underperforms 30B/32B/14B on non-numerical scores. GPT-4o answers only 39.8% of numerical questions.

Evidence is measured, with paired t-tests, Bonferroni correction, Cohen's d, and human validation of the judge (96% agreement) and filters (F1 0.94). But the scaling claim rests on 1 open model family, and the 2 web-augmented systems differ in search stack as well as model, so retrieval integration is not isolated.

U.S.-only, English-only, 2020–2025 sources; rural counties are thinner. Treat retrieval augmentation as model-dependent rather than a fix for place-specific facts.

Written from the abstract by claude-opus-5 on 2026-10-17, with every figure checked against it. Not a substitute for the paper.

Referenced by

Claims in this catalog that draw on this source, and whether as support or counterpoint.