expert-authored-saudi-dialect-prompts-scored-against-rubric-criteria-derived-expert CLM-0386On expert-authored Saudi-dialect prompts scored against rubric criteria derived from expert ground truths, four frontier models all score under 55% and their dominant failure is distorting register and flattening pragmatic nuance (37.3% of 466 errors) rather than stating falsehoods (11.2%), so MSA-fluent models are not dialectally competent.
Capability: Handling culture-specific norms and viewpoints
Observed on
31 Saudi-dialect prompts, 124 model-prompt evaluations, Claude Opus 5, Gemini 3.7, GPT-5.6, Kimi K3; rubric scoring with penalties for introduced errors..
Sources
- Single benchmark, small (31 prompts), expert-authored and expert-scored; error taxonomy applied by the authors, so category shares depend on their rubric. No human baseline reported, and no comparison against MSA prompts to isolate dialect as the cause.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For a domain-specific language essentially absent from pretraining corpora (FORM symbolic algebra), QLoRA fine-tuning an 8B model on a few thousand examples validated by executing them through the language's own interpreter beats frontier open-weight models of up to 756B parameters given syntax documentation in context, on execution rate and strict output match for the larger benchmarks, while general reasoning scores drop under 3 points.Generating and editing working code · reviewed by AI
- When an 8B model verifies natural-language claims against knowledge-graph triples, scoring the evidence twice under opposing prompts (argue it supports, argue it refutes) and adjudicating between the two arguments avoids the bias toward a "Supported" verdict that a single-perspective assessment shows, and is the largest single contributor to accuracy in that pipeline.Checking claims against evidence · reviewed by AI
- Across 14 models, syllogisms posed in symbolic form are solved far better than the same inferences in natural language: top models reach near-perfect symbolic accuracy (about 99.6%) while scoring around chance (52%) on the natural-language understanding variant, mean 81.7% versus 56% overall, and few-shot prompting did not help, so formal-pattern competence does not carry over to reading the logic out of ordinary sentences.Deducing validly from logical structure · reviewed by AI
- Across 30 open LLaMA/Mistral/Qwen models and 8 classification tasks, a model's self-reported numeric confidence tracks its own logit-based confidence only weakly at the instance level (mean Pearson r about 0.135) even though setting-level averages of the two correlate moderately, and instruction-tuned models show larger linguistic-vs-internal gaps and worse calibration error than base models.Saying it does not know · reviewed by AI
- On long-context tasks over 8K-128K-token documents in the six official UN languages, models answer word-level span questions well (about 73% average accuracy across 12 models) but degrade sharply as the required unit coarsens to sentence selection (about 21%), paragraph filling and document-level summarization and translation, so long-context competence measured by needle-style word retrieval overstates comprehension of larger discourse units.Losing information in long inputs · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: A larger or independent Saudi-dialect rubric evaluation where frontier models exceed 55% macro-average, or where factual hallucination rather than register/pragmatic distortion is the leading error category.