arxiv-2511-05040 SRC-0538 · paper

UA-Code-Bench: A Competitive Programming Benchmark for Evaluating LLM Code Generation in Ukrainian

M. Syromiatnikov, V. Ruvinskaya

Created: 2025-11-03 · Ingested: 2026-09-23

https://arxiv.org/abs/2511.05040(opens in a new tab)

Evaluating the real capabilities of large language models in low-resource languages still represents a challenge, as many existing benchmarks focus on widespread tasks translated from English or evaluate only simple language understanding. This paper introduces UA-Code-Bench, a new open-source benchmark established for a thorough evaluation of language models'code generation and competitive programming problem-solving abilities in Ukrainian. The benchmark comprises 500 problems from the Eolymp platform, evenly distributed across five complexity levels from very easy to very hard. A diverse set

In brief

Competitive programming problems stated in Ukrainian remain unsolved about half the time by the strongest reasoning models, so low-resource-language code generation is not a solved transfer case.

UA-Code-Bench draws 500 problems from the Eolymp platform, evenly split across 5 complexity levels from very easy to very hard. 13 proprietary and open-source models generated Python solutions from a one-shot prompt, and submissions were judged inside the Eolymp environment against hidden tests rather than by string match or LLM grader.

The top performers, OpenAI o3 and GPT-5, solve only half of the problems. The authors also break down accuracy by difficulty level and report solution uniqueness plus runtime and memory consumption of the generated code.

The correctness signal is measured, not argued, and hidden-test execution is a stronger check than most translated benchmarks offer. What is missing is an isolation of language: no matched English version of the same 500 problems is reported here, so the gap between Ukrainian difficulty and problem difficulty is not separated.

Only Python, only one-shot prompting, no repeated sampling or pass@k, and no test-time budget comparison across models.

Treat it as a hard execution-verified ceiling measurement for Ukrainian prompts, not as evidence about where the failure originates.

Written from the abstract by claude-opus-5 on 2026-10-24, with every figure checked against it. Not a substitute for the paper.

Referenced by

Claims in this catalog that draw on this source, and whether as support or counterpoint.

No claims cite this yet.