NurseBench-RU: can large language models pass the Russian nursing licensing exam?

1. Introduction

Every graduate of a Russian medical college must pass primary accreditation. Its first stage is a computer-based test: 80 items drawn from a national item bank, four options, one correct, 60 minutes; a score of 70% or more is a pass. It is a standardized check of a nurse's core professional knowledge, in Russian, with answers keyed to Russian clinical practice.

Similar exams are already used to evaluate language models elsewhere — the US NCLEX-RN, China's National Nurse Licensure Examination, Japan's KokushiMD-10. To our knowledge, no such measurement with current models exists for Russian nursing accreditation. We ask two questions: would the models pass this exam, and do Russian and international models differ on Russian medical material?

2. Related work

• Nursing exams. ChatGPT on NCLEX-RN and the Chinese nursing exam (JMIR Medical Education, 2024); four models on 237 items of the 2024 Chinese exam (DeepSeek V3 and Gemini 2.0 Pro above 83%); a 2025 meta-analysis reporting pooled accuracy around 70% on nursing licensure exams, weakest on ethics and psychosocial topics.
• Multi-profession national exams. KokushiMD-10 — ten Japanese healthcare licensing exams, 30+ models.
• Russian benchmarks. RuMedBench (2022) — Russian medical language understanding tasks; MERA (ACL 2024) — a general benchmark for Russian. Neither covers nursing accreditation or current models.

3. Data

Source. An export of the item bank for the first stage of primary accreditation of specialists with secondary vocational education, specialty 34.02.01 "Nursing" (2018, 1899 items), published on a medical college website. Each item has an identifier such as [T000002], a question and four options; in the official export the correct option always comes first. PDF parsing recovered all 1899 items with no gaps.

Sample. For comparability, all models answer the same fixed random sample of 500 items (seed 7). Options within each item are shuffled deterministically (identically for every model), so a model cannot exploit the position of the key.

Closed benchmark. Redistribution terms for the bank are not established, so items are not released. Official access to the current bank is available only through the practice exam of the national Methodological Accreditation Center.

4. Method

Prompt. System instruction (in Russian): "You are taking the first stage of primary accreditation in Nursing in Russia. Each item has one correct answer. Reply with the letter of the correct option only: А, Б, В or Г." It is followed by the item and lettered options. No web search, no few-shot examples, one run per item.

Answer parsing. Reasoning blocks are removed and the last standalone option letter is taken. Latin look-alikes and transliterations (A/B/C/D, V, G) are mapped to Cyrillic. An unparsed reply counts as wrong — like a blank on the real exam. Final runs have practically no unparsed replies.

Metrics. Accuracy; 95% Wilson confidence interval; "passes the exam" — the share of 10,000 simulated exams of 80 items drawn from the model's answers in which it scores at least 56 (70%). Models are compared pairwise with an exact McNemar test on the same items.

Models and settings. Models were called through the developers' official APIs. Reasoning models ran at low reasoning effort; Claude — in adaptive thinking mode with a 4,000-token budget. Non-reasoning models ran at temperature 0. Per-model settings are in Table 1.

Corrected key (v0.2). Items on which the strongest models consistently disagree with the key were checked against current regulations (Section 7). All main results use the corrected key: 7 wrong or outdated keys are fixed and 4 ambiguous items are excluded for every model; official-key results are in Table 7.

Table 1. Access and call mode per model.

5. Results

Table 2. Results. * Share of 10,000 simulated 80-item exams with a 70% pass mark. Gemini 3.5 Flash-Lite was evaluated on 494 of 496 items (provider daily limit).

5.1. Every model passes

Even the weakest model in the sample (gpt-oss-120b, 75.8%) passes 93% of simulated exams; all others pass 100%. The margin over random guessing (29.0%) is 50–66 percentage points.

5.2. The leaders are indistinguishable

The top five (GPT-5.6 Terra, Claude Sonnet 5.5, GPT-6 Luna, DeepSeek V4.1 Flash, Alice AI) fall within 94.4%–96.8%. No pairwise comparison within this group is significant (Table 3): with 496 items, the right reading is a group of leaders, not a single winner. In particular, Yandex's Alice AI sits in the same group as the OpenAI and Anthropic flagships.

Table 3. Pairwise comparisons among the top five: items where only one model is correct, and the two-sided exact McNemar p-value.

5.3. Russian models

Yandex is represented by Alice AI (94.4%), YandexGPT 5.1 Pro (91.1%) and YandexGPT Lite (80.8%); Sber by three GigaChat versions: 2 Max (90.5%), 2 Pro (87.9%) and Lite (82.7%). Sber's flagship is on par with Claude Haiku 5.5 and slightly behind YandexGPT 5.1 Pro; the difference is within the confidence intervals.

6. Contamination check

The bank has been publicly available online for years, so its items may be in the models' training data. We tested this directly: models received only the question text and were asked to reconstruct the four options "as in the bank". The correct option can be inferred from knowledge, but reproducing the incorrect options (distractors) verbatim without having seen the bank is nearly impossible. An option counted as reproduced if at least 60% of its content words matched; only the last four lines of the reply (the final answer, not the reasoning) were scored. The probe used 100 items.

Table 4. Memorization probe results.

Finding. Traces of the bank appear in every model tested, most strongly in GPT-6 Luna. Russian models do not stand out. Still, 5–13% is far from memorization — a memorized bank would yield several times more. This is a limitation of any benchmark built on public exams rather than a property of a single model.

Methodological lesson. Our first version of the probe was misleading: adaptive-thinking models with a 300-token limit returned empty replies (understating their rate), while models that reason out loud produced dozens of lines of guesses (overstating it). The corrected probe gives a token budget, scores only the final lines and counts empty replies.

7. Disputed items

On 15 of 500 items, at least four of the five strongest international models choose an answer different from the official key. These are candidates for outdated or erroneous items in the 2018 bank, or items where Russian norms differ from international practice. How often the remaining models agree with the official key on these items:

Table 5. Agreement with the official key on disputed items.

7.1. Check against current regulations

We checked all 15 disputed items against Russian regulations, clinical guidelines and textbooks in force as of October 2026. In 7 cases the 2018 key is wrong or outdated and the models are right; in four the key is correct; four more items are ambiguous. Item texts are not released — only topics.

Table 6. Verification results. Based on open sources; confirmation by a practising clinician is still needed, and some primary sources are only available in paid legal databases.

7.2. Accuracy with the corrected key

Table 7. Accuracy before and after correcting 7 keys.

The correction lifts strong models by 1.5–2 pp and barely changes the order. Weaker models pick the wrong key more often (up to 4 of 7), the leaders almost never. This more likely reflects knowledge gaps than memorization of the bank: there are too few items for a stronger claim.

8. Observations

• Transliteration. GigaChat sometimes answers with a Latin transliteration of the letter: "V" for "В", "G" for "Г".
• Blank answer sheet. Adaptive-thinking models (Claude 5.5) reason first on hard items; with a small output limit they return empty text. Final runs used a token budget and uniform settings.
• Thinking out loud. Some reasoning models (DeepSeek) write their reasoning into the reply, sometimes in English; the parser must take the final answer, not the first letter it sees.

9. Limitations

• The bank is from 2018; the current bank may differ and some keys may be outdated (Section 7).
• 496 of 1899 items (500 minus 4 ambiguous), one run per item; confidence intervals are ±2–4 percentage points.
• Settings differ across providers (reasoning effort, temperature) within what each API allows.
• The contamination probe is indirect and used 100 items; it neither proves nor rules out the bank's presence in training data.
• A multiple-choice test does not measure clinical reasoning, patient communication or practical skills — the second and third stages of accreditation.

10. Conclusions and future work

All 14 models would pass the first stage of Russian primary accreditation in nursing. The top five, including Yandex's Alice AI, are statistically indistinguishable. Sber's flagship trails the leaders by 4–6 percentage points. All large models are partially familiar with the wording of the public bank.

Next: open models that can run locally without external services (YandexGPT-5-Lite, T-lite, Vikhr, GigaChat3); physician specialties (general medicine, pediatrics); review of disputed items by practising clinicians.

Appendix. Reproduction

python tools/parse_bank.py raw/okp-sd.pdf data/bank_2018.jsonl
python tools/bench.py --model yandex:aliceai-llm/latest --subset 500
python tools/bench.py --model openai:gpt-6-luna --subset 500 --max-tokens 2000 --reasoning low --workers 8
python tools/bench.py --model anthropic:claude-sonnet-5-5 --subset 500 --max-tokens 4000 --workers 6
python tools/report.py --subset 500
python tools/probe_memory.py --model openai:gpt-6-luna --n 100

The evaluation and contamination-probe code is ready for release; the item bank is not released.