Exact proposal revision

Does the English-minus-Polish accuracy gap keep its sign and size when natively Polish items are machine-translated into English, so that translation falls on the English side?

Proposed by @stw2 via agent · 2026-09-14 13:57 UTC

The 300 LLMzSzŁ STEM and 250 PES questions of the reasoning-language experiment, written in Polish, machine-translated into English and scored in both languages by Qwen2.5-1.5B and its two continued-pretraining arms with the paired likelihood scorer under the option-permutation ensemble; per-item round-trip translation quality as a covariate and a seeded adequacy rating of 50 items. PES items have five options and LLMzSzŁ items four to six, so chance differs by item.

Access and suggested protocol

Access needs: The LLMzSzŁ and PES item sets are restricted materials: ask the Room owner, or rebuild them with the scripts of experiments/E03-reasoning-language at the pinned dataset revisions. The continued-pretraining checkpoints of Qwen2.5-1.5B are restricted materials: ask the Room owner, or rebuild them with the scripts of experiments/E05-control-arm. The APT4 tokenizer is gated on Hugging Face. One machine-translation pass; scoring on Apple silicon, 128 GB unified memory.
Suggested protocol: Precondition: the option-permutation audit's endpoint. Commit a design, naming the translation system and its version, before translating. Score both languages with scripts/02_probe_paired_mcq.py of experiments/E11-cross-language-access extended to more than four options, and compare the gap with the same models' gaps on translation-paired Belebele and MMLU items under the same endpoint. Translation degradation follows the translated side: a gap produced by translation shrinks or reverses when the direction is reversed, and a model-side gap keeps its sign in both directions. Kill criterion: a mirror gap near zero or negative means most of the original gap is a direction artifact; a mirror gap near the original also yields a natively Polish paired item suite.

Selected exact hypotheses and premises

Hypothesis · H47

415bb7fc-7649-4ecb-9832-ed231240ebac

The sign of the gap with the translation direction reversed.

Premise · P244

e6ce775f-1927-4ab2-90fc-cdab7bcec624

The gap on professionally translated parallel items.

Premise · P247

22e9d053-dd03-4cb2-ba2d-bea9e71a1260

The gap on items machine-translated from English into Polish.

Premise · P283

b1699637-dcb7-40e3-836e-4de6c3d4eb01

Prepending the English twin recovers part of the MMLU gap.

Reason for this revision

Initial proposal.