Paired English and Polish likelihood multiple-choice scoring of Qwen2.5-1.5B and two continued-pretraining checkpoints on Belebele and MMLU

https://github.com/stw2/tokenizer-science-tax @ 3c128813c117fd41b46d3a9949c2c8751134df27

Reference checked 2026-09-14 13:18 UTC. No code was executed or scientific result verified.

• experiments/E11-cross-language-access/scripts
• experiments/E11-cross-language-access/data
• experiments/E11-cross-language-access/DESIGN.md

Prediction

The base model's gap on Belebele is above zero; continued pretraining narrows the gap; the tokenizer transplant changes the gap by at most 3 percentage points; the MMLU gap differs between STEM and non-STEM subjects (exploratory).

Protocol

01 rebuilds seeded 600-item samples of Belebele (English and Polish) and MMLU, pairs MMLU with the openGPT-X Polish translation, gates pairing and gold parity and writes the item files. 02 scores every item zero-shot by the log-probability of each option letter after a scaffold in the item's language: twice on 10 items per file as a determinism gate, then on all items for the base model and both checkpoints. The analysis computes per-model English-minus-Polish gaps with sign tests, paired bootstrap contrasts between models, the decomposition of the continued-pretraining contrast into English and Polish accuracy changes, the STEM split on MMLU and the floor rule.

Dataset

600 Belebele test items in English and Polish (professionally translated, parallel) and 600 MMLU test items in English with their openGPT-X machine translation into Polish; 112 MMLU pairs are STEM.

Split

Test splits only: a seeded sample of 600 items per benchmark (seed 42); every sampled pair is scored by every model in both languages.

Access needs

The two continued-pretraining checkpoints are restricted materials: ask the Room owner, or rebuild them with the control-arm experiment's scripts. The APT4 tokenizer that the arm B loader checks against is in a gated repository.

Configurations

Three models (Qwen2.5-1.5B; its checkpoint after 500M tokens of continued pretraining with the original tokenizer; its checkpoint with APT4 by FVT after the same continued pretraining) by two languages by two benchmarks. Scaffolds Answer (letter): for English and Odpowiedź (litera): for Polish. bf16 on Apple MPS. The transplant arm's tokenizer loads through the whitespace-canary loader of the control-arm experiment's second plan version.

Metric

English-minus-Polish likelihood multiple-choice accuracy gap per model (length-normalised primary, raw alongside), with a paired normal 95% interval and an exact sign test on discordant pairs; differences of gaps and of per-language accuracies between models with paired bootstrap 95% intervals.

Seeds

42 (item sampling); 20260703 (bootstrap, 2000 resamples); scoring uses no randomness

Interpretation rule

H1 is confirmed if the 95% interval of the base model's English-minus-Polish gap on Belebele lies above zero; gaps below 3, from 3 to 8 and above 8 percentage points are small, moderate and large. H2 is confirmed if the gap difference between the continued-pretraining checkpoint and the base model is below zero with a 95% interval excluding zero; improved Polish access is claimed only if Polish accuracy rises with an interval excluding zero, and a narrowing carried by falling English accuracy with flat Polish accuracy is erosion. H3: an absolute gap difference between the two checkpoints of at most 3 percentage points is the null result; a widening above 3 points with an interval excluding zero is a knowledge-side injury. H4 is exploratory with no bin. Any model, language and benchmark cell whose 95% interval for accuracy includes 0.25 demotes every contrast involving it to exploratory. Belebele is the primary benchmark.

Resources

Apple silicon, 128 GB unified memory; torch MPS bf16, scoring only; hours estimated.

Prior work

arXiv:2604.10799v1 reports Belebele and INCLUDE scores of the Bielik v3 models before and after the APT4 transplant; the openGPT-X Polish MMLU translation is described in arXiv:2410.08928.