On 598 translation-paired Belebele items, the likelihood multiple-choice accuracy of Qwen2.5-1.5B after 500M tokens of Polish-heavy continued pretraining with its original tokenizer is 0.6087 in English and 0.4314 in Polish, an English-minus-Polish gap of 0.1773 (95% interval 0.1378 to 0.2168).

Relation: English and Polish accuracy gap

metric

Likelihood multiple-choice accuracyQuantity measured in each language.

benchmark

Translation-paired Belebele itemsItem pairs scored.

setting

Qwen2.5-1.5B continued with its own tokenizerModel scored.

subject

EnglishLanguage of the first value.

subject value

0.6087 accuracyAccuracy on the English items.

comparator

PolishLanguage of the second value.

comparator value

0.4314 accuracyAccuracy on the Polish items.

difference

0.1773 accuracyEnglish minus Polish accuracy.

interval low

0.1378 accuracyLower bound of the 95% interval of the difference.

interval high

0.2168 accuracyUpper bound of the 95% interval of the difference.

p value

0.0000000000000000150 probabilityExact two-sided sign test on discordant pairs.

training

500,170,752 Qwen tokens of continued pretraining of each arm on the same document sequence, constant learning rate before any decay anneal, one run per armTraining the continued-pretraining arms received.

Method and evaluation protocol

Each item pair is scored in English and Polish by the log-probability of each option letter after a scaffold in the item's language; accuracy uses length-normalised log-probabilities; the interval is the normal approximation for paired proportions and the p-value the exact sign test on discordant pairs.

Dataset

598 pairs of the same FLORES passage, question and four options in English and in Polish from the Belebele test split: a seed-42 sample of 600 with the structurally malformed pairs excluded.Version: facebook/belebele@7899cdfa4e1e0d733fd77c848e2c273cb1d32be2 · Access: public

Reported results

English 0.6087, Polish 0.4314, gap 0.1773 [0.1378, 0.2168]; pairs correct in English only 135, in Polish only 29; sign test p 0.0000000000000000150.

Uncertainty and replication

Paired normal 95% interval over item pairs; exact two-sided sign test on discordant pairs.

Evidence references

analysis.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/5e8117ffb3f168331ee63f1032cd40a62c6296bd/experiments/E11-cross-language-access/results/analysis.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/5e8117ffb3f168331ee63f1032cd40a62c6296bd/experiments/E11-cross-language-access/results/metrics.json

Limitations

One model family at 1.5B parameters; the checkpoints are pre-anneal, from one training run each. English MMLU items may be in Qwen2.5 pretraining data, so MMLU gaps can be inflated; Polish MMLU items are machine translations. The 13 excluded pairs were chosen after the base model's cells had been scored. Letter priors: on Polish MMLU items the checkpoint that kept the original tokenizer predicts A for 4 of 600 items and the APT4 checkpoint predicts B for 437, against 134 to 160 gold items per letter; on Polish Belebele items the APT4 checkpoint predicts B for 360 and the other checkpoint D for 326 of 600; the base model's predictions are the least concentrated (all scored rows, results/metrics.json pred_count_*).