After the same 500M tokens of Polish-heavy continued pretraining, the English-minus-Polish accuracy gap of Qwen2.5-1.5B with APT4 by FVT is wider than with its original tokenizer on neither translation-paired Belebele items (difference -0.0418, 95% interval -0.0903 to 0.0100) nor translation-paired MMLU items (difference -0.0153, 95% interval -0.0662 to 0.0357).

Relation: Widened on a count of benchmarks

metric

English-minus-Polish accuracy gapQuantity compared.

subject

Qwen2.5-1.5B with APT4 by Fast Vocabulary Transfer, continuedModel with APT4 by FVT after the same continued pretraining.

comparator

Qwen2.5-1.5B continued with its own tokenizerModel after continued pretraining with its original tokenizer.

count widened

0 benchmarksBenchmarks where the subject's gap is wider with an interval above zero.

count total

2 benchmarksBenchmarks compared.

benchmarks

translation-paired Belebele items; translation-paired MMLU itemsBenchmarks compared.

training

500,170,752 Qwen tokens of continued pretraining of each arm on the same document sequence, constant learning rate before any decay anneal, one run per armTraining the continued-pretraining arms received.

Method and evaluation protocol

Count the benchmarks whose paired bootstrap 95% interval of the gap difference lies above zero.

Dataset

Translation-paired Belebele and MMLU items after the exclusions.Version: unspecified · Access: public

Reported results

0 of 2 benchmarks; both point estimates are below zero.

Uncertainty and replication

Paired bootstrap 95% intervals per benchmark.

Evidence references

analysis.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/5e8117ffb3f168331ee63f1032cd40a62c6296bd/experiments/E11-cross-language-access/results/analysis.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/5e8117ffb3f168331ee63f1032cd40a62c6296bd/experiments/E11-cross-language-access/results/metrics.json

Limitations

On Belebele, the primary benchmark, the difference lies outside the pre-registered 3-point null bin while its interval includes zero, so H3 is neither confirmed nor refuted there. One model family at 1.5B parameters; the checkpoints are pre-anneal, from one training run each. English MMLU items may be in Qwen2.5 pretraining data, so MMLU gaps can be inflated; Polish MMLU items are machine translations. The 13 excluded pairs were chosen after the base model's cells had been scored. Letter priors: on Polish MMLU items the checkpoint that kept the original tokenizer predicts A for 4 of 600 items and the APT4 checkpoint predicts B for 437, against 134 to 160 gold items per letter; on Polish Belebele items the APT4 checkpoint predicts B for 360 and the other checkpoint D for 326 of 600; the base model's predictions are the least concentrated (all scored rows, results/metrics.json pred_count_*).