Swapping the answer-scaffold language changes the likelihood-scored accuracy of Qwen2.5-1.5B after 500M tokens of continued pretraining with its own tokenizer by 0.0167 on Polish and 0.0017 on English questions of 600 Belebele translation pairs, and by 0.0183 and -0.0233 on 600 MMLU translation pairs; every absolute change is below 0.03.

Relation: Changes by

model

Qwen2.5-1.5B continued with its own tokenizerModel scored.

intervention

Scaffold language swapChange applied to each prompt.

metric

Likelihood multiple-choice accuracyQuantity whose change is stated.

evaluation set 1

Belebele translation pairsFirst set of items.

evaluation set 2

MMLU translation pairsSecond set of items.

change set 1 polish

0.0167 proportionChange on the Polish questions of the first set.

change set 1 english

0.0017 proportionChange on the English questions of the first set.

change set 2 polish

0.0183 proportionChange on the Polish questions of the second set.

change set 2 english

-0.0233 proportionChange on the English questions of the second set.

continued pretraining tokens

500170752 tokensTokens of continued pretraining of the model at the checkpoint scored.

threshold

0.03 proportionBound on the absolute change.

within threshold

trueWhether every absolute change is below threshold.

Method and evaluation protocol

Accuracy of each question language with the other language's answer scaffold minus accuracy with the matched scaffold, on the same items; length-normalised likelihood over four option letters, torch MPS bf16, no hooks.

Dataset

600 Belebele and 600 MMLU translation pairs, all pairs without exclusions.Version: unspecified · Access: public

Reported results

Belebele: Polish 0.0167, English 0.0017; MMLU: Polish 0.0183, English -0.0233; baseline accuracy Belebele English 0.6100, Polish 0.4300, MMLU English 0.4500, Polish 0.3417.

Uncertainty and replication

Point differences of accuracies on 600 items; no interval is computed for this contrast.

Evidence references

behavioral_baseline_armA_500M.jsonl · publichttps://github.com/stw2/tokenizer-science-tax/blob/633b3ba1ce916e432ae5ba4e0a706ad58b53b2e9/experiments/E12-activation-patching/results/behavioral_baseline_armA_500M.jsonlbehavioral_xscaffold_armA_500M.jsonl · publichttps://github.com/stw2/tokenizer-science-tax/blob/633b3ba1ce916e432ae5ba4e0a706ad58b53b2e9/experiments/E12-activation-patching/results/behavioral_xscaffold_armA_500M.jsonlanalysis.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/633b3ba1ce916e432ae5ba4e0a706ad58b53b2e9/experiments/E12-activation-patching/results/analysis.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/633b3ba1ce916e432ae5ba4e0a706ad58b53b2e9/experiments/E12-activation-patching/results/metrics.json

Limitations

All 600 pairs per benchmark: the audited exclusions are not applied in this stage. Letter-prior collapse in the models' multiple-choice predictions was not audited and may shift accuracy changes of the continued-pretraining arms. One scoring run per condition.