Swapping the answer-scaffold language changes the likelihood-scored accuracy of Qwen2.5-1.5B with APT4 by Fast Vocabulary Transfer after 500M tokens of continued pretraining by 0.0250 on Polish and -0.0483 on English questions of 600 Belebele translation pairs, and by 0.0383 and -0.0600 on 600 MMLU translation pairs; not every absolute change is below 0.03.

Relation: Changes by

model

Qwen2.5-1.5B with APT4 by Fast Vocabulary Transfer, continuedModel scored.

intervention

Scaffold language swapChange applied to each prompt.

metric

Likelihood multiple-choice accuracyQuantity whose change is stated.

evaluation set 1

Belebele translation pairsFirst set of items.

evaluation set 2

MMLU translation pairsSecond set of items.

change set 1 polish

0.0250 proportionChange on the Polish questions of the first set.

change set 1 english

-0.0483 proportionChange on the English questions of the first set.

change set 2 polish

0.0383 proportionChange on the Polish questions of the second set.

change set 2 english

-0.0600 proportionChange on the English questions of the second set.

continued pretraining tokens

500170752 tokensTokens of continued pretraining of the model at the checkpoint scored.

threshold

0.03 proportionBound on the absolute change.

within threshold

falseWhether every absolute change is below threshold.

Method and evaluation protocol

Accuracy of each question language with the other language's answer scaffold minus accuracy with the matched scaffold, on the same items; length-normalised likelihood over four option letters, torch MPS bf16, no hooks.

Dataset

600 Belebele and 600 MMLU translation pairs, all pairs without exclusions.Version: unspecified · Access: public

Reported results

Belebele: Polish 0.0250, English -0.0483; MMLU: Polish 0.0383, English -0.0600; baseline accuracy Belebele English 0.6117, Polish 0.4767, MMLU English 0.4550, Polish 0.3617.

Uncertainty and replication

Point differences of accuracies on 600 items; no interval is computed for this contrast.

Evidence references

behavioral_baseline_armB_500M.jsonl · publichttps://github.com/stw2/tokenizer-science-tax/blob/633b3ba1ce916e432ae5ba4e0a706ad58b53b2e9/experiments/E12-activation-patching/results/behavioral_baseline_armB_500M.jsonlbehavioral_xscaffold_armB_500M.jsonl · publichttps://github.com/stw2/tokenizer-science-tax/blob/633b3ba1ce916e432ae5ba4e0a706ad58b53b2e9/experiments/E12-activation-patching/results/behavioral_xscaffold_armB_500M.jsonlanalysis.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/633b3ba1ce916e432ae5ba4e0a706ad58b53b2e9/experiments/E12-activation-patching/results/analysis.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/633b3ba1ce916e432ae5ba4e0a706ad58b53b2e9/experiments/E12-activation-patching/results/metrics.json

Limitations

All 600 pairs per benchmark: the audited exclusions are not applied in this stage. Letter-prior collapse in the models' multiple-choice predictions was not audited and may shift accuracy changes of the continued-pretraining arms. One scoring run per condition.