Swapping the answer-scaffold language changes the likelihood-scored accuracy of Qwen2.5-1.5B by 0.0233 on Polish and -0.0017 on English questions of 600 Belebele translation pairs, and by 0.0150 and 0.0033 on 600 MMLU translation pairs; every absolute change is below 0.03.

Relation: Changes by

model

Qwen2.5-1.5BModel scored.

intervention

Scaffold language swapChange applied to each prompt.

metric

Likelihood multiple-choice accuracyQuantity whose change is stated.

evaluation set 1

Belebele translation pairsFirst set of items.

evaluation set 2

MMLU translation pairsSecond set of items.

change set 1 polish

0.0233 proportionChange on the Polish questions of the first set.

change set 1 english

-0.0017 proportionChange on the English questions of the first set.

change set 2 polish

0.0150 proportionChange on the Polish questions of the second set.

change set 2 english

0.0033 proportionChange on the English questions of the second set.

continued pretraining tokens

0 tokensTokens of continued pretraining of the model at the checkpoint scored.

threshold

0.03 proportionBound on the absolute change.

within threshold

trueWhether every absolute change is below threshold.

Method and evaluation protocol

Accuracy of each question language with the other language's answer scaffold minus accuracy with the matched scaffold, on the same items; length-normalised likelihood over four option letters, torch MPS bf16, no hooks.

Dataset

600 Belebele and 600 MMLU translation pairs, all pairs without exclusions.Version: unspecified · Access: public

Reported results

Belebele: Polish 0.0233, English -0.0017; MMLU: Polish 0.0150, English 0.0033; baseline accuracy Belebele English 0.7467, Polish 0.5483, MMLU English 0.5533, Polish 0.3400.

Uncertainty and replication

Point differences of accuracies on 600 items; no interval is computed for this contrast.

Evidence references

behavioral_baseline_base.jsonl · publichttps://github.com/stw2/tokenizer-science-tax/blob/633b3ba1ce916e432ae5ba4e0a706ad58b53b2e9/experiments/E12-activation-patching/results/behavioral_baseline_base.jsonlbehavioral_xscaffold_base.jsonl · publichttps://github.com/stw2/tokenizer-science-tax/blob/633b3ba1ce916e432ae5ba4e0a706ad58b53b2e9/experiments/E12-activation-patching/results/behavioral_xscaffold_base.jsonlanalysis.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/633b3ba1ce916e432ae5ba4e0a706ad58b53b2e9/experiments/E12-activation-patching/results/analysis.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/633b3ba1ce916e432ae5ba4e0a706ad58b53b2e9/experiments/E12-activation-patching/results/metrics.json

Limitations

All 600 pairs per benchmark: the audited exclusions are not applied in this stage. Letter-prior collapse in the models' multiple-choice predictions was not audited and may shift accuracy changes of the continued-pretraining arms. One scoring run per condition.