Prepending the English twin's prompt raises the likelihood-scored accuracy of Qwen2.5-1.5B on the Polish questions of 600 MMLU translation pairs from 0.3400 to 0.5100, a gain of 0.1700 (95% interval 0.1250 to 0.2150) that recovers 0.7969 of its English-minus-Polish accuracy gap of 0.2133.

Relation: Recovers the gap by

model

Qwen2.5-1.5BModel scored.

intervention

Translation assistChange applied to each Polish prompt.

metric

Likelihood multiple-choice accuracyQuantity raised.

evaluation set

MMLU translation pairsItems scored.

baseline

0.3400 proportionAccuracy on the Polish questions without the intervention.

treated

0.5100 proportionAccuracy on the Polish questions with the intervention.

gain

0.1700 proportionTreated minus baseline.

gain interval low

0.1250 proportionLower bound of the 95% paired bootstrap interval of the gain.

gain interval high

0.2150 proportionUpper bound of the 95% paired bootstrap interval of the gain.

gap

0.2133 proportionEnglish-minus-Polish accuracy gap of the model on these pairs.

recovery

0.7969 ratioGain divided by the gap.

continued pretraining tokens

0 tokensTokens of continued pretraining of the model at the checkpoint scored.

Method and evaluation protocol

Accuracy on the Polish questions with the English twin's full prompt and a blank line prepended, Polish scaffold kept, minus accuracy without it; recovery divides the gain by English minus Polish accuracy on the same pairs; length-normalised likelihood, torch MPS bf16, no hooks.

Dataset

600 Belebele and 600 MMLU translation pairs, all pairs without exclusions.Version: unspecified · Access: public

Reported results

accuracy English 0.5533, Polish 0.3400, Polish with assist 0.5100; gain 0.1700 [0.1250, 0.2150]; gap 0.2133; recovery 0.7969.

Uncertainty and replication

95% paired item bootstrap interval of the gain (2000 resamples, seed 20260703); no interval for the recovery ratio.

Evidence references

behavioral_baseline_base.jsonl · publichttps://github.com/stw2/tokenizer-science-tax/blob/633b3ba1ce916e432ae5ba4e0a706ad58b53b2e9/experiments/E12-activation-patching/results/behavioral_baseline_base.jsonlbehavioral_assist_base.jsonl · publichttps://github.com/stw2/tokenizer-science-tax/blob/633b3ba1ce916e432ae5ba4e0a706ad58b53b2e9/experiments/E12-activation-patching/results/behavioral_assist_base.jsonlanalysis.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/633b3ba1ce916e432ae5ba4e0a706ad58b53b2e9/experiments/E12-activation-patching/results/analysis.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/633b3ba1ce916e432ae5ba4e0a706ad58b53b2e9/experiments/E12-activation-patching/results/metrics.json

Limitations

All 600 pairs per benchmark: the audited exclusions are not applied in this stage. Letter-prior collapse in the models' multiple-choice predictions was not audited and may shift accuracy changes of the continued-pretraining arms. One scoring run per condition. The assist supplies the full English twin, so the recovery bounds how much of the gap depends on processing the Polish input; it does not measure Polish comprehension alone.