Prepending the English twin's prompt raises the likelihood-scored accuracy of Qwen2.5-1.5B on the Polish questions of 600 Belebele translation pairs from 0.5483 to 0.6367, a gain of 0.0883 (95% interval 0.0483 to 0.1317) that recovers 0.4454 of its English-minus-Polish accuracy gap of 0.1983.

Relation: Recovers the gap by

model

Qwen2.5-1.5BModel scored.

intervention

Translation assistChange applied to each Polish prompt.

metric

Likelihood multiple-choice accuracyQuantity raised.

evaluation set

Belebele translation pairsItems scored.

baseline

0.5483 proportionAccuracy on the Polish questions without the intervention.

treated

0.6367 proportionAccuracy on the Polish questions with the intervention.

gain

0.0883 proportionTreated minus baseline.

gain interval low

0.0483 proportionLower bound of the 95% paired bootstrap interval of the gain.

gain interval high

0.1317 proportionUpper bound of the 95% paired bootstrap interval of the gain.

gap

0.1983 proportionEnglish-minus-Polish accuracy gap of the model on these pairs.

recovery

0.4454 ratioGain divided by the gap.

continued pretraining tokens

0 tokensTokens of continued pretraining of the model at the checkpoint scored.

Method and evaluation protocol

Accuracy on the Polish questions with the English twin's full prompt and a blank line prepended, Polish scaffold kept, minus accuracy without it; recovery divides the gain by English minus Polish accuracy on the same pairs; length-normalised likelihood, torch MPS bf16, no hooks.

Dataset

600 Belebele and 600 MMLU translation pairs, all pairs without exclusions.Version: unspecified · Access: public

Reported results

accuracy English 0.7467, Polish 0.5483, Polish with assist 0.6367; gain 0.0883 [0.0483, 0.1317]; gap 0.1983; recovery 0.4454.

Uncertainty and replication

95% paired item bootstrap interval of the gain (2000 resamples, seed 20260703); no interval for the recovery ratio.

Evidence references

behavioral_baseline_base.jsonl · publichttps://github.com/stw2/tokenizer-science-tax/blob/633b3ba1ce916e432ae5ba4e0a706ad58b53b2e9/experiments/E12-activation-patching/results/behavioral_baseline_base.jsonlbehavioral_assist_base.jsonl · publichttps://github.com/stw2/tokenizer-science-tax/blob/633b3ba1ce916e432ae5ba4e0a706ad58b53b2e9/experiments/E12-activation-patching/results/behavioral_assist_base.jsonlanalysis.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/633b3ba1ce916e432ae5ba4e0a706ad58b53b2e9/experiments/E12-activation-patching/results/analysis.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/633b3ba1ce916e432ae5ba4e0a706ad58b53b2e9/experiments/E12-activation-patching/results/metrics.json

Limitations

All 600 pairs per benchmark: the audited exclusions are not applied in this stage. Letter-prior collapse in the models' multiple-choice predictions was not audited and may shift accuracy changes of the continued-pretraining arms. One scoring run per condition. The assist supplies the full English twin, so the recovery bounds how much of the gap depends on processing the Polish input; it does not measure Polish comprehension alone.