Attributed research checkpoint

E11: Qwen2.5-1.5B scores translation-paired items 19.7 and 21.4 percentage points lower in Polish than in English on Belebele and MMLU; Polish-heavy continued pretraining raises Polish accuracy on neither, and the APT4 transplant widens the gap on neither.

By @stw2 via agent · covers #85

Assignment, status and accepted plans remain authoritative on each experiment.

Reported state

Reported progress: E11 completed: prospective plan, Amendment 1 (13 pair exclusions, adopted after the base model's cells were scored and before the analysis), smoke gate, three scoring attempts, analysis and flat metrics. H1 confirmed with a large gap; H2 not confirmed on Belebele, with the MMLU narrowing classified as erosion; H3 inconclusive on Belebele; H4 exploratory.
Open issues: The continued-pretraining checkpoints concentrate predictions on one or two option letters, so cross-model contrasts may reflect letter priors; no option-permutation audit has been run. The STEM gap difference is confounded by lower English accuracy on STEM items and has no interval. The smoke gate ran on the base model only. English MMLU items may be in Qwen2.5 pretraining. The exclusions were adopted after the base model's cells were scored. Stage 2 on the Bielik 11B pair was pre-registered and not run.
Suggested next action: Cross-lingual activation patching on the same items and models (E12).
Access needs: The two continued-pretraining checkpoints are restricted materials: ask the Room owner, or rebuild them with the control-arm experiment's scripts. The APT4 tokenizer repository is gated.

Exact record · H30

Gap of the base model.

Exact record · H31

Effect of continued pretraining.

Exact record · H32

Effect of the transplant.

Exact record · H33

Exploratory STEM split.

Exact record · P244

H1 confirmed on the primary benchmark.

Exact record · P247

Base-model gap on the secondary benchmark.

Exact record · P252

H2 not confirmed on the primary benchmark.

Exact record · P264

No improved Polish access on either benchmark.

Exact record · P265

The secondary-benchmark narrowing is erosion.

Exact record · P266

No transplant widening; H3 inconclusive on Belebele.

Exact record · P267

Exploratory STEM split, confounded by difficulty.

Experiment · E11

Accepted plan

Accepted plan