Attributed research checkpoint
E5: at matched continued pretraining of Qwen2.5-1.5B, most of the Polish gain comes from the training, while the APT4 transplant arm stays 0.12 to 0.25 bits per byte behind on English and formal texts.
Assignment, status and accepted plans remain authoritative on each experiment.
Reported state
- Reported progress
- E5 completed with three retrospective plan versions (design, Amendment 1, Amendment 2): both arms trained to 0.5B tokens and scored at every 100M milestone; the 1B extension was not run.
- Open issues
- The design was committed with the results and fixed no decision rule, and a substantial share is undefined. One run per arm; values before the decay anneal the design owed; Fast Vocabulary Transfer instead of FOCUS; 1.5B parameters and 0.5B tokens. The math_clean values of the untouched model and of arm A at 100M use 800 statements, the others 200. The time-zero arm-B BPB and multiple-choice rows were scored with an earlier tokenizer loader and not re-scored. The multiple-choice sets are Polish and near chance. The training text was not kept. Whether the formal residual is permanent or slow convergence is open.
- Suggested next action
- Extend both arms to 1B tokens from their last state and anneal both before comparing (the unlaunched Amendment 2), as a follow-up proposal.
- Access needs
- The APT4 tokenizer is gated on Hugging Face. Checkpoints, the English holdout, four E1 corpora and two E3 item sets are restricted materials: ask the Room owner.
Exact references
Exact record · H13
f0470dd9-4d77-4e01-a0a7-f3c1296a5ac2The attribution E5 tests; no threshold was fixed.
Exact record · P123
bac97d43-6754-4355-8c4b-8c7767494b1fMost of the Polish movement comes from continued pretraining; inconclusive without a decision rule.
Exact record · P108
0c90c839-74ad-4f13-8f54-0b2dfbc9afbbTransplant residual on English is most of the change.
Exact record · P117
d9ab6218-cf24-498a-90f1-ebd986fb3508Continued pretraining alone moves English little.
Exact record · P125
f1355715-5d7f-4a04-a4bf-249f33aa288eResidual still closing at 500M; persistence undecided.
Exact record · P129
e2da0e41-7694-4b22-a1ea-14d13dc51f38Residual no longer closing between 400M and 500M.
Exact record · P133
683a738d-67cc-4630-b105-e5ad71c2d77aTokenizer-specific arithmetic deficit after matched training.
Exact record · P132
2566f4fe-c7ef-43f0-bb66-0e7ca02bbe3eArithmetic decline from continued pretraining alone.
Experiment · E5
Open experiment →Accepted plan
Exact plan →Accepted plan
Exact plan →Accepted plan
Exact plan →