Experiment · E5
How much of the change from an untouched model to the same model after an APT4 transplant and continued pretraining is attributable to the continued pretraining rather than to the tokenizer replacement, when the original-tokenizer model receives identical continued pretraining?
Qwen2.5-1.5B in three arms: continued pretraining with its own tokenizer; the same continued pretraining after an APT4 transplant with Fast Vocabulary Transfer initialisation; the untouched model. 0.5B Qwen tokens of an 80/20 Polish and English mix without mathematics or code. Bits per byte on Polish, English, scientific and synthetic arithmetic texts, Polish multiple-choice items and a digit-arithmetic probe every 100M tokens.
Prerequisites and protocol
- Access needs
- The APT4 tokenizer is gated on Hugging Face. Checkpoints, the English holdout, four E1 corpora and two E3 item sets are restricted materials: ask the Room owner. The training text and packed data were not kept.
- Suggested protocol
- Scripts 00 to 11 in experiments/E05-control-arm, in the order of its README.
Accepted plan
Plan accepted by @stw2
Matched-budget continued pretraining, banked at 0.5B, with a 1B extension that was not launched- requires Qwen2.5-1.5B · checkpoint · M124 Qwen2.5-1.5B · Download
- requires APT4 tokenizer.json · tokenizer · M125 APT4 tokenizer.json (Bielik-PL-11B-v3.0-Instruct) · Ask the reporter
- requires FineWeb2-HQ · training data · M126 FineWeb2-HQ · Download
- requires SlimPajama-6B · training data · M127 SlimPajama-6B · Download
- requires math_clean · evaluation data · M128 math_clean evaluation statements · Download
- requires E1 en-arxiv-abstracts · evaluation data · M27 en-arxiv-abstracts.jsonl · Download
- requires E1 en-latex-methods · evaluation data · M28 en-latex-methods.jsonl · Ask the reporter
- requires E1 en-python-code · evaluation data · M29 en-python-code.jsonl · Ask the reporter
- requires E1 en-gsm8k · evaluation data · M30 en-gsm8k.jsonl · Download
- requires E1 pl-wiki-science · evaluation data · M32 pl-wiki-science.jsonl · Download
- requires E1 pl-science-pes · evaluation data · M31 pl-science-pes.jsonl · Ask the reporter
- requires E1 pl-informal · evaluation data · M33 pl-informal.jsonl · Ask the reporter
- requires E3 llmzszl_stem items · evaluation data · M93 items_llmzszl_stem.jsonl · Ask the reporter
- requires E3 PES items · evaluation data · M94 items_pes.jsonl · Ask the reporter
- requires E3 GSM8K-PL items · evaluation data · M92 items_gsm8k_pl.jsonl · Download
- requires E2 probes · evaluation data · M63 probes.jsonl · Download
Responsibility and reported status
Taking starts no computation. Progress and completion are author reports; completion does not mean scientific success. Release does not prove a process stopped.
Room members can take or report on this experiment. Visit the Room to request membership.
Related findings
Execution attempts · 18
18 succeeded
Each attempt pins one public commit and configuration and delivers its own reported start and outcome. Reported times come from the registrant’s tool; received times are the server’s. An attempt without a delivered outcome stays unknown. Attempts verify no computation or result, and findings never require them.
planned attempt · stw2/tokenizer-science-tax @ 4d64dedcbe57 · under an accepted plan
A35 Builds the arm-B model: APT4 transplanted into Qwen2.5-1.5B with Fast Vocabulary Transfer initialisation.
Succeededplanned attempt · stw2/tokenizer-science-tax @ 4d64dedcbe57 · under an accepted plan
planned attempt · stw2/tokenizer-science-tax @ 4d64dedcbe57 · under an accepted plan
planned attempt · stw2/tokenizer-science-tax @ 4d64dedcbe57 · under an accepted plan
planned attempt · stw2/tokenizer-science-tax @ 4d64dedcbe57 · under an accepted plan
planned attempt · stw2/tokenizer-science-tax @ b4c55e6e7d26 · under an accepted plan
A40 Trains arm A: Qwen2.5-1.5B with its own tokenizer, 0.5B tokens of continued pretraining.
Succeededplanned attempt · stw2/tokenizer-science-tax @ 4d64dedcbe57 · under an accepted plan
planned attempt · stw2/tokenizer-science-tax @ b4c55e6e7d26 · under an accepted plan
planned attempt · stw2/tokenizer-science-tax @ b4c55e6e7d26 · under an accepted plan
planned attempt · stw2/tokenizer-science-tax @ b4c55e6e7d26 · under an accepted plan
A44 Freezes the digit-probe spec: items, exemplars, prompt formats and decoding settings drawn from E2's probes.
Succeededplanned attempt · stw2/tokenizer-science-tax @ b4c55e6e7d26 · under an accepted plan
planned attempt · stw2/tokenizer-science-tax @ b4c55e6e7d26 · under an accepted plan
planned attempt · stw2/tokenizer-science-tax @ b4c55e6e7d26 · under an accepted plan
A47 Scores bits per byte of the transplant at time zero and arm B's five milestones on ten texts.
Succeededplanned attempt · stw2/tokenizer-science-tax @ b4c55e6e7d26 · under an accepted plan
A48 Scores multiple-choice accuracy of the transplant at time zero and arm B's five milestones.
Succeededplanned attempt · stw2/tokenizer-science-tax @ b4c55e6e7d26 · under an accepted plan
planned attempt · stw2/tokenizer-science-tax @ b4c55e6e7d26 · under an accepted plan
planned attempt · stw2/tokenizer-science-tax @ 9a57aa40f4e7 · under an accepted plan
A51 Writes flat headline values from the analysis, the scoring rows and the transplant statistics.
Succeededplanned attempt · stw2/tokenizer-science-tax @ 9a57aa40f4e7 · under an accepted plan
Responsibility and plan history
Contribute with your agent: “Find The tokenizer science tax, thread The missing control arm. Help me prepare the hypotheses, an open experiment, a plan or a checkpoint I select. Show me the meaning for review before publishing.”
Existing results can go straight to Publish a claim or finding. Hypotheses and experiments are optional.
Research guide →