Difference value
After the budget of procedure applied to subject, the treatment arm's metric minus the control arm's metric is value, with its 95% bootstrap interval.
Key difference_value · version 8df69c78-d45c-441b-a44e-cd87b14448e7
Finding · P138 · Author-curated
After 1B tokens of recovery pretraining of an APT4 FVT transplant of Qwen2.5-0.5B, pooled Polish bits per byte with a 30% math and code share exceeds that without math and code by 0.0187 (95% interval 0.0164 to 0.0205).
Structured assertion
Relation: Difference value
- subject
- APT4 FVT transplantModel family recovered.
- base model
- Qwen2.5-0.5BBase model of the transplant and reference of the gaps.
- procedure
- Continued pretrainingTraining applied.
- variable
- Math and code shareQuantity varied.
- treatment value
- 30 percentMath and code share of the treatment.
- control value
- 0 percentMath and code share of the control.
- budget
- 1000000000 tokensRecovery pretraining tokens per arm.
- metric
- Pooled Polish bits per byteMetric compared.
- value
- 0.0187 bits per byteTreatment minus control.
- interval low
- 0.0164 bits per byteLower bound of the 95% bootstrap interval.
- interval high
- 0.0205 bits per byteUpper bound of the 95% bootstrap interval.
Experimental provenance
- Method and evaluation protocol
- Per-document scores of the final checkpoints of the 0% and 30% arms on the pooled 800 Polish documents; byte-weighted bits per byte; 2,000 bootstrap resamples of paired documents, seed 20260707.
- Dataset
- The first 200 documents of four Polish texts: FineWeb2-HQ holdout, Polish Wikipedia science articles, PES questions, reviews.Version: unspecified · Access: restricted
- Reported results
- 0.0187 [0.0164, 0.0205] bits per byte.
- Uncertainty and replication
- 95% percentile interval from 2,000 paired bootstrap resamples. One training run per arm, so the intervals cover evaluation sampling only, not training variance.
- Evidence references
- analysis.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/2457520fc798b19bee6893ac58e0357ec4c467d5/experiments/E06-math-code-rescue/results/analysis.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/2457520fc798b19bee6893ac58e0357ec4c467d5/experiments/E06-math-code-rescue/results/metrics.jsonperdoc/mix00_final__pl.npz · publichttps://github.com/stw2/tokenizer-science-tax/blob/2457520fc798b19bee6893ac58e0357ec4c467d5/experiments/E06-math-code-rescue/results/perdoc/mix00_final__pl.npzperdoc/mix00_final__pl_wiki_sci.npz · publichttps://github.com/stw2/tokenizer-science-tax/blob/2457520fc798b19bee6893ac58e0357ec4c467d5/experiments/E06-math-code-rescue/results/perdoc/mix00_final__pl_wiki_sci.npzperdoc/mix00_final__pl_pes.npz · publichttps://github.com/stw2/tokenizer-science-tax/blob/2457520fc798b19bee6893ac58e0357ec4c467d5/experiments/E06-math-code-rescue/results/perdoc/mix00_final__pl_pes.npzperdoc/mix00_final__pl_informal.npz · publichttps://github.com/stw2/tokenizer-science-tax/blob/2457520fc798b19bee6893ac58e0357ec4c467d5/experiments/E06-math-code-rescue/results/perdoc/mix00_final__pl_informal.npzperdoc/mix30_final__pl.npz · publichttps://github.com/stw2/tokenizer-science-tax/blob/2457520fc798b19bee6893ac58e0357ec4c467d5/experiments/E06-math-code-rescue/results/perdoc/mix30_final__pl.npzperdoc/mix30_final__pl_wiki_sci.npz · publichttps://github.com/stw2/tokenizer-science-tax/blob/2457520fc798b19bee6893ac58e0357ec4c467d5/experiments/E06-math-code-rescue/results/perdoc/mix30_final__pl_wiki_sci.npzperdoc/mix30_final__pl_pes.npz · publichttps://github.com/stw2/tokenizer-science-tax/blob/2457520fc798b19bee6893ac58e0357ec4c467d5/experiments/E06-math-code-rescue/results/perdoc/mix30_final__pl_pes.npzperdoc/mix30_final__pl_informal.npz · publichttps://github.com/stw2/tokenizer-science-tax/blob/2457520fc798b19bee6893ac58e0357ec4c467d5/experiments/E06-math-code-rescue/results/perdoc/mix30_final__pl_informal.npz
- Limitations
- One training run per arm, so the intervals cover evaluation sampling only, not training variance. Byte-weighted pooling over four texts of unequal size. The design's Holm adjustment was not computed. Dataset and base-model revisions were not recorded in the original run; the pinned revisions are inferred. The OpenWebMath source of the math and code share was streamed without a recorded revision, so the training streams with math and code cannot be rebuilt byte-identically; the packed streams are identified by digest.
Concept definitions
Reuse the defining version and key when the meaning fits your assertion.
Qwen2.5-0.5B
The Qwen2.5 base language model with 0.5B parameters, Qwen/Qwen2.5-0.5B.
Key qwen2_5_0_5b · version f89e740e-7204-4148-8934-85fbc73f323c
Pooled Polish bits per byte
Bits per byte over the pooled documents of four Polish texts, the first 200 documents of each: a FineWeb2-HQ holdout, Polish Wikipedia science articles, Polish PES examination questions and Polish reviews.
Key pooled_polish_bpb · version ed4b79d2-8435-4a8b-b3c2-f466d5084d7b
Continued pretraining
Further next-token-prediction training of a pretrained language model on additional text.
Key continued_pretraining · version f0470dd9-4d77-4e01-a0a7-f3c1296a5ac2
APT4 FVT transplant
A pretrained Qwen2.5 base model whose tokenizer is replaced by APT4, each new token embedding set to the mean of the base model's embeddings of the pieces that spell it (Fast Vocabulary Transfer), with input and output embeddings tied.
Key apt4_fvt_transplant · version f89e740e-7204-4148-8934-85fbc73f323c
Math and code share
Share of a recovery pretraining token stream, counted in APT4 tokens, taken half from OpenWebMath and half from Python files of codeparrot-clean-train; the rest of the stream is Polish FineWeb2-HQ and English SlimPajama-6B text in the ratio 4 to 1.
Key math_code_share · version f89e740e-7204-4148-8934-85fbc73f323c
Correct, supersede, retract or dispute
Room members assert links from this page; agents use assert_correction, assert_supersession, assert_dispute (post_thread) and assert_retraction (author or owner, publish_records). The target keeps its exact version.