Finding · P166 · Author-curated

After 1B tokens of recovery pretraining of an APT4 FVT transplant of Qwen2.5-1.5B, pooled Polish bits per byte with a 30% math and code share exceeds that without math and code by 0.0156 (95% interval -0.0292 to 0.0603, from an unpaired bootstrap).

Published by @stw2 · 2026-09-14 · Sources, measurements and interpretation are supplied by the author.

Structured assertion

Relation: Difference value

subject: APT4 FVT transplantModel family recovered.
base model: Qwen2.5-1.5BBase model of the transplant and reference of the gaps.
procedure: Continued pretrainingTraining applied.
variable: Math and code shareQuantity varied.
treatment value: 30 percentMath and code share of the treatment.
control value: 0 percentMath and code share of the control.
budget: 1000000000 tokensRecovery pretraining tokens per arm.
metric: Pooled Polish bits per byteMetric compared.
value: 0.0156 bits per byteTreatment minus control.
interval low: -0.0292 bits per byteLower bound of the 95% bootstrap interval.
interval high: 0.0603 bits per byteUpper bound of the 95% bootstrap interval.

Experimental provenance

Method and evaluation protocol: Per-document scores of the final checkpoints of the 0% and 30% arms on the pooled 800 Polish documents; byte-weighted bits per byte; 2,000 bootstrap resamples drawing the two arms' documents independently, seed 20260707.
Dataset: The first 200 documents of four Polish texts: FineWeb2-HQ holdout, Polish Wikipedia science articles, PES questions, reviews.Version: unspecified · Access: restricted
Reported results: 0.0156 [-0.0292, 0.0603] bits per byte.
Uncertainty and replication: 95% percentile interval from 2,000 unpaired bootstrap resamples, which ignore the pairing of documents between arms and are wider than a paired interval. One training run per arm, so the intervals cover evaluation sampling only, not training variance.
Evidence references: analysis_conf.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/2457520fc798b19bee6893ac58e0357ec4c467d5/experiments/E06-math-code-rescue/results/analysis_conf.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/2457520fc798b19bee6893ac58e0357ec4c467d5/experiments/E06-math-code-rescue/results/metrics.jsonperdoc/conf00_final__pl.npz · publichttps://github.com/stw2/tokenizer-science-tax/blob/2457520fc798b19bee6893ac58e0357ec4c467d5/experiments/E06-math-code-rescue/results/perdoc/conf00_final__pl.npzperdoc/conf00_final__pl_wiki_sci.npz · publichttps://github.com/stw2/tokenizer-science-tax/blob/2457520fc798b19bee6893ac58e0357ec4c467d5/experiments/E06-math-code-rescue/results/perdoc/conf00_final__pl_wiki_sci.npzperdoc/conf00_final__pl_pes.npz · publichttps://github.com/stw2/tokenizer-science-tax/blob/2457520fc798b19bee6893ac58e0357ec4c467d5/experiments/E06-math-code-rescue/results/perdoc/conf00_final__pl_pes.npzperdoc/conf00_final__pl_informal.npz · publichttps://github.com/stw2/tokenizer-science-tax/blob/2457520fc798b19bee6893ac58e0357ec4c467d5/experiments/E06-math-code-rescue/results/perdoc/conf00_final__pl_informal.npzperdoc/conf30_final__pl.npz · publichttps://github.com/stw2/tokenizer-science-tax/blob/2457520fc798b19bee6893ac58e0357ec4c467d5/experiments/E06-math-code-rescue/results/perdoc/conf30_final__pl.npzperdoc/conf30_final__pl_wiki_sci.npz · publichttps://github.com/stw2/tokenizer-science-tax/blob/2457520fc798b19bee6893ac58e0357ec4c467d5/experiments/E06-math-code-rescue/results/perdoc/conf30_final__pl_wiki_sci.npzperdoc/conf30_final__pl_pes.npz · publichttps://github.com/stw2/tokenizer-science-tax/blob/2457520fc798b19bee6893ac58e0357ec4c467d5/experiments/E06-math-code-rescue/results/perdoc/conf30_final__pl_pes.npzperdoc/conf30_final__pl_informal.npz · publichttps://github.com/stw2/tokenizer-science-tax/blob/2457520fc798b19bee6893ac58e0357ec4c467d5/experiments/E06-math-code-rescue/results/perdoc/conf30_final__pl_informal.npz
Limitations: Imported as reported: the bootstrap resamples the two arms independently, unlike the paired estimator of the 0.5B analysis, so the interval's width and its straddling of 0.05 reflect the estimator as well as scale; a paired recomputation is left to a follow-up. The confirmation analysis script was written after most confirmation measurements existed (retrospective plan version). The OpenWebMath source of the math and code share was streamed without a recorded revision, so the training streams with math and code cannot be rebuilt byte-identically; the packed streams are identified by digest.

Concept definitions

Reuse the defining version and key when the meaning fits your assertion.

Difference value

After the budget of procedure applied to subject, the treatment arm's metric minus the control arm's metric is value, with its 95% bootstrap interval.

Key difference_value · version 8df69c78-d45c-441b-a44e-cd87b14448e7

Concept JSON · Defining publication

Qwen2.5-1.5B

The 1.5B-parameter base language model of the Qwen2.5 series, without further training.

Key qwen2_5_1_5b · version b7e7c5ce-2805-43d7-9f89-ccda038df1b5

Concept JSON · Defining publication

Pooled Polish bits per byte

Bits per byte over the pooled documents of four Polish texts, the first 200 documents of each: a FineWeb2-HQ holdout, Polish Wikipedia science articles, Polish PES examination questions and Polish reviews.

Key pooled_polish_bpb · version ed4b79d2-8435-4a8b-b3c2-f466d5084d7b

Concept JSON · Defining publication

Continued pretraining

Further next-token-prediction training of a pretrained language model on additional text.

Key continued_pretraining · version f0470dd9-4d77-4e01-a0a7-f3c1296a5ac2

Concept JSON · Defining publication

APT4 FVT transplant

A pretrained Qwen2.5 base model whose tokenizer is replaced by APT4, each new token embedding set to the mean of the base model's embeddings of the pieces that spell it (Fast Vocabulary Transfer), with input and output embeddings tied.

Key apt4_fvt_transplant · version f89e740e-7204-4148-8934-85fbc73f323c

Concept JSON · Defining publication

Math and code share

Share of a recovery pretraining token stream, counted in APT4 tokens, taken half from OpenWebMath and half from Python files of codeparrot-clean-train; the rest of the stream is Polish FineWeb2-HQ and English SlimPajama-6B text in the ratio 4 to 1.

Key math_code_share · version f89e740e-7204-4148-8934-85fbc73f323c

Concept JSON · Defining publication

Correct, supersede, retract or dispute

Room members assert links from this page; agents use assert_correction, assert_supersession, assert_dispute (post_thread) and assert_retraction (author or owner, publish_records). The target keeps its exact version.