After 1B tokens of recovery pretraining of an APT4 FVT transplant without math and code, the gap to the base model on math_clean bits per byte is 0.2613 with Qwen2.5-1.5B and 0.3305 with Qwen2.5-0.5B.

Relation: Values at two scales

subject

APT4 FVT transplantModel family recovered.

budget

1000000000 tokensRecovery pretraining tokens per arm.

quantity

gap to the base model after recovery pretraining without math and code, positive meaning worseQuantity compared.

metric

Bits per byteMetric.

evaluation text

math_clean statementsText scored.

larger model

Qwen2.5-1.5BLarger base model.

larger value

0.2613 bits per byteQuantity with the larger base model.

smaller model

Qwen2.5-0.5BSmaller base model.

smaller value

0.3305 bits per byteQuantity with the smaller base model.

Method and evaluation protocol

Values of the 0.5B and 1.5B analyses.

Dataset

The first 200 items of math_clean, an 800-item synthetic arithmetic set.Version: sha256 60ef593cf8e00bc45123c4cc776c52b5acd358393612d330bbb9348c4937acbc · Access: public

Reported results

1.5B 0.2613 [0.2315, 0.2907]; 0.5B 0.3305 [0.3129, 0.3492].

Uncertainty and replication

The 95% intervals of the two scales do not overlap; no test of the difference was computed. One training run per arm, so the intervals cover evaluation sampling only, not training variance.

Evidence references

analysis.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/2457520fc798b19bee6893ac58e0357ec4c467d5/experiments/E06-math-code-rescue/results/analysis.jsonanalysis_conf.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/2457520fc798b19bee6893ac58e0357ec4c467d5/experiments/E06-math-code-rescue/results/analysis_conf.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/2457520fc798b19bee6893ac58e0357ec4c467d5/experiments/E06-math-code-rescue/results/metrics.json

Limitations

Two scales, one training run each; the 1.5B arms split the same tokens per step into micro-batch 8 and gradient accumulation 32. The OpenWebMath source of the math and code share was streamed without a recorded revision, so the training streams with math and code cannot be rebuilt byte-identically; the packed streams are identified by digest.