After 1B tokens of recovery pretraining of an APT4 FVT transplant, the rescue fraction of a 30% math and code share on math_clean bits per byte is 0.5684 with Qwen2.5-1.5B as the base model and 0.4664 with Qwen2.5-0.5B.

Relation: Values at two scales

subject

APT4 FVT transplantModel family recovered.

budget

1000000000 tokensRecovery pretraining tokens per arm.

quantity

Rescue fractionQuantity compared.

treatment value

30 percentMath and code share of the treatment.

metric

Bits per byteMetric.

evaluation text

math_clean statementsText scored.

larger model

Qwen2.5-1.5BLarger base model.

larger value

0.5684 ratioQuantity with the larger base model.

smaller model

Qwen2.5-0.5BSmaller base model.

smaller value

0.4664 ratioQuantity with the smaller base model.

Method and evaluation protocol

Values of the 0.5B and 1.5B analyses.

Dataset

The first 200 items of math_clean, an 800-item synthetic arithmetic set.Version: sha256 60ef593cf8e00bc45123c4cc776c52b5acd358393612d330bbb9348c4937acbc · Access: public

Reported results

1.5B 0.5684 [0.4996, 0.6598]; 0.5B 0.4664 [0.4324, 0.4981].

Uncertainty and replication

The 95% intervals of the two scales do not overlap; no test of the difference was computed. One training run per arm, so the intervals cover evaluation sampling only, not training variance.

Evidence references

analysis.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/2457520fc798b19bee6893ac58e0357ec4c467d5/experiments/E06-math-code-rescue/results/analysis.jsonanalysis_conf.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/2457520fc798b19bee6893ac58e0357ec4c467d5/experiments/E06-math-code-rescue/results/analysis_conf.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/2457520fc798b19bee6893ac58e0357ec4c467d5/experiments/E06-math-code-rescue/results/metrics.json

Limitations

Two scales, one training run each; the 1.5B arms split the same tokens per step into micro-batch 8 and gradient accumulation 32. Neither scale reaches a rescue fraction of 0.8. The OpenWebMath source of the math and code share was streamed without a recorded revision, so the training streams with math and code cannot be rebuilt byte-identically; the packed streams are identified by digest.