After 1B tokens of recovery pretraining of an APT4 FVT transplant without math and code, the gap to the base model on digit-probe accuracy is 0.1889 with Qwen2.5-1.5B and 0.2733 with Qwen2.5-0.5B.

Relation: Values at two scales

subject

APT4 FVT transplantModel family recovered.

budget

1000000000 tokensRecovery pretraining tokens per arm.

quantity

gap to the base model after recovery pretraining without math and code, positive meaning worseQuantity compared.

metric

Digit-probe accuracyMetric.

larger model

Qwen2.5-1.5BLarger base model.

larger value

0.1889 accuracyQuantity with the larger base model.

smaller model

Qwen2.5-0.5BSmaller base model.

smaller value

0.2733 accuracyQuantity with the smaller base model.

Method and evaluation protocol

Values of the 0.5B and 1.5B analyses.

Dataset

900 completions: 300 synthetic arithmetic items in three format cells, 4-shot, greedy; specification configs/probe_digits_spec.json.Version: sha256 e11c98fc129e4e4db6566140286c15a0fa6f88f7d7a4abdecad3a3b8f3ca53f8 · Access: public

Reported results

1.5B 0.1889 [0.1567, 0.2211]; 0.5B 0.2733 [0.2411, 0.3033].

Uncertainty and replication

The 95% intervals of the two scales do not overlap; no test of the difference was computed. One training run per arm, so the intervals cover evaluation sampling only, not training variance.

Evidence references

analysis.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/2457520fc798b19bee6893ac58e0357ec4c467d5/experiments/E06-math-code-rescue/results/analysis.jsonanalysis_conf.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/2457520fc798b19bee6893ac58e0357ec4c467d5/experiments/E06-math-code-rescue/results/analysis_conf.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/2457520fc798b19bee6893ac58e0357ec4c467d5/experiments/E06-math-code-rescue/results/metrics.json

Limitations

Two scales, one training run each; the 1.5B arms split the same tokens per step into micro-batch 8 and gradient accumulation 32. The OpenWebMath source of the math and code share was streamed without a recorded revision, so the training streams with math and code cannot be rebuilt byte-identically; the packed streams are identified by digest. The transplant's generation configuration names end-of-sequence id 4 while training separated documents with id 2; answers are cut at the first newline.