Bits per byte aggregated from per-document scores of the final checkpoints of three recovery pretraining runs of an APT4 FVT transplant of Qwen2.5-0.5B differs from their batched checkpoint scores by at most 0.0015 on math_clean and the Polish holdout.

Relation: Scorer difference at most

subject

APT4 FVT transplantModels scored.

metric

Bits per byteMetric.

scorer 1

per-document scorer, one window per forward passFirst scorer.

scorer 2

checkpoint scorer, windows padded into batches of 8Second scorer.

texts

math_clean; Polish FineWeb2-HQ holdoutTexts compared.

value

0.0015 bits per byteLargest absolute difference.

band

0.005 bits per byteTolerance set before measurement.

Method and evaluation protocol

Absolute differences between aggregated per-document bits per byte and the 1B-token checkpoint rows, per arm and text.

Dataset

The first 200 documents of ten evaluation texts: Polish and English web holdouts, English arXiv abstracts, LaTeX method sections, Python code and GSM8K problems, Polish Wikipedia science articles, PES questions and reviews, math_clean.Version: unspecified · Access: restricted

Reported results

largest difference 0.001506 (30% arm, math_clean); tolerance 0.005.

Uncertainty and replication

Deterministic scoring; no interval.

Evidence references

analysis.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/2457520fc798b19bee6893ac58e0357ec4c467d5/experiments/E06-math-code-rescue/results/analysis.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/2457520fc798b19bee6893ac58e0357ec4c467d5/experiments/E06-math-code-rescue/results/metrics.json

Limitations

Checked on two texts only; the differences come from bf16 numerics of batched and unbatched forward passes. The OpenWebMath source of the math and code share was streamed without a recorded revision, so the training streams with math and code cannot be rebuilt byte-identically; the packed streams are identified by digest.