At the first checkpoint, after 100,139,008 tokens of recovery pretraining of an APT4 FVT transplant of Qwen2.5-0.5B, bits per byte is 2.0426, 1.9824, 1.9466 on math_clean and 0.8468, 0.6936, 0.6641 on English Python code for the 0%, 10% and 30% math and code shares.

Relation: Values by share on two texts

subject

APT4 FVT transplantModel family recovered.

budget

100139008 tokensTraining tokens at the checkpoint.

metric

Bits per byteMetric.

text 1

math_clean statementsEvaluation text math_clean.

text 1 00

2.0426 bits per byteArm with 0% math and code on math_clean.

text 1 10

1.9824 bits per byteArm with 10% math and code on math_clean.

text 1 30

1.9466 bits per byteArm with 30% math and code on math_clean.

text 2

English Python codeEvaluation text English Python code.

text 2 00

0.8468 bits per byteArm with 0% math and code on English Python code.

text 2 10

0.6936 bits per byteArm with 10% math and code on English Python code.

text 2 30

0.6641 bits per byteArm with 30% math and code on English Python code.

Method and evaluation protocol

Bits per byte of each arm's first checkpoint row on the first 200 documents of math_clean and Python code.

Dataset

The first 200 documents of ten evaluation texts: Polish and English web holdouts, English arXiv abstracts, LaTeX method sections, Python code and GSM8K problems, Polish Wikipedia science articles, PES questions and reviews, math_clean.Version: unspecified · Access: restricted

Reported results

math_clean: 2.0426, 1.9824, 1.9466; sci_python: 0.8468, 0.6936, 0.6641.

Uncertainty and replication

Point values without intervals. One training run per arm, so the intervals cover evaluation sampling only, not training variance.

Evidence references

bpb_mix00.jsonl · publichttps://github.com/stw2/tokenizer-science-tax/blob/2457520fc798b19bee6893ac58e0357ec4c467d5/experiments/E06-math-code-rescue/results/bpb_mix00.jsonlbpb_mix10.jsonl · publichttps://github.com/stw2/tokenizer-science-tax/blob/2457520fc798b19bee6893ac58e0357ec4c467d5/experiments/E06-math-code-rescue/results/bpb_mix10.jsonlbpb_mix30.jsonl · publichttps://github.com/stw2/tokenizer-science-tax/blob/2457520fc798b19bee6893ac58e0357ec4c467d5/experiments/E06-math-code-rescue/results/bpb_mix30.jsonlmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/2457520fc798b19bee6893ac58e0357ec4c467d5/experiments/E06-math-code-rescue/results/metrics.json

Limitations

One training run per arm, so the intervals cover evaluation sampling only, not training variance. The OpenWebMath source of the math and code share was streamed without a recorded revision, so the training streams with math and code cannot be rebuilt byte-identically; the packed streams are identified by digest.