On math_clean arithmetic statements, the bits per byte of Qwen2.5-1.5B with APT4 by Fast Vocabulary Transfer exceed those of Qwen2.5-1.5B with its own tokenizer by 0.3843, 0.3264, 0.2837, 0.2506 and 0.2515 after 100M, 200M, 300M, 400M and 500M tokens of the same continued pretraining.

Relation: Gap trajectory

metric

Bits per byteQuantity whose difference is taken.

subject

Qwen2.5-1.5B with APT4 by Fast Vocabulary Transfer, continuedModel whose metric is reduced by the comparator's.

comparator

Qwen2.5-1.5B continued with its own tokenizerModel whose metric is subtracted.

evaluation text

math_clean statementsText scored.

language

EnglishLanguage of the text.

milestones

100,139,008; 200,278,016; 300,417,024; 400,031,744; 500,170,752 tokensTraining tokens at which the gaps are measured.

gaps

+0.3843; +0.3264; +0.2837; +0.2506; +0.2515Subject minus comparator at each milestone, in bits per byte.

setting

the same document sequence, constant learning rate before any decay anneal, one run per armTraining both models received.

Method and evaluation protocol

Next-token negative log-likelihood in bits summed over non-overlapping 2048-token windows of each document, the first token of each window unscored, divided by the documents' UTF-8 bytes; bf16 on Apple silicon, 128 GB unified memory; arm-B tokenizers loaded through a whitespace canary and the frozen APT4 reference.

Dataset

The first 200 of 800 math_clean statements; 800 for the untouched model and arm A at 100M.Version: unspecified · Access: public

Reported results

Gaps +0.3843, +0.3264, +0.2837, +0.2506, +0.2515; successive ratios 0.8494, 0.8693, 0.8833, 1.0034.

Uncertainty and replication

One run per arm and one scoring pass per milestone; no interval. Five points do not distinguish decay toward zero from decay toward a positive residual.

Evidence references

bpb_armA.jsonl · publichttps://github.com/stw2/tokenizer-science-tax/blob/9a57aa40f4e774136f51b6730c33266d59094a45/experiments/E05-control-arm/results/bpb_armA.jsonlbpb_armB.jsonl · publichttps://github.com/stw2/tokenizer-science-tax/blob/9a57aa40f4e774136f51b6730c33266d59094a45/experiments/E05-control-arm/results/bpb_armB.jsonlanalysis.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/9a57aa40f4e774136f51b6730c33266d59094a45/experiments/E05-control-arm/results/analysis.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/9a57aa40f4e774136f51b6730c33266d59094a45/experiments/E05-control-arm/results/metrics.json

Limitations

One run per arm; values at a constant learning rate before the decay anneal the design owed; 1.5B parameters and 0.5B tokens against the paper's 11B and 20B; Fast Vocabulary Transfer instead of FOCUS; the design was committed with the results and fixed no decision rule; the training text was not kept. The math_clean values of the untouched model and of arm A at 100M are over 800 statements and all others over the first 200; the rows were not re-scored. The extension to 1B tokens that was to decide between a permanent residual and slow convergence was never launched.