On English arXiv abstracts, the bits per byte of Qwen2.5-1.5B with APT4 by Fast Vocabulary Transfer exceed those of Qwen2.5-1.5B with its own tokenizer by 0.2271, 0.1847, 0.1627, 0.1485 and 0.1395 after 100M, 200M, 300M, 400M and 500M tokens of the same continued pretraining.

Relation: Gap trajectory

metric

Bits per byteQuantity whose difference is taken.

subject

Qwen2.5-1.5B with APT4 by Fast Vocabulary Transfer, continuedModel whose metric is reduced by the comparator's.

comparator

Qwen2.5-1.5B continued with its own tokenizerModel whose metric is subtracted.

evaluation text

English arXiv abstractsText scored.

language

EnglishLanguage of the text.

milestones

100,139,008; 200,278,016; 300,417,024; 400,031,744; 500,170,752 tokensTraining tokens at which the gaps are measured.

gaps

+0.2271; +0.1847; +0.1627; +0.1485; +0.1395Subject minus comparator at each milestone, in bits per byte.

setting

the same document sequence, constant learning rate before any decay anneal, one run per armTraining both models received.

Method and evaluation protocol

Next-token negative log-likelihood in bits summed over non-overlapping 2048-token windows of each document, the first token of each window unscored, divided by the documents' UTF-8 bytes; bf16 on Apple silicon, 128 GB unified memory; arm-B tokenizers loaded through a whitespace canary and the frozen APT4 reference.

Dataset

The first 200 documents of E1's arXiv abstracts corpus.Version: unspecified · Access: public

Reported results

Gaps +0.2271, +0.1847, +0.1627, +0.1485, +0.1395; successive ratios 0.8130, 0.8808, 0.9130, 0.9391.

Uncertainty and replication

One run per arm and one scoring pass per milestone; no interval. Five points do not distinguish decay toward zero from decay toward a positive residual.

Evidence references

bpb_armA.jsonl · publichttps://github.com/stw2/tokenizer-science-tax/blob/9a57aa40f4e774136f51b6730c33266d59094a45/experiments/E05-control-arm/results/bpb_armA.jsonlbpb_armB.jsonl · publichttps://github.com/stw2/tokenizer-science-tax/blob/9a57aa40f4e774136f51b6730c33266d59094a45/experiments/E05-control-arm/results/bpb_armB.jsonlanalysis.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/9a57aa40f4e774136f51b6730c33266d59094a45/experiments/E05-control-arm/results/analysis.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/9a57aa40f4e774136f51b6730c33266d59094a45/experiments/E05-control-arm/results/metrics.json

Limitations

One run per arm; values at a constant learning rate before the decay anneal the design owed; 1.5B parameters and 0.5B tokens against the paper's 11B and 20B; Fast Vocabulary Transfer instead of FOCUS; the design was committed with the results and fixed no decision rule; the training text was not kept. The extension to 1B tokens that was to decide between a permanent residual and slow convergence was never launched.