After 0.5B tokens of the same continued pretraining, bits per byte on the English SlimPajama holdout is 0.9868 for Qwen2.5-1.5B with APT4 by Fast Vocabulary Transfer and 0.8540 for Qwen2.5-1.5B with its own tokenizer.

Relation: Metric comparison

metric

Bits per byteQuantity compared.

subject

Qwen2.5-1.5B with APT4 by Fast Vocabulary Transfer, continuedModel whose value is subject_value.

subject value

0.9868 bits per byteBits per byte of the subject.

comparator

Qwen2.5-1.5B continued with its own tokenizerModel whose value is comparator_value.

comparator value

0.8540 bits per byteBits per byte of the comparator.

evaluation text

English SlimPajama holdoutText scored.

language

EnglishLanguage of the text.

setting

after 500,170,752 Qwen tokens of the same document sequence, constant learning rate before any decay anneal, one run per armTraining both models received.

Method and evaluation protocol

Next-token negative log-likelihood in bits summed over non-overlapping 2048-token windows of each document, the first token of each window unscored, divided by the documents' UTF-8 bytes; bf16 on Apple silicon, 128 GB unified memory; arm-B tokenizers loaded through a whitespace canary and the frozen APT4 reference.

Dataset

The first 200 of 2,000 held-out SlimPajama-6B documents.Version: unspecified · Access: restricted

Reported results

Arm B 0.9867610698285049, arm A 0.8540055295047299; gap (arm B minus arm A) +0.1328; continued-pretraining effect on the same text (arm A minus untouched) +0.0246.

Uncertainty and replication

One scoring pass per model; no interval.

Evidence references

bpb_armA.jsonl · publichttps://github.com/stw2/tokenizer-science-tax/blob/9a57aa40f4e774136f51b6730c33266d59094a45/experiments/E05-control-arm/results/bpb_armA.jsonlbpb_armB.jsonl · publichttps://github.com/stw2/tokenizer-science-tax/blob/9a57aa40f4e774136f51b6730c33266d59094a45/experiments/E05-control-arm/results/bpb_armB.jsonlanalysis.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/9a57aa40f4e774136f51b6730c33266d59094a45/experiments/E05-control-arm/results/analysis.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/9a57aa40f4e774136f51b6730c33266d59094a45/experiments/E05-control-arm/results/metrics.json

Limitations

One run per arm; values at a constant learning rate before the decay anneal the design owed; 1.5B parameters and 0.5B tokens against the paper's 11B and 20B; Fast Vocabulary Transfer instead of FOCUS; the design was committed with the results and fixed no decision rule; the training text was not kept.