After 0.5B tokens of the same continued pretraining, bits per byte on math_clean arithmetic statements is 1.7672 for Qwen2.5-1.5B with APT4 by Fast Vocabulary Transfer and 1.5157 for Qwen2.5-1.5B with its own tokenizer.

Relation: Metric comparison

metric

Bits per byteQuantity compared.

subject

Qwen2.5-1.5B with APT4 by Fast Vocabulary Transfer, continuedModel whose value is subject_value.

subject value

1.7672 bits per byteBits per byte of the subject.

comparator

Qwen2.5-1.5B continued with its own tokenizerModel whose value is comparator_value.

comparator value

1.5157 bits per byteBits per byte of the comparator.

evaluation text

math_clean statementsText scored.

language

EnglishLanguage of the text.

setting

after 500,170,752 Qwen tokens of the same document sequence, constant learning rate before any decay anneal, one run per armTraining both models received.

Method and evaluation protocol

Next-token negative log-likelihood in bits summed over non-overlapping 2048-token windows of each document, the first token of each window unscored, divided by the documents' UTF-8 bytes; bf16 on Apple silicon, 128 GB unified memory; arm-B tokenizers loaded through a whitespace canary and the frozen APT4 reference.

Dataset

The first 200 of 800 math_clean statements; 800 for the untouched model and arm A at 100M.Version: unspecified · Access: public

Reported results

Arm B 1.7671920928889029, arm A 1.5157282441289601; gap (arm B minus arm A) +0.2515; continued-pretraining effect on the same text (arm A minus untouched) +0.0135.

Uncertainty and replication

One scoring pass per model; no interval.

Evidence references

bpb_armA.jsonl · publichttps://github.com/stw2/tokenizer-science-tax/blob/9a57aa40f4e774136f51b6730c33266d59094a45/experiments/E05-control-arm/results/bpb_armA.jsonlbpb_armB.jsonl · publichttps://github.com/stw2/tokenizer-science-tax/blob/9a57aa40f4e774136f51b6730c33266d59094a45/experiments/E05-control-arm/results/bpb_armB.jsonlanalysis.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/9a57aa40f4e774136f51b6730c33266d59094a45/experiments/E05-control-arm/results/analysis.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/9a57aa40f4e774136f51b6730c33266d59094a45/experiments/E05-control-arm/results/metrics.json

Limitations

One run per arm; values at a constant learning rate before the decay anneal the design owed; 1.5B parameters and 0.5B tokens against the paper's 11B and 20B; Fast Vocabulary Transfer instead of FOCUS; the design was committed with the results and fixed no decision rule; the training text was not kept. The math_clean values of the untouched model and of arm A at 100M are over 800 statements and all others over the first 200; the rows were not re-scored.