Continued pretraining of Qwen2.5-1.5B with its own tokenizer on 0.5B tokens raises bits per byte on GSM8K problems from 0.0524 to 0.4966.

Relation: Metric comparison

metric

Bits per byteQuantity compared.

subject

Qwen2.5-1.5B continued with its own tokenizerModel whose value is subject_value.

subject value

0.4966 bits per byteBits per byte of the subject.

comparator

Qwen2.5-1.5BModel whose value is comparator_value.

comparator value

0.0524 bits per byteBits per byte of the comparator.

evaluation text

GSM8K problemsText scored.

language

EnglishLanguage of the text.

setting

500,170,752 Qwen tokens of continued pretraining at a constant learning rate before any decay anneal, one runTraining of the subject.

scope

first 200 documents of the corpusDocuments scored.

Method and evaluation protocol

Next-token negative log-likelihood in bits summed over non-overlapping 2048-token windows of each document, the first token of each window unscored, divided by the documents' UTF-8 bytes; bf16 on Apple silicon, 128 GB unified memory; arm-B tokenizers loaded through a whitespace canary and the frozen APT4 reference.

Dataset

The first 200 documents of E1's GSM8K problems corpus.Version: unspecified · Access: public

Reported results

Arm A 0.49659928844408574, untouched 0.05236162490935402; continued-pretraining effect (arm A minus untouched) +0.4442.

Uncertainty and replication

One scoring pass per model; no interval. The untouched model's row was scored twice with identical values.

Evidence references

bpb_armA.jsonl · publichttps://github.com/stw2/tokenizer-science-tax/blob/9a57aa40f4e774136f51b6730c33266d59094a45/experiments/E05-control-arm/results/bpb_armA.jsonlanalysis.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/9a57aa40f4e774136f51b6730c33266d59094a45/experiments/E05-control-arm/results/analysis.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/9a57aa40f4e774136f51b6730c33266d59094a45/experiments/E05-control-arm/results/metrics.json

Limitations

One run per arm; values at a constant learning rate before the decay anneal the design owed; 1.5B parameters and 0.5B tokens against the paper's 11B and 20B; Fast Vocabulary Transfer instead of FOCUS; the design was committed with the results and fixed no decision rule; the training text was not kept. GSM8K is likely in Qwen2.5's pretraining data; the design treats this text as a contamination-decay reference and excludes it from inference.