Continued pretraining of Qwen2.5-1.5B with its own tokenizer on 0.5B tokens lowers bits per byte on Polish Wikipedia science articles from 1.0930 to 0.8782.

Relation: Metric comparison

metric

Bits per byteQuantity compared.

subject

Qwen2.5-1.5B continued with its own tokenizerModel whose value is subject_value.

subject value

0.8782 bits per byteBits per byte of the subject.

comparator

Qwen2.5-1.5BModel whose value is comparator_value.

comparator value

1.0930 bits per byteBits per byte of the comparator.

evaluation text

Polish Wikipedia science articlesText scored.

language

PolishLanguage of the text.

setting

500,170,752 Qwen tokens of continued pretraining at a constant learning rate before any decay anneal, one runTraining of the subject.

scope

first 200 documents of the corpusDocuments scored.

Method and evaluation protocol

Next-token negative log-likelihood in bits summed over non-overlapping 2048-token windows of each document, the first token of each window unscored, divided by the documents' UTF-8 bytes; bf16 on Apple silicon, 128 GB unified memory; arm-B tokenizers loaded through a whitespace canary and the frozen APT4 reference.

Dataset

The first 200 documents of E1's Polish Wikipedia science corpus.Version: unspecified · Access: public

Reported results

Arm A 0.8781854339415027, untouched 1.0929525011709564; continued-pretraining effect (arm A minus untouched) -0.2148.

Uncertainty and replication

One scoring pass per model; no interval. The untouched model's row was scored twice with identical values.

Evidence references

bpb_armA.jsonl · publichttps://github.com/stw2/tokenizer-science-tax/blob/9a57aa40f4e774136f51b6730c33266d59094a45/experiments/E05-control-arm/results/bpb_armA.jsonlanalysis.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/9a57aa40f4e774136f51b6730c33266d59094a45/experiments/E05-control-arm/results/analysis.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/9a57aa40f4e774136f51b6730c33266d59094a45/experiments/E05-control-arm/results/metrics.json

Limitations

One run per arm; values at a constant learning rate before the decay anneal the design owed; 1.5B parameters and 0.5B tokens against the paper's 11B and 20B; Fast Vocabulary Transfer instead of FOCUS; the design was committed with the results and fixed no decision rule; the training text was not kept.