After 0.5B tokens of the same continued pretraining, bits per byte on Polish Wikipedia science articles is 0.9116 for Qwen2.5-1.5B with APT4 by Fast Vocabulary Transfer and 0.8782 for Qwen2.5-1.5B with its own tokenizer.

Relation: Metric comparison

metric

Bits per byteQuantity compared.

subject

Qwen2.5-1.5B with APT4 by Fast Vocabulary Transfer, continuedModel whose value is subject_value.

subject value

0.9116 bits per byteBits per byte of the subject.

comparator

Qwen2.5-1.5B continued with its own tokenizerModel whose value is comparator_value.

comparator value

0.8782 bits per byteBits per byte of the comparator.

evaluation text

Polish Wikipedia science articlesText scored.

language

PolishLanguage of the text.

setting

after 500,170,752 Qwen tokens of the same document sequence, constant learning rate before any decay anneal, one run per armTraining both models received.

scope

first 200 documents of the corpusDocuments scored.

Method and evaluation protocol

Next-token negative log-likelihood in bits summed over non-overlapping 2048-token windows of each document, the first token of each window unscored, divided by the documents' UTF-8 bytes; bf16 on Apple silicon, 128 GB unified memory; arm-B tokenizers loaded through a whitespace canary and the frozen APT4 reference.

Dataset

The first 200 documents of E1's Polish Wikipedia science corpus.Version: unspecified · Access: public

Reported results

Arm B 0.9116095430921759, arm A 0.8781854339415027; gap (arm B minus arm A) +0.0334; continued-pretraining effect on the same text (arm A minus untouched) -0.2148.

Uncertainty and replication

One scoring pass per model; no interval.

Evidence references

bpb_armA.jsonl · publichttps://github.com/stw2/tokenizer-science-tax/blob/9a57aa40f4e774136f51b6730c33266d59094a45/experiments/E05-control-arm/results/bpb_armA.jsonlbpb_armB.jsonl · publichttps://github.com/stw2/tokenizer-science-tax/blob/9a57aa40f4e774136f51b6730c33266d59094a45/experiments/E05-control-arm/results/bpb_armB.jsonlanalysis.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/9a57aa40f4e774136f51b6730c33266d59094a45/experiments/E05-control-arm/results/analysis.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/9a57aa40f4e774136f51b6730c33266d59094a45/experiments/E05-control-arm/results/metrics.json

Limitations

One run per arm; values at a constant learning rate before the decay anneal the design owed; 1.5B parameters and 0.5B tokens against the paper's 11B and 20B; Fast Vocabulary Transfer instead of FOCUS; the design was committed with the results and fixed no decision rule; the training text was not kept.