On the Polish FineWeb2-HQ holdout after 0.5B tokens, the continued-pretraining share of the bits-per-byte movement is 0.9482.

Relation: Metric value

metric

Continued-pretraining share of the movementQuantity computed.

value

0.9482 ratioValue of the metric.

cpt effect

0.2698 bits per byteUntouched model minus the original-tokenizer arm, in bits per byte.

transplant residual

0.0147 bits per byteTransplant arm minus the original-tokenizer arm, in bits per byte.

evaluation text

Polish FineWeb2-HQ holdoutText scored.

language

PolishLanguage of the text.

setting

after 500,170,752 Qwen tokens of the same document sequence, constant learning rate before any decay anneal, one run per armTraining both arms received.

Method and evaluation protocol

Next-token negative log-likelihood in bits summed over non-overlapping 2048-token windows of each document, the first token of each window unscored, divided by the documents' UTF-8 bytes; bf16 on Apple silicon, 128 GB unified memory; arm-B tokenizers loaded through a whitespace canary and the frozen APT4 reference. The share is computed in 09_analyze.py from the untouched, arm-A and arm-B values.

Dataset

The first 200 of 2,000 held-out FineWeb2-HQ pol_Latn documents.Version: unspecified · Access: public

Reported results

cpt_share_of_pl_gain 0.9481685040854919; continued-pretraining effect -0.2698; gap +0.0147.

Uncertainty and replication

Point value from single runs; no interval.

Evidence references

analysis.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/9a57aa40f4e774136f51b6730c33266d59094a45/experiments/E05-control-arm/results/analysis.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/9a57aa40f4e774136f51b6730c33266d59094a45/experiments/E05-control-arm/results/metrics.json

Limitations

One run per arm; values at a constant learning rate before the decay anneal the design owed; 1.5B parameters and 0.5B tokens against the paper's 11B and 20B; Fast Vocabulary Transfer instead of FOCUS; the design was committed with the results and fixed no decision rule; the training text was not kept.