At time zero, bits per byte of the FOCUS-initialised APT4 transplant of Qwen2.5-1.5B is at most that of the FVT-initialised transplant, which is at most that of the random-initialised transplant, on 9 of 9 domains.

Relation: Ordering holds on a count of domains

metric

Bits per byteQuantity compared.

model

APT4 transplant of Qwen2.5-1.5BModels measured.

first

FOCUSInitialisation with the lowest value where the ordering holds.

second

Frequency-based Vocabulary TransferInitialisation with the middle value.

third

Random embedding initialisationInitialisation with the highest value.

domains

the nine primary evaluation domainsDomains compared.

setting

time zero: no training after embedding initialisation; tokenizers loaded through a whitespace canary with the pristine APT4 tokenizer as fallbackTraining state and scoring condition.

count holding

9 domainsDomains on which the ordering holds.

count total

9 domainsDomains compared.

Method and evaluation protocol

Compare the bits per byte of the three transplants on each primary domain.

Dataset

First 200 documents of each of the nine primary evaluation domains: the Polish FineWeb2-HQ holdout, the English SlimPajama holdout, Polish reviews, Polish Wikipedia science articles, Polish PES examination questions, English arXiv abstracts, English LaTeX method sections, English Python code and math_clean statements.Version: unspecified · Access: restricted

Reported results

sci_latex: 2.7403 / 2.8089 / 7.4799; sci_python: 3.0649 / 3.1712 / 7.8283; math_clean: 2.7019 / 2.7159 / 10.6312; sci_arxiv: 2.4625 / 2.4753 / 6.1204; pl_wiki_sci: 2.7274 / 3.131 / 4.3701; pl_pes: 2.5278 / 2.8151 / 4.9149; pl: 2.543 / 2.9275 / 4.0875; en: 2.5683 / 2.6351 / 6.2473; pl_informal: 2.5347 / 2.8999 / 4.2017

Uncertainty and replication

Point values per domain; the FVT−FOCUS interval includes zero on two domains.

Evidence references

analysis.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/e7d4bb2df26254f88ecbf390a7cbf69ac210e0e3/experiments/E07-embedding-init/results/analysis.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/e7d4bb2df26254f88ecbf390a7cbf69ac210e0e3/experiments/E07-embedding-init/results/metrics.json

Limitations

The design text was first committed together with the stage-1 results. Init damage compares APT4 transplants with the base model under its own tokenizer at 2,048-token windows, which cover different byte spans, and fertility inflates absolute damage on English formal text. One build and one scoring run per initialisation at 1.5B.