At time zero, the formal-to-prose ratio of mean init damage of APT4 transplants of Qwen2.5-1.5B is 1.434 for random and 0.802 for FVT initialisation; only the random ratio exceeds 1.15.

Relation: Ratios against a threshold

metric

Init damageQuantity averaged over each group of domains.

model

APT4 transplant of Qwen2.5-1.5BModels measured.

subject domains

Formal domainsDomains whose mean is the numerator.

comparator domains

Prose domainsDomains whose mean is the denominator.

setting

time zero: no training after embedding initialisation; transplant rows scored with each model directory's tokenizer loaded by AutoTokenizerTraining state and scoring condition.

random ratio

1.434 ratioRatio for random initialisation.

fvt ratio

0.802 ratioRatio for FVT initialisation.

threshold

1.15 ratioValue both ratios are compared with.

both exceed threshold

falseWhether both ratios exceed the threshold.

Method and evaluation protocol

Init damage per domain is transplant bits per byte minus base bits per byte on the first 200 documents; means over the three formal and the six prose domains are divided, per initialisation.

Dataset

First 200 documents of each of the nine primary evaluation domains: the Polish FineWeb2-HQ holdout, the English SlimPajama holdout, Polish reviews, Polish Wikipedia science articles, Polish PES examination questions, English arXiv abstracts, English LaTeX method sections, English Python code and math_clean statements.Version: unspecified · Access: restricted

Reported results

Random 6.8381 / 4.7688 = 1.434; FVT 2.0111 / 2.5065 = 0.802; threshold 1.15 for both.

Uncertainty and replication

Point values; the design's bootstrap intervals on pooled means were not computed.

Evidence references

analysis.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/884186c96aedece6ef61da4e12e239ef5df929a5/experiments/E07-embedding-init/results/analysis.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/884186c96aedece6ef61da4e12e239ef5df929a5/experiments/E07-embedding-init/results/metrics.json

Limitations

The transplant rows were scored through a tokenizer load that deleted spaces and newlines, which inflated prose and deflated code bits per byte; the base row was unaffected. The design text was first committed together with the stage-1 results. Init damage compares APT4 transplants with the base model under its own tokenizer at 2,048-token windows, which cover different byte spans, and fertility inflates absolute damage on English formal text. One build and one scoring run per initialisation at 1.5B.