Stage 1: bits per byte at time zero of random, FVT and FOCUS APT4 transplants of Qwen2.5-1.5B on ten domains

https://github.com/stw2/tokenizer-science-tax @ e54c7e02bb1ea89dbaff6911c87331d705602b1e

Reference checked 2026-09-14 13:12 UTC. No code was executed or scientific result verified.

• experiments/E07-embedding-init/scripts
• experiments/E07-embedding-init/data/aux
• experiments/E07-embedding-init/DESIGN.md

Prediction

Init damage is concentrated on formal domains for random and FVT initialisation; FOCUS at most FVT at most random holds on nearly every domain; the init-quality spread is concentrated on formal domains; the FVT−FOCUS gap is larger on formal than on prose domains.

Protocol

01 trains fastText auxiliary embeddings on the first 200 MB of Polish training text tokenized with APT4 and freezes the 32,000-row matrix; aux_sanity checks it; 02 builds three transplants that share the base model, APT4 and the handling of overlap, byte and special pieces and differ only in how new-token embeddings are initialised; 03 scores the base model and the three transplants in bits per byte on the first 200 documents of ten domains with 2,048-token windows, storing per-document values; 04 computes init damage, init-quality spread and FVT−FOCUS gap per domain with paired document bootstrap intervals, the four hypothesis bins and gates G1 and G4; 05 writes flat headline values.

Dataset

Qwen2.5-1.5B and APT4; the first 200 documents of Polish web text, English web text, Polish reviews, Polish Wikipedia science articles, Polish PES examination questions, English arXiv abstracts, English LaTeX method sections, English Python code and synthetic arithmetic problems, plus GSM8K problems as a contamination reference; the first 200 MB of Polish FineWeb2-HQ training text for the FOCUS auxiliary embeddings.

Split

Evaluation holdouts and frozen corpora; no transplant is trained in stage 1. GSM8K problems are excluded from every endpoint because the base model has memorised them.

Access needs

The APT4 tokenizer is gated on Hugging Face. The transplant models, the auxiliary corpus, the Polish training text, the English web holdout and four corpora are restricted materials or are rebuilt by script at the pinned revisions.

Configurations

Random: normal with the base embedding matrix's global mean and standard deviation. FVT: mean of the constituent Qwen rows. FOCUS: sparsemax-weighted overlap rows by cosine similarity of fastText vectors (skip-gram, 200 dimensions, character n-grams 3 to 6, 3 epochs, 4 workers). Overlap pieces copy their Qwen row in FVT and FOCUS. Scoring: 200 documents, 2,048-token windows, no special tokens, bf16.

Metric

Bits per byte per domain; init damage, init-quality spread and FVT−FOCUS gap per domain, the last two with 95% paired document bootstrap intervals of 1000 resamples.

Seeds

20260705 for the random initialisation, the fastText training and the bootstrap.

Interpretation rule

H1 passes if the formal-to-prose ratio of mean init damage exceeds 1.15 for both random and FVT initialisation; H2 if FOCUS at most FVT at most random holds on at least 7 of 9 domains; H3 if the formal-to-prose ratio of mean init-quality spread exceeds 1.25; H4 if the mean FVT−FOCUS gap over formal domains exceeds its mean over prose domains. If H1 and H3 both fail, the outcome is a null. Gates: the base model reproduces the full-parameter experiment's time-zero base row within 0.005 bits per byte on 9 domains; the rebuilt FVT transplant's sanity loss is about 10.3; each initialisation rebuilds byte-identically; FVT and FOCUS are equal on overlap rows; vocabulary 32,000 with tied embeddings.

Resources

Apple silicon, 128 GB unified memory: CPU for fastText and builds, MPS for scoring, about one hour of scoring; no training.

Prior work

arXiv:2604.10799v1 chooses FOCUS citing aggregate results on Bielik 1.5B v3 and reports no per-domain comparison of initialisations.