Stage 1 re-score after the tokenizer-load erratum

https://github.com/stw2/tokenizer-science-tax @ 6c8a461e502d64cdecbb61d981a99dac9addbb67

Reference checked 2026-09-14 13:12 UTC. No code was executed or scientific result verified.

• experiments/E07-embedding-init/scripts/03_score_bpb.py
• experiments/E07-embedding-init/DESIGN.md

Prediction

Unchanged from the first plan version; only measured values and verdicts may move.

Protocol

03, with every tokenizer load gated on a whitespace round-trip canary and the pristine APT4 tokenizer used when a transplant directory loads lossily, re-scores the base model and the three unchanged transplants into a fresh row file; the unchanged 04 recomputes endpoints, bins and gates; 05 writes flat headline values. The auxiliary embeddings and the three builds stand.

Dataset

Qwen2.5-1.5B and APT4; the first 200 documents of Polish web text, English web text, Polish reviews, Polish Wikipedia science articles, Polish PES examination questions, English arXiv abstracts, English LaTeX method sections, English Python code and synthetic arithmetic problems, plus GSM8K problems as a contamination reference; the first 200 MB of Polish FineWeb2-HQ training text for the FOCUS auxiliary embeddings.

Split

Evaluation holdouts and frozen corpora; no transplant is trained in stage 1. GSM8K problems are excluded from every endpoint because the base model has memorised them.

Access needs

The APT4 tokenizer is gated on Hugging Face. The transplant models, the auxiliary corpus, the Polish training text, the English web holdout and four corpora are restricted materials or are rebuilt by script at the pinned revisions.

Configurations

As the first version. Canary: the text 'Oblicz: 37 + 43', a newline and '80' must survive encode and decode; the fallback requires the directory's vocabulary to equal the pristine APT4 tokenizer.json (sha256 536b4dff).

Metric

Bits per byte per domain; init damage, init-quality spread and FVT−FOCUS gap per domain, the last two with 95% paired document bootstrap intervals of 1000 resamples.

Seeds

20260705 for the random initialisation, the fastText training and the bootstrap.

Interpretation rule

H1 passes if the formal-to-prose ratio of mean init damage exceeds 1.15 for both random and FVT initialisation; H2 if FOCUS at most FVT at most random holds on at least 7 of 9 domains; H3 if the formal-to-prose ratio of mean init-quality spread exceeds 1.25; H4 if the mean FVT−FOCUS gap over formal domains exceeds its mean over prose domains. If H1 and H3 both fail, the outcome is a null. Gates: the base model reproduces the full-parameter experiment's time-zero base row within 0.005 bits per byte on 9 domains; the rebuilt FVT transplant's sanity loss is about 10.3; each initialisation rebuilds byte-identically; FVT and FOCUS are equal on overlap rows; vocabulary 32,000 with tied embeddings. Acceptance of the re-score: the re-scored base row equals the earlier base row, the re-scored FVT row equals the independently built FVT transplant's time-zero row on every shared domain, and G1 still passes.

Resources

Apple silicon, 128 GB unified memory: MPS, about one hour of scoring.

Prior work

arXiv:2604.10799v1 chooses FOCUS citing aggregate results on Bielik 1.5B v3 and reports no per-domain comparison of initialisations.