The fall in bits per byte of the FVT-initialised APT4 transplant of Qwen2.5-1.5B after 100M tokens of embeddings-only training is between 0.707 (Polish PES examination questions) and 0.92 (English Python code) of its fall after 100M tokens of full-parameter training on the same token stream, over 9 domains.

Relation: Range over cells

quantity

fall in bits per byte from time zero after 100M tokens of embeddings-only training divided by the fall after 100M tokens of full-parameter training of the same FVT transplant on the same token streamQuantity ranged over.

model

APT4 transplant of Qwen2.5-1.5BModels measured.

cells

the nine primary domainsCells the range is taken over.

count cells

9 domainsCells.

min value

0.707 ratioSmallest value.

min cell

Polish PES examination questionsDomain with the smallest value.

max value

0.92 ratioLargest value.

max cell

English Python codeDomain with the largest value.

Method and evaluation protocol

Numerator: time-zero FVT row minus the 100M-token embeddings-only row. Denominator: the full-parameter experiment's time-zero FVT row minus its 100M-token row. Divided per domain.

Dataset

Bits-per-byte rows on the first 200 documents of the nine primary domains, from this experiment and from full-parameter continued pretraining of the same FVT transplant.Version: unspecified · Access: restricted

Reported results

sci_latex: 1.647 / 1.9168 = 0.859; sci_python: 2.1181 / 2.3017 = 0.92; math_clean: 0.7206 / 0.8726 = 0.826; sci_arxiv: 1.2647 / 1.5409 = 0.821; pl_wiki_sci: 1.6334 / 2.0411 = 0.8; pl_pes: 1.087 / 1.5369 = 0.707; pl: 1.4155 / 1.8038 = 0.785; en: 1.2675 / 1.5542 = 0.816; pl_informal: 0.9312 / 1.2172 = 0.765

Uncertainty and replication

Point estimates; the full-parameter rows have no per-document values, so no interval.

Evidence references

analysis_s2.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/7d450d1f62209ff5ac5415543443d2a25db74b7e/experiments/E07-embedding-init/results/analysis_s2.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/7d450d1f62209ff5ac5415543443d2a25db74b7e/experiments/E07-embedding-init/results/metrics.json

Limitations

The denominator comes from another experiment's scored rows (bpb_armB.jsonl). Descriptive endpoint without bins. One training run per initialisation at 1.5B. The trainer's loss and gradient-norm logs, the smoke-run gate results, the transfer digests and the token stream's digest were not kept. Windows of 2,048 tokens.