The embeddings-only training tokens after which the bits per byte of APT4 transplants of Qwen2.5-1.5B has fallen by 90% of its fall at 150M tokens range from 64,879,649 (random initialisation, English Python code) to 109,334,957 (FOCUS initialisation, math_clean statements) over 27 initialisation and domain cells.

Relation: Range over cells

quantity

tokens of embeddings-only training until bits per byte has fallen by 90% of its fall at 150M tokens, interpolated linearly between checkpointsQuantity ranged over.

model

APT4 transplant of Qwen2.5-1.5BModels measured.

cells

random, FVT and FOCUS initialisation on the nine primary domainsCells the range is taken over.

count cells

27 cellsCells.

min value

64879649 tokensSmallest value.

min cell

random initialisation, English Python codeCell with the smallest value.

max value

109334957 tokensLargest value.

max cell

FOCUS initialisation, math_clean statementsCell with the largest value.

Method and evaluation protocol

Per arm and domain, the token count at which bits per byte first reaches its time-zero value minus 90% of its 150M-token fall, interpolated between checkpoints.

Dataset

First 200 documents of the evaluation domains, scored at the 150M-token checkpoints of embeddings-only training on the packed APT4 stream.Version: unspecified · Access: restricted

Reported results

fvt: 76598349 to 104178306; focus: 77420582 to 109334957; random: 64879649 to 89027515

Uncertainty and replication

Point values from six checkpoints; no interval.

Evidence references

analysis_s2.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/7d450d1f62209ff5ac5415543443d2a25db74b7e/experiments/E07-embedding-init/results/analysis_s2.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/7d450d1f62209ff5ac5415543443d2a25db74b7e/experiments/E07-embedding-init/results/metrics.json

Limitations

Descriptive endpoint without bins. One training run per initialisation at 1.5B. The trainer's loss and gradient-norm logs, the smoke-run gate results, the transfer digests and the token stream's digest were not kept. Windows of 2,048 tokens.