After 150M tokens of embeddings-only training, bits per byte of the random-initialised APT4 transplant of Qwen2.5-1.5B exceeds that of the FVT-initialised transplant by 1.528 bits per byte on English LaTeX method sections (95% interval 1.4721 to 1.5757).

Relation: Difference with interval

metric

Bits per byteQuantity compared.

model

APT4 transplant of Qwen2.5-1.5BModels measured.

subject

Random embedding initialisationInitialisation of the model whose value is the minuend.

comparator

Frequency-based Vocabulary TransferInitialisation of the model whose value is subtracted.

evaluation text

English LaTeX method sectionsCorpus scored, first 200 documents.

training tokens

150000000 tokensTokens of embeddings-only continued pretraining, nominal checkpoint size, before the value is measured.

value

1.528 bits per byteDifference.

interval low

1.4721 bits per byteLower bound of the 95% interval.

interval high

1.5757 bits per byteUpper bound of the 95% interval.

bin

RESIDUALPre-registered class: CLOSED, PARTIAL or RESIDUAL.

Method and evaluation protocol

Paired document bootstrap (1000 resamples, seed 20260705) of random-minus-FVT bits per byte at the 150M-token checkpoints.

Dataset

First 200 documents of the evaluation domains, scored at the 150M-token checkpoints of embeddings-only training on the packed APT4 stream.Version: unspecified · Access: restricted

Reported results

1.528 [1.4721, 1.5757], RESIDUAL; random 2.62, FVT 1.0919.

Uncertainty and replication

95% paired document bootstrap interval.

Evidence references

analysis_s2.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/7d450d1f62209ff5ac5415543443d2a25db74b7e/experiments/E07-embedding-init/results/analysis_s2.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/7d450d1f62209ff5ac5415543443d2a25db74b7e/experiments/E07-embedding-init/results/metrics.json

Limitations

One training run per initialisation at 1.5B. The trainer's loss and gradient-norm logs, the smoke-run gate results, the transfer digests and the token stream's digest were not kept. Windows of 2,048 tokens.