After 150M tokens of embeddings-only training, bits per byte of the random-initialised APT4 transplant of Qwen2.5-1.5B exceeds that of the FVT-initialised transplant by 1.4148 bits per byte on English Python code (95% interval 1.374 to 1.4596).

Relation: Difference with interval

metric

Bits per byteQuantity compared.

model

APT4 transplant of Qwen2.5-1.5BModels measured.

subject

Random embedding initialisationInitialisation of the model whose value is the minuend.

comparator

Frequency-based Vocabulary TransferInitialisation of the model whose value is subtracted.

evaluation text

English Python codeCorpus scored, first 200 documents.

training tokens

150000000 tokensTokens of embeddings-only continued pretraining, nominal checkpoint size, before the value is measured.

value

1.4148 bits per byteDifference.

interval low

1.374 bits per byteLower bound of the 95% interval.

interval high

1.4596 bits per byteUpper bound of the 95% interval.

bin

RESIDUALPre-registered class: CLOSED, PARTIAL or RESIDUAL.

Method and evaluation protocol

Paired document bootstrap (1000 resamples, seed 20260705) of random-minus-FVT bits per byte at the 150M-token checkpoints.

Dataset

First 200 documents of the evaluation domains, scored at the 150M-token checkpoints of embeddings-only training on the packed APT4 stream.Version: unspecified · Access: restricted

Reported results

1.4148 [1.374, 1.4596], RESIDUAL; random 2.3822, FVT 0.9674.

Uncertainty and replication

95% paired document bootstrap interval.

Evidence references

analysis_s2.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/7d450d1f62209ff5ac5415543443d2a25db74b7e/experiments/E07-embedding-init/results/analysis_s2.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/7d450d1f62209ff5ac5415543443d2a25db74b7e/experiments/E07-embedding-init/results/metrics.json

Limitations

One training run per initialisation at 1.5B. The trainer's loss and gradient-norm logs, the smoke-run gate results, the transfer digests and the token stream's digest were not kept. Windows of 2,048 tokens.