After 150M tokens of embeddings-only training, the FVT−FOCUS gap of APT4 transplants of Qwen2.5-1.5B on the Polish FineWeb2-HQ holdout is 0.0861 bits per byte (95% interval 0.0805 to 0.0904), 0.224 of its time-zero value 0.3844.

Relation: Persistence of a difference

metric

Bits per byteQuantity compared.

model

APT4 transplant of Qwen2.5-1.5BModels measured.

subject

Frequency-based Vocabulary TransferInitialisation of the model whose value is the minuend.

comparator

FOCUSInitialisation of the model whose value is subtracted.

evaluation text

Polish FineWeb2-HQ holdoutCorpus scored, first 200 documents.

training tokens

150000000 tokensTokens of embeddings-only continued pretraining, nominal checkpoint size, before the value is measured.

value

0.0861 bits per byteDifference after training.

interval low

0.0805 bits per byteLower bound of the 95% interval.

interval high

0.0904 bits per byteUpper bound of the 95% interval.

baseline value

0.3844 bits per byteDifference at time zero.

ratio

0.224 ratioDifference after training divided by the difference at time zero.

bin

VANISHEDPre-registered class: VANISHED, SHRINKS or PERSISTS.

Method and evaluation protocol

Paired document bootstrap (1000 resamples, seed 20260705) of FVT-minus-FOCUS bits per byte at the 150M-token checkpoints, divided by the time-zero difference.

Dataset

First 200 documents of the evaluation domains, scored at the 150M-token checkpoints of embeddings-only training on the packed APT4 stream.Version: unspecified · Access: restricted

Reported results

0.0861 [0.0805, 0.0904]; time zero 0.3844; ratio 0.224; VANISHED.

Uncertainty and replication

95% paired document bootstrap interval on the 150M-token value; none on the ratio.

Evidence references

analysis_s2.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/7d450d1f62209ff5ac5415543443d2a25db74b7e/experiments/E07-embedding-init/results/analysis_s2.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/7d450d1f62209ff5ac5415543443d2a25db74b7e/experiments/E07-embedding-init/results/metrics.json

Limitations

The auxiliary fastText text (the first 209,715,204 bytes of the Polish FineWeb2-HQ training text) and the Polish FineWeb2-HQ holdout were drawn from the same source without a shared exclusion: a line check in the port found that 3 of the 180 scored holdout documents with lines of 200 to 4,000 bytes share at least one such line with the auxiliary text, and none shares half of them. One training run per initialisation at 1.5B. The trainer's loss and gradient-norm logs, the smoke-run gate results, the transfer digests and the token stream's digest were not kept. Windows of 2,048 tokens.