After 150M tokens of embeddings-only training, the random-minus-FVT bits-per-byte gap of APT4 transplants of Qwen2.5-1.5B is RESIDUAL on 3 of 3 formal domains, giving the class FLOOR-RESIDUAL.

Relation: Count in a bin

metric

Bits per byteQuantity compared.

model

APT4 transplant of Qwen2.5-1.5BModels measured.

subject

Random embedding initialisationInitialisation whose value is the minuend.

comparator

Frequency-based Vocabulary TransferInitialisation whose value is subtracted.

domains

Formal domainsDomains classified.

training tokens

150000000 tokensTokens of embeddings-only continued pretraining, nominal checkpoint size, before the value is measured.

bin

RESIDUAL: gap at least 0.20 bits per byte with the 95% interval excluding zeroBin counted.

count in bin

3 domainsDomains in the bin.

count total

3 domainsDomains classified.

class

FLOOR-RESIDUALPre-registered class given by the per-domain bins: HEALS, PARTIAL or FLOOR-RESIDUAL.

Method and evaluation protocol

Apply the pre-registered per-domain bins and class rule.

Dataset

First 200 documents of the evaluation domains, scored at the 150M-token checkpoints of embeddings-only training on the packed APT4 stream.Version: unspecified · Access: restricted

Reported results

sci_latex: RESIDUAL; sci_python: RESIDUAL; math_clean: RESIDUAL; class FLOOR-RESIDUAL.

Uncertainty and replication

Per-domain 95% paired bootstrap intervals.

Evidence references

analysis_s2.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/7d450d1f62209ff5ac5415543443d2a25db74b7e/experiments/E07-embedding-init/results/analysis_s2.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/7d450d1f62209ff5ac5415543443d2a25db74b7e/experiments/E07-embedding-init/results/metrics.json

Limitations

One training run per initialisation at 1.5B. The trainer's loss and gradient-norm logs, the smoke-run gate results, the transfer digests and the token stream's digest were not kept. Windows of 2,048 tokens.