Fast Vocabulary Transfer of APT4 into Qwen2.5-1.5B initialises 31,741 pieces as means of their Qwen re-encodings, which average 2.4116 Qwen pieces; 256 byte-fallback pieces and 3 special pieces receive fixed rows, and 0 pieces have an empty re-encoding.

Relation: Initialisation counts

method

Frequency-based Vocabulary TransferInitialisation method.

source model

Qwen2.5-1.5BModel whose embeddings are the source.

target tokenizer

APT4 tokenizerTokenizer whose pieces are initialised.

method pieces

31741 piecesPieces initialised from their re-encodings.

mean source pieces

2.4116 piecesMean number of source pieces per re-encoded piece.

byte pieces

256 piecesByte-fallback pieces given the mean embedding.

special pieces

3 piecesSpecial pieces given fixed rows.

empty pieces

0 piecesPieces whose re-encoding is empty.

Method and evaluation protocol

Each APT4 piece, with the word-boundary marker replaced by a space, is re-encoded with the Qwen tokenizer and its embedding set to the mean of those Qwen rows; <s> and </s> take the <|endoftext|> row; <unk> and byte pieces take the mean embedding; embeddings are tied.

Dataset

The APT4 vocabulary of Bielik-PL-11B-v3.0-Instruct and the Qwen2.5-1.5B embedding matrix.Version: unspecified · Access: restricted

Reported results

fvt 31741, byte 256, special 3, empty 0, mean Qwen pieces per re-encoded piece 2.411581235625847.

Uncertainty and replication

Exact counts.

Evidence references

fvt_stats.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/9a57aa40f4e774136f51b6730c33266d59094a45/experiments/E05-control-arm/results/fvt_stats.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/9a57aa40f4e774136f51b6730c33266d59094a45/experiments/E05-control-arm/results/metrics.json

Limitations

The statistics file was not committed in the archive; it was copied from the scoring machine's time-zero model directory. The script as archived did not write the eos fix of the model that was trained.