Between digits, the no-break space, narrow no-break space and minus sign add 1, 1 and 1 tokens with the Polish-only 32k tokenizer with separator pieces, 2, 3 and 3 with the Polish-only 32k tokenizer, 2, 3 and 3 with APT4, and 1, 3 and 1 with the Mistral-derived tokenizer.

Relation: Tokens per character by tokenizer

metric

Tokens added between digitsQuantity measured.

characters

U+00A0 no-break space; U+202F narrow no-break space; U+2212 minus signCharacters measured, in the order of every values role.

subject

Polish-only 32k tokenizer with separator piecesTokenizer with fixed pieces for the characters.

subject values

1; 1; 1Tokens per character with the subject.

comparator

Polish-only 32k tokenizerSame-text tokenizer without the pieces.

comparator values

2; 3; 3Tokens per character with the comparator.

reference

APT4 tokenizerFirst reference tokenizer.

reference values

2; 3; 3Tokens per character with the first reference.

second reference

Mistral-derived tokenizerSecond reference tokenizer.

second reference values

1; 3; 1Tokens per character with the second reference.

Method and evaluation protocol

Encode '1<character>234' and '1234' after the carrier 'a ' without special tokens and count the tokens that intersect the number span.

Dataset

Synthetic number strings; no corpus.Version: unspecified · Access: public

Reported results

Separator arm 1; 1; 1; Polish-only arm 2; 3; 3; APT4 2; 3; 3; Mistral-derived 1; 3; 1. The Polish-only arm learned none of the three characters as pieces.

Uncertainty and replication

Exact counts; no sampling.

Evidence references

policy_audit.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/c5299916db7f792a209ff2944e66c673c3ebb817/experiments/E09-vocabulary-allocation/results/policy_audit.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/c5299916db7f792a209ff2944e66c673c3ebb817/experiments/E09-vocabulary-allocation/results/metrics.json

Limitations

Fixed pieces act as segmentation boundaries during training, so no merge includes them (no fused word-boundary minus piece). The audit script was last written seconds before its only recorded output.