On numbers of at least four digits grouped with no-break spaces, APT4 uses 10.4214 tokens per number and the Mistral-derived tokenizer 9.0303.

Relation: Metric comparison

metric

Tokens per numberQuantity compared.

subject

APT4 tokenizerTokenizer after the tokenizer change.

subject value

10.4214 tokens per numberValue of the metric for the subject.

comparator

Mistral-derived tokenizerTokenizer before the tokenizer change.

comparator value

9.0303 tokens per numberValue of the metric for the comparator.

setting

No-break-space-grouped number formatNumber format measured.

scope

synthetic integers from 10^4 to 10^9 and decimals whose integer part has at least four digitsNumbers measured.

Method and evaluation protocol

Encode each number after the prefix "a " without special tokens and count the tokens whose character spans overlap it; classify digit policy by comparing token boundaries inside 5,000 integers with single-digit and three-digit chunk predictions.

Dataset

Synthetic number sets S1 to S8 generated with seed 20260703, tokenized by tokenizer files at pinned revisions.Version: unspecified · Access: public

Reported results

10.4214 against 9.0303 tokens per number; APT4 encodes the no-break space as two byte tokens, the Mistral-derived tokenizer as one token.

Uncertainty and replication

Means over deterministic sets.

Evidence references

audit.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/2fc1a1b415a06b7648cdb37319ad573ccdc96a81/experiments/E02-digit-policy/results/audit.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/2fc1a1b415a06b7648cdb37319ad573ccdc96a81/experiments/E02-digit-policy/results/metrics.json

Limitations

Narrow no-break spaces are three byte tokens under both tokenizers.