Of 15 separator, sign, percent and unit exemplars, APT4 and the Mistral-derived tokenizer segment 5 differently: three containing a no-break space, one with a U+2212 minus sign and one with a degree sign after a space.

Relation: Differ on a count of exemplars

subject

APT4 tokenizerTokenizer compared.

comparator

Mistral-derived tokenizerTokenizer compared with.

count differing

5 exemplarsExemplars segmented differently.

count total

15 exemplarsExemplars compared.

differing exemplars

no-break space as thousands separator, before a percent sign and before a unit; U+2212 minus sign before a number; degree sign after a spaceExemplars segmented differently.

identical exemplars

comma, space, narrow no-break space and period grouping with their decimal separators; ASCII minus sign; percent sign attached and after a space; units attached, after a space and after a decimal commaExemplars segmented identically.

Method and evaluation protocol

Tokenize fixed exemplars of grouped numbers, signs, percent signs and units after the prefix "a " and compare the tokens covering each separator, sign or unit.

Dataset

Fixed exemplars of grouped numbers, signs, percent signs and units defined in scripts/04_audit.py.Version: unspecified · Access: public

Reported results

No-break space: two byte tokens under APT4, one token under the Mistral-derived tokenizer; U+2212: three byte tokens against one; a degree sign after a space: two tokens against one.

Uncertainty and replication

Exact segmentation of fixed exemplars.

Evidence references

audit.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/2fc1a1b415a06b7648cdb37319ad573ccdc96a81/experiments/E02-digit-policy/results/audit.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/2fc1a1b415a06b7648cdb37319ad573ccdc96a81/experiments/E02-digit-policy/results/metrics.json

Limitations

One exemplar per construction; the degree-sign difference was not reported in the original report.