Of the 9 tokenizers of Table 1 of arXiv:2604.10799v1, 7 split every classified integer into one token per digit, the SmolLM3 (Llama 3.2) tokenizer splits every one into three-digit chunks from the left, and APT3 follows neither pattern.

Relation: Digit policy class counts

tokenizers total

9 tokenizersTokenizers classified.

count single digit

7 tokenizersTokenizers with the single-digit policy.

single digit tokenizers

APT4; Mistral-derived tokenizer; Qwen3; Gemma 3; EuroLLM; Mistral Small 3.1; ApertusTokenizers with the single-digit policy.

count three digit left to right

1 tokenizersTokenizers splitting digits into three-digit chunks from the left.

three digit left to right tokenizers

SmolLM3 (Llama 3.2)Tokenizers splitting digits into three-digit chunks from the left.

count free bpe

1 tokenizersTokenizers following neither pattern.

free bpe tokenizers

APT3Tokenizers following neither pattern.

integers classified

5000 integersIntegers classified per tokenizer.

Method and evaluation protocol

Encode each number after the prefix "a " without special tokens and count the tokens whose character spans overlap it; classify digit policy by comparing token boundaries inside 5,000 integers with single-digit and three-digit chunk predictions.

Dataset

Synthetic number sets S1 to S8 generated with seed 20260703, tokenized by tokenizer files at pinned revisions.Version: unspecified · Access: public

Reported results

Single-digit: APT4; Mistral-derived tokenizer; Qwen3; Gemma 3; EuroLLM; Mistral Small 3.1; Apertus. Three-digit chunks from the left: SmolLM3 (Llama 3.2). Neither: APT3. Gemma 3 and EuroLLM have multi-digit vocabulary tokens (176 and 4) but split every classified integer one token per digit.

Uncertainty and replication

Exact counts over deterministic sets.

Evidence references

audit.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/2fc1a1b415a06b7648cdb37319ad573ccdc96a81/experiments/E02-digit-policy/results/audit.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/2fc1a1b415a06b7648cdb37319ad573ccdc96a81/experiments/E02-digit-policy/results/metrics.json

Limitations

The Apertus and Mistral Small 3.1 tokenizers have identical audit profiles although their tokenizer files differ.